For a while we’ve offered Grafana Loki as default logging solution. For shipping logs to Loki we were using the included Promtail. However, more recently, we’ve also supported other logging solutions, like Elasticsearch and Logz.io for customers with more advanced needs. To facilitate this we use the Fluent Bit log processor.
This means that we’ve been maintaining two similar components, and for some setups this also resulted in these 2 DaemonSets running in parallel. In short, this wasn’t efficient.
Therefore we’re standardizing completely on Fluent Bit only for log processing, which outputs to Grafana Loki and many other solutions.
In general, we support:
- Grafana Loki is always enabled, for at least the system/infrastructure logs.
- By default, all logs are shipped to Loki, but is configurable (eg. only the
productionnamespace, or no application logs at all). - Alternatively, we offer support for shipping app logs to Elasic- or Opensearch, Logz.io and plain S3. Others are also possible upon request.
Towards you, the cluster user, this change is mostly transparent, however there are some considerations.
- Some Loki labels have changed (different names), so make sure to check your queries against the new labels. Most notably you can use any K8s labels you’ve defined on your workloads in addition to
namespace_name,pod_name,container_name,host, etc. - The
joblabel is used to differentiate between different log sources: eg.job="kube"selects the K8s workload logs, whilejob="kubelet"select Kubelet system logs. - For K8s event logs, the
namespace_nameis no longer overriden by the event’s namespace. You can still get the same functionality with the following query (leveraging thejsonparser function introduced in Loki 2.0):{job="kube",app="eventrouter"} | json | event_metadata_namespace="<namespace>"
Make sure to check out our updated documentation for more examples.
These changes have been rolled out to all non-production environments. Production rollouts will follow next week.