We intend to make breaking changes to the Loki labels we assign to logs collected from your clusters. These changes aim to improve Loki’s performance by reducing high-cardinality labels and removing duplicates (e.g. namespace removed in favor of namespace_name). If you rely on any of the labels we will no longer include automatically, your queries may need to be updated. We plan to roll out this change in the week of 2025-06-02. Your feedback is important to us, please let us know if you have any questions or concerns about this change.
In addition, we already rolled out an improvement for better parsing of non-json logs. You can already notice this change in logs like those of nginx-ingress that are no longer unnecessarily escaped.
What is changing?
We will no longer be adding all Kubernetes labels automatically as Loki stream labels.
Duplicates we had before will also be removed:
namespacewill be removed in favor ofnamespace_name.
The following labels are added by default:
container_image
container_name
namespace_name
stream
job # This is a special label used to identify the source of the logs (eg job=kube for container logs)
The following Loki labels will be indexed (if used as Kubernetes label):
app
app_kubernetes_io_component
app_kubernetes_io_instance
app_kubernetes_io_name
app_kubernetes_io_part_of
component
cronjob
k8s_app
namespace_name
release
service_name
statefulset_kubernetes_io_pod_name
All other Kubernetes labels will be added as Loki structured metadata instead of stream labels. This means that they will not be indexed, but you can still use them in your queries with a small modification. For example, if you were using pod_name in your queries:
this query: {namespace_name="prod", app_kubernetes_io_name="my-app", pod-name="my-app-abc1234-56789"}
will become: {namespace_name="prod", app_kubernetes_io_name="my-app"} | pod-name="my-app-abc1234-56789"
So instead of using the label as a stream label, you will use it as a filter in the query. Practically this means that the pod-name filtering will happen on the querier side instead of the backend.
Important: If you require any other index labels, please reach out to us as we can customise them for your use-case through the
spec.fluent_bit.config.loki_output_labelsparameter in your cluster definition file.