Kubernetes events are a great resource to debug and troubleshoot problems with workloads and other cluster components. The problem is that the K8s API stores them for only 1 hour. To be able to persist those events further in time, we used an open-source component called eventrouter, which streamed all cluster events into Loki. This project has been deprecated and unmaintained for a while now, so we needed to find a replacement for it.
We’ve decided to go with a Grafana Agent setup with its eventhandler integration. It works in a similar way as the eventrouter, but streams the events directly to Loki, instead of outputing them as Pod logs and letting the logging system process them further. It also stores the events in a different format in Loki, so your current queries might vary slightly. You’ll see that each event entry has less clutter, but it should have all relevant data.
You can still query the events with the app=eventrouter label selector and you can drill further down by namespace with the namespace=<ns> selector. E.g.
{app="eventrouter", namespace="infrastructure"} | logfmt | kind="ReplicaSet"
We’ve already rolled this out to all clusters.