We discovered an issue with our Elasticsearch monitoring for Prometheus that was introduced a while back in a rutinary chart upgrade. Because of this problem some Elasticsearch metrics were not being reported into Prometheus, like available storage space for example, and as a result there were some problematic situations in an Elasticsearch cluster that we didn’t pick up in time.
We’ve already resolved the issue and rolled out the fix to all Elasticsearch setups.