Grafana Agent

Sometimes applications are instrumented with more metrics than we want to afford, from a resources perspective. In such cases we can choose to selectively drop some metrics before they are ingested.

Discover high-cardinality metrics

Prometheus’s /tsdb-status endpoint is quite handy for discovering high cardinality metrics you may want to drop prior to ingestion.

In this guide, we will show how to drop all the go_gc_* metrics. To check how many go_gc_* metrics you currently have, run the following query:

count by (__name__) ({__name__=~"go_gc_.+"})

Means for dropping metrics

Metrics can be dropped by using the drop action in several different places:

• Under <scrape_config> section (<metric_relabel_configs> subsection). For example: all the self-monitoring scrape jobs that e.g. COS Lite has in place.
• Under <remote_write> section (<write_relabel_configs> subsection). For example: prometheus can be told to drop metrics before pushing them to another prometheus over remote-write API. This use case is not addressed in this guide.

Drop metrics via metric name in scrape config

Charms that integrate with prometheus or grafana agent, provide a “scrape config” to MetricsEndpointProvider (imported from charms.prometheus_k8s.v0.prometheus_scrape).

Let’s take for example the alertmanager self-metrics that prometheus scrapes. If we do not want prometheus or grafana agent to ingest any go_gc_* metrics from alertmanager, then we need to adjust the scrape job specified in the alertmanager charm:

diff --git a/src/charm.py b/src/charm.py
index fa3678c..f0e943b 100755
--- a/src/charm.py
+++ b/src/charm.py
@@ -250,6 +250,13 @@ class AlertmanagerCharm(CharmBase):
             "scheme": metrics_endpoint.scheme,
             "metrics_path": metrics_path,
             "static_configs": [{"targets": [target]}],
+            "metric_relabel_configs": [
+                {
+                    "source_labels": ["__name__"],
+                    "regex": "go_gc_.+",
+                    "action": "drop",
+                }
+            ]
         }
 
         return [config]

Drop metrics via the scrape-config charm

In a typical scrape-config deployment such as:

graph LR
some-external-target --- scrape-target --- scrape-config --- prometheus

We can specify the drop action via a config option for the scrape-config charm:

$ juju config sc metric_relabel_configs="$(cat <<EOF
- source_labels: ["__name__"]
  regex: "go_gc_.+"
  action: "drop"
EOF
)"

References

• Official docs: <relabel_config>
• Dropping metrics at scrape time with Prometheus (robustperception, 2015)
• How relabeling in Prometheus works (grafana.com, 2022)
• How to drop and delete metrics in Prometheus (gh:tanmay-bhat, 2022)
• Playgrounds:
  • https://demo.promlens.com/
  • https://relabeler.promlabs.com/