Grafana

A common behavioural flaw we see in Grafana is that sometimes the Dashboards do not display information. We see this materialised in Panels that display legends such as “N/A” or “No data”.

Verify that Panel PromQL expression is valid

First of all we need to make sure that the PromQL expression that is executed in a Panel with an N/A or a No data legend is valid.

In order to do that, we need to edit that Panel by clicking on Edit option:

Once we are in the edition page, we see the PromQL expression below the graphic:

Sometimes a PromQL expression does not return data because there is no time series matching the labels we are filtering by. Keep in mind that in PromQL, labels are placed between { }.

So let’s try executing the same query, but without filters. In order to do that we need to:

• Remove the labels inside { }
• Replace $__rate_interval variable with a Time duration value.

So, our modified query looks like this:

(sum by(instance) (irate(node_cpu_seconds_totals{}[5m])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{}[5m])))) * 100

Now, let’s click on Menu/Explore

and execute modified query:

As we can see this query still does not return any results.

But if we look a little closer, we see that we are calculating the quotient of two sub-expressions that each have two different metrics: node_cpu_seconds_total and node_cpu_seconds_totals.

Confirm all metric names exists. If the metric (with the given labels) was not yet ingested by Prometheus, or if there is a typo in the metric name, then you will see “no data”. You can check if a metric exists by evaluating it in isolation in the “Explore” tab.

Bingo! we have found the error in our expression. Let’s go back to the Panel edition and fix the query there:

Verify that Drop-downs expressions are valid

Sometimes we have more than one panel that shows N/A or No data legends instead of a numerical value or a valid graphic.

In the header of our Dashboard, it is common to have drop-downs to apply filters to the dashboard graphics.

Let’s verify those drop-downs. To do that we have to click in the gear in the header and go to the Variables option:

Some of those variables defined in the Dashboard can be used as drop-downs. Besides those variables can be of different types: comma separated values, data source, query, etc.

Often the error lies in the variables of the query type. As in the previous case, the expression does not return values either because we refer to a metric that does not exist, or because we apply a combination of labels that does not produce a result.

In our example we have an error in the name of the metric. It is node_uname_info instead of node_uname_inf

If we apply that fix and go back to the Dashboard, we can see that the drop-downs have values and all graphics are working. Everything looks good!

Directly inspect relation data

Sometimes you may need to directly inspect the relation data incoming into grafana.

To get a listing of the dashboards that reached grafana over relation data, use juju show-unit together with jq:

juju show-unit grafana/0 --format=json \
  | jq -r '."grafana/0"."relation-info" | .[] | select(.endpoint == "grafana-dashboard") | ."application-data".dashboards' \
  | jq '.templates'

To decode the contents of a particular dashboard, say "file:prometheus-k8s_rev1.json",

juju show-unit grafana/0 --format=json \
  | jq -r '."grafana/0"."relation-info" | .[] | select(.endpoint == "grafana-dashboard") | ."application-data".dashboards' \
  | jq -r '.templates | select(has("file:prometheus-k8s_rev1.json")) | ."file:prometheus-k8s_rev1.json".content' \
  | base64 -d | xz --decompress --stdout