Alertmanager

  • By Dylan Stephano-Shachter
Channel Version Revision Published Runs on
latest/stable 36 36 21 Oct 2022
Ubuntu 20.04
latest/candidate 36 36 21 Oct 2022
Ubuntu 20.04
latest/beta 36 36 21 Oct 2022
Ubuntu 20.04
latest/edge 37 37 22 Nov 2022
Ubuntu 20.04
1.0/stable 36 36 21 Oct 2022
Ubuntu 20.04
1.0/candidate 36 36 21 Oct 2022
Ubuntu 20.04
1.0/beta 36 36 21 Oct 2022
Ubuntu 20.04
1.0/edge 36 36 21 Oct 2022
Ubuntu 20.04
juju deploy alertmanager-k8s
Show information

Platform:

Alertmanager Operator (k8s)

Test Suite Discourse status

This Charmed Operator handles instantiation, scaling, configuration, and Day 2 operations specific to Alertmanager.

This operator drives the Alertmanager application, and it can be composed with other operators to deliver a complex application or service, such as COS Lite.

Alertmanager receives alerts from supporting applications, such as Prometheus or Loki, then deduplicates, groups and routes them to the configured receiver(s).

Getting started

Basic deployment

Once you have a controller and model ready, you can deploy alertmanager using the Juju CLI:

juju deploy --channel=beta alertmanager-k8s

The available channels are listed at the top of the page and can also be retrieved with Charmcraft CLI:

$ charmcraft status alertmanager-k8s

Track    Base                  Channel    Version    Revision    Resources
latest   ubuntu 20.04 (amd64)  stable     -          -           -
                               candidate  -          -           -
                               beta       9          9           alertmanager-image (r1)
                               edge       9          9           alertmanager-image (r1)

Once the Charmed Operator is deployed, the status can be checked by running:

juju status --relations --storage --color
Configuration

In order to have alerts dispatched to your receiver(s) of choice, a configuration file must be provided to Alertmanager using the config_file option:

juju config alertmanager-k8s \
  config_file='@path/to/alertmanager.yml'

Note that if you use templates, you should use the templates_file config option instead of having a templates section in your yaml configuration file. (This is a slight deviation from the official alertmanager config spec.)

Use the templates_file option to push templates that are being used by the configuration file:

juju config alertmanager-k8s \
  config_file='@path/to/alertmanager.yml' \
  templates_file='@path/to/templates.tmpl'

All templates need to go into this single config option, instead of the 'templates' section of the main configuration file. The templates will be pushed to the workload container, and the configuration file will be updated accordingly.

Refer to the official templates documentation for more details.

To verify Alertmanager is using the expected configuration you can use the show-config action:

juju run-action alertmanager-k8s/0 show-config --wait
Dashboard and HTTP API

The Alertmanager dashboard and HTTP API can be accessed at the default port (9093) on the Alertmanager IP address, which is determinable with a juju status command.

To obtain the load-balanaced application IP,

juju status alertmanager-k8s --format=json \
  | jq -r '.applications."alertmanager-k8s".address'

Similarly, to obtain an individual unit's IP address:

juju status alertmanager-k8s --format=json \
  | jq -r '.applications."alertmanager-k8s".units."alertmanager-k8s/0".address'

Clustering

Forming a cluster

Alertmanager supports clustering and all you need to do to create/update a cluster is to rescale the application. This can be done in two ways.

Let's say we have one alertmanager unit running and we want to scale the deployment to three units.

With juju add-unit we can achieve that using the --num-units argument and the number of units we want to add:

juju add-unit alertmanager-k8s --num-units 2

or using juju scale-application and the total number of units we want:

juju scale-application alertmanager-k8s 3

Regardless of which of the two options you use, juju status --relations --color will show you the status of the cluster.

Internally, HA is achieved by providing each Alertmanager instance at least one IP address of another instance. The cluster would then auto-update with subsequent changes to the units present.

Verification
Pebble plan

Cluster information is passed to Alertmanager via --cluster.peer command line arguments. This can be verified by looking at the current pebble plan:

> juju exec --unit alertmanager-k8s/0 -- \
  PEBBLE_SOCKET=/charm/containers/alertmanager/pebble.socket \
  pebble plan

services:
    alertmanager:
        summary: alertmanager service
        startup: enabled
        override: replace
        command: alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager --web.listen-address=:9093 --cluster.listen-address=0.0.0.0:9094 --cluster.peer=10.1.179.220:9094 --cluster.peer=10.1.179.221:9094
HTTP API

To manually verify a cluster is indeed formed, you can query the alertmanager HTTP API directly:

> curl -s $ALERTMANAGER_IP:9093/api/v1/status \
  | jq '.data.clusterStatus.peers[].address'
"10.1.179.220:9094"
"10.1.179.221:9094"
"10.1.179.217:9094"

OCI Images

This charm is published on Charmhub with alertmanager images from ubuntu/prometheus-alertmanager, however, it should also work with the official quay.io/prometheus/alertmanager.

To try the charm with a different image you can use juju refresh. For example:

juju refresh alertmanager-k8s \
  --resource alertmanager-image=quay.io/prometheus/alertmanager

(Note: currently, refreshing to a different image only works when deploying from a local charm - lp/1954462.)

Resource revisions

Workload images are archived on charmhub by revision number.

Resource Revision Image
alertmanager-image r1 0.21-20.04_beta

You can use charmcraft to see the mapping between charm revisions and resource revisions:

charmcraft status alertmanager-k8s

Official alertmanager documentation

For further details about Alertmanager configuration and usage, please refer to the official Alertmanager documentation.

Additional Information