automatic labels to prometheus alertmanager rules - kubernetes

Question

I'm using prometheus-community/prometheus chart
I'd like to add the following labels automatically to any alert manager rule firing
env=prod
cluster=project-prod-eks
so that I don't these labels manually to each alert rule.
- alert: NGINXTooMany400s
expr: 100 * ( sum( nginx_ingress_controller_requests{status=~"4.+"} ) / sum(nginx_ingress_controller_requests) ) > 5
for: 1m
labels:
severity: warning
env: prod
cluster: project-prod-eks <---------------HOW to inject them?
annotations:
description: Too many 4XXs
summary: More than 5% of all requests returned 4XX, this requires your attention
so that I can do something like
- alert: NGINXTooMany400s
expr: 100 * ( sum( nginx_ingress_controller_requests{status=~"4.+"} ) / sum(nginx_ingress_controller_requests) ) > 5
for: 1m
labels:
severity: warning
annotations:
description: Too many 4XXs on {{ $labels.env }} / {{ $labels.cluster }} <----- THIS
summary: More than 5% of all requests returned 4XX, this requires your attention
Any ideas?

Answer 1

You can add external_labels to your prometheus.yml:
global:
# The labels to add to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
env: prod
cluster: project-prod-eks
The community chart has it in values.yml:
serverFiles:
prometheus.yml:
global:
external_labels:
foo: bar
...

Answer 2

so I did slightly differently that modifying the "serverFiles", see below
server:
nodeSelector:
prometheus: "true"
baseURL: "https://prometheus.project.io"
enabled: true
retention: "30d"
strategy:
type: RollingUpdate
global:
scrape_interval: 30s
external_labels:
env: prod
client: client-name
project: project-name
cluster: project-prod-eks