How to config email alert in using grafana and prometheus-operator - grafana

Question

I installed prometheus-operator (include prometheus/alertmanager/grafana) via helm. Then I access Grafana UI and config alert via email. When I click send an email test, I got the message “ SMTP not configured, check your grafana.ini config file’s [smtp] section”
But I don’t know where the grafana.ini to can change in this case.
[root#k8s-master ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5bbc8f45cb-nlqgh 1/1 Running 1 15h
kube-system calico-node-lk2j5 1/1 Running 1 15h
kube-system calico-node-v6wzs 1/1 Running 1 15h
kube-system calico-node-zfh5r 1/1 Running 1 15h
kube-system coredns-5c98db65d4-79c2g 1/1 Running 1 15h
kube-system coredns-5c98db65d4-bqj7g 1/1 Running 1 15h
kube-system etcd-k8s-master 1/1 Running 1 15h
kube-system kube-apiserver-k8s-master 1/1 Running 1 15h
kube-system kube-controller-manager-k8s-master 1/1 Running 2 15h
kube-system kube-proxy-8qmdt 1/1 Running 1 15h
kube-system kube-proxy-qwgbc 1/1 Running 1 15h
kube-system kube-proxy-vhqjd 1/1 Running 1 15h
kube-system kube-scheduler-k8s-master 1/1 Running 1 15h
monitoring alertmanager-prometheus-operator-alertmanager-0 2/2 Running 3 15h
monitoring prometheus-operator-grafana-64848fc9bb-dbnwc 2/2 Running 3 15h
monitoring prometheus-operator-kube-state-metrics-5d46566c59-ck4np 1/1 Running 2 15h
monitoring prometheus-operator-operator-64dcc7bfc-lpdj6 2/2 Running 2 15h
monitoring prometheus-operator-prometheus-node-exporter-ns4kg 1/1 Running 1 15h
monitoring prometheus-operator-prometheus-node-exporter-tdhwq 1/1 Running 2 15h
monitoring prometheus-operator-prometheus-node-exporter-xt8z9 1/1 Running 2 15h
monitoring prometheus-prometheus-operator-prometheus-0 3/3 Running 4 15h

Answer 1

You will be able to override this configuration, using helm variables, thanks to alertmanager.config key.
This key convert yaml into configuration for alertmanager, so you can use every alertmanager configuration.
You should probably also change grafana.ini configuration to configure smtp into grafana (test seems to use that configuration). You can check this configuration in Grafana via "Server admin" > "Settings", search "smtp".
As a reference, you can do something like the following for alertmanager :
helm upgrade --install prometheus stable/prometheus-operator \
-f helm/prometheus-operator.yml \
-f helm/grafana-custom.staging.yml \
--set-string alertmanager.config.global.smtp_smarthost="my.smtp.tld:465" \
--set-string alertmanager.config.global.smtp_auth_username="my#email.tld" \
--set-string alertmanager.config.global.smtp_from="my#email.tld" \
--set-string alertmanager.config.global.smtp_auth_password="MyAmazingPassword" \
--set-string grafana.'grafana\.ini'.smtp.enabled=true \
--set-string grafana.'grafana\.ini'.smtp.host="my.smtp.tld:465" \
--set-string grafana.'grafana\.ini'.smtp.from_address="my#email.tld" \
--set-string grafana.'grafana\.ini'.smtp.user="my#email.tld" \
--set-string grafana.'grafana\.ini'.smtp.password="MyAmazingPassword"

Answer 2

The grafana.ini is loaded through configmaps in prometheus-operator helm deployment. If you have already installed it via helm then you can just modify the configmap and then restart the grafana pod. Below is minimum config with which i was able to use SMTP.
[smtp]
enabled = true
host = your.smtp.server.name:25
skip_verify = true
from_address = "grafana#xyz.com"
from_name = Grafana
To get the configmap, run below command and edit the configmap(include namespace in below command if prometheus-operator is deployed in a speratae name space than default).
kubectl get configmap | grep grafana
After editing configmap, restart grafana pod(restarts of other pods is no needed).
Note: Skip_verify = true is not recommended.