K8S - Not able to see alerts via - alertmanager - kubernetes

Question

I've Prometheus operator which is working as expected
https://github.com/coreos/prometheus-operator
Now I want to apply the alert manager from scratch
After reading the docs im came out with those yamls.
but the problem is when I entered to the UI
Nothing is shown, any idea what I miss here ?
http://localhost:9090/alerts
I use port forwarding ...
This is all the config files I've apply to my k8s cluster
I just want to do some simple test to see that it working and then extend it to our needs...
alertmanger_main.yml
---
apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
name: main
labels:
alertmanager: main
spec:
replicas: 3
version: v0.14.0
alertmanger_service.yml
apiVersion: v1
kind: Service
metadata:
name: alertmanager-main
spec:
type: LoadBalancer
ports:
- name: web
port: 9093
protocol: TCP
targetPort: web
selector:
alertmanager: main
testalert.yml
kind: ConfigMap
apiVersion: v1
metadata:
name: prometheus-example-rules
labels:
role: prometheus-rulefiles
prometheus: prometheus
data:
example.rules.yaml: |+
groups:
- name: ./example.rules
rules:
- alert: ExampleAlert
expr: vector(1)
alertmanager.yml
global:
resolve_timeout: 5m
route:
group_by: ['job']
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: 'webhook'
receivers:
- name: 'webhook'
webhook_configs:
- url: 'http://alertmanagerwh:30500/'
and to create secret I use
kubectl create secret generic alertmanager-main --from-file=alertmanager.yaml
what I need is some basic alerts in K8S and I follow the documatation but didnt find any good step by step tutorial
to check my sys for monitoring namespace
~  kubectl get pods -n monitoring 13.4m  Sun Feb 17 18:48:16 2019
NAME READY STATUS RESTARTS AGE
kube-state-metrics-593czc6b4-mrtkb 2/2 Running 0 12h
monitoring-grafana-771155cbbb-scqvx 1/1 Running 0 12h
prometheus-operator-79f345dc67-nw5zc 1/1 Running 0 12h
prometheus-prometheus-0 3/3 Running 1 12h
~  kubectl get svc -n monitoring 536ms  Sun Feb 17 21:04:51 2019
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-main NodePort 100.22.170.666 <none> 9093:30904/TCP 4m53s
kube-state-metrics ClusterIP 100.34.212.596 <none> 8080/TCP 4d7h
monitoring-grafana ClusterIP 100.67.230.884 <none> 80/TCP 4d7h
prometheus-operated ClusterIP None <none> 9090/TCP 4d7h
I've also now changed the service to LoadBalancer and I try to enter like
~  kubectl get svc -n monitoring 507ms  Sun Feb 17 21:23:56 2019
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-main LoadBalancer 100.22.170.666 38.482.152.331 9093:30904/TCP 23m
when I hit the browser with
38.482.152.331:9093
38.482.152.331:30904
nothing happen...

Answer 1

When you consider using AlertManager, besides the general configuration and applying alert rules, AlertManager requires being integrated with a Prometheus server. The Prometheus instance can then track any incoming series of events, and once it detects any rule which is recognized, it triggers an alert to the nested alertmanager.
In order to enable alerting it might be necessary to append the following config the to Prometheus instance:
alerting:
alertmanagers:
- static_configs:
- targets:
- 'alertmanagerIP:9093'
Specifically, for AlertManager implementation in CoreOS, you can follow the steps described in the official Alerting documentation; however, below you can find example for Prometheus pod alerting configuration kept from the mentioned guideline:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: example
spec:
replicas: 2
alerting:
alertmanagers:
- namespace: default
name: alertmanager-example
port: web
serviceMonitorSelector:
matchLabels:
team: frontend
resources:
requests:
memory: 400Mi
ruleSelector:
matchLabels:
role: prometheus-rulefiles
prometheus: example