Handling alerts triggered by boolean condition; keep alertmanager from auto-resolving until manually cleared - kubernetes

Question

given the following promql with alertmanager integration:
what's a good way to alter this expression so that when a pod restart is indicated by this promql and an alert triggered (expected) that an alert isn't automatically resolved on the next check since on the next check it's likely a restart will no longer register (unless a pod is flapping or something). We want to be alerted so we can check it out so we want to manually resolve the alert:
sum(rate(kube_pod_container_status_restarts_total[2m])*100) by (namespace,container) > 0
A potential alertmanager rule would be the following, but we don't yet have an actual rule.
- alert: Unexpected POD restart(s) occurring
expr: sum(rate(kube_pod_container_status_restarts_total[2m])*100) by (namespace,container) > 0
for: 10s
annotations:
summary: POD restarts occurring

Answer 1

An alert is raised when your query produces at least one metric and the alert is resolved when it no longer produces a metric.
sum by (...) (rate(kube_pod_container_status_restarts_total[2m])) > 0
means that it calculates the rate over the last two minutes. So if a restart occurs, the this expression will result in 0 after 2 minutes and your alert is resolved.
What you could do is simply increase the time interval:
sum by (...) (rate(kube_pod_container_status_restarts_total[1h])) > 0
so the alert stays on for an hour.
May be you dont want to check for every single restart, but if there are pods that restart within a specific time at least twice, so that might be the better expression:
sum by (...) (increase(kube_pod_container_status_restarts_total[24h])) > 2
meaning if a container restert at least two times within a day that might be a situation you want to have a look at.