Datadog keeps creating monitors when dynamic source id changes - kubernetes

Question

I have a Datadog monitor generated by terraform.
The main query is as follows:
sum(last_1m):avg:app.application.health{application.health:healthy,cluster_name:${local.eks_cluster_name},!source:api-service-full} by {source}.as_count() < 60"
The issue is that after a system restart the {source} container changes it's name.
For example from app-tier-1-1abc-agent
to app-tier-1-def2-agent
The Datadog instead of updating, or removing the old monitors just creates new ones and leaves the old monitors in Alarm and N/A.
Is there anyway to improve this? All ideas appreciated, thanks!

Answer 1

Solved this by sending API calls to edit each monitor query on shutdown and startup.
I made a very clunky bash script, because I couldn't find a way to store -data for the curl in a variable in bash, but if using other scripting languages this could have been done in much less code, example is for 2 monitors
monitor_id=$(curl -L -X GET "https://api.datadoghq.com/api/v1/monitor" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DATADOG_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" \
--data-raw '{"tags":["application_id:'$APP_ID_LOWERCASE'"]}'| jq -r ' .[] | select((.name |endswith("XXXXXXX Heartbeats") or endswith("XXXXX HeartBeat monitoring")) and (.tags[]=="application_id:'$APP_ID_LOWERCASE'")) | .id')
# curl gets cluster name used in queries
CLUSTER_NAME=$(curl -L -X GET "https://api.datadoghq.com/api/v1/monitor" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DATADOG_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" \
--data-raw '{"tags":["application_id:'$APP_ID_LOWERCASE'"]}'| jq -r ' .[] | select((.name |endswith("XXXX HeartBeat monitoring")) and (.tags[]=="application_id:'$APP_ID_LOWERCASE'")) | .query' | awk -F',cluster_name:|,' '{print $2}')
# For each monitor id edit monitor query
while IFS= read -r monitors
do
# curl gets monitor name
monitor_name=$(curl -L -X GET "https://api.datadoghq.com/api/v1/monitor/"$monitors"" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "DD-API-KEY: ${DATADOG_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" | jq -r .name)
# checking which monitor query to send based on name, the queries are hardcoded because I couldn't find a way to set the query as a variable in bash
if [[ $monitor_name == *"XXXXXX HeartBeat monitoring"* ]]; then
shutdown_query=$(curl -L -X PUT "https://api.datadoghq.com/api/v1/monitor/"$monitors"" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DATADOG_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" \
--data-raw '{"query":"sum(last_1m):avg:health{application.health:healthy,cluster_name:'${CLUSTER_NAME}',!source:XXXXXX-full-1,source:DUMMY-VALUE-TO-RESEt-QUERY} by {source}.as_count() < 60"}')
elif
[[ $monitor_name == *"XXXXXX Instance Heartbeats"* ]]; then
shutdown_query=$(curl -L -X PUT "https://api.datadoghq.com/api/v1/monitor/"$monitors"" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DATADOG_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" \
--data-raw '{"query":"sum(last_1m):avg:heartbeat{cluster_name:'${CLUSTER_NAME}',source:DUMMY-VALUE-TO-RESEt-QUERY} by {source}.as_count() < 3"}')
fi
done <<< "$monitor_id"
Then just remove the dummy query value on startup and it will pick up your new monitors while forgetting the non existing ones