Prometheus DNS Service Discovery with ECS [closed] - amazon-ecs

Question

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 2 years ago.
Improve this question
I'm trying to get ECS Service Discovery working with Prometheus.
Currently my ECS container gets added to Route 53 like so:
+-----------------------------------------------+------+--------------------------------------------------------+
| Name | Type | Value |
+-----------------------------------------------+------+--------------------------------------------------------+
| my-service.local. | SRV | 1 1 8080 123456-7890-1234-5678-12345.my-service.local. |
| 123456-7890-1234-5678-12345.my-service.local. | A | 10.0.11.111 |
+-----------------------------------------------+------+--------------------------------------------------------+
I assume if I added more running containers to ECS, I would get more Alias records in Route 53 with the name 123456-7890-1234-5678-12345.my-service.local.
In my Prometheus configuration file, I have supplied the following under scrape_config:
- job_name: 'cadvisor'
scrape_interval: 5s
dns_sd_configs:
- names:
- 'my-service.local'
type: 'SRV'
However, when I check the target status in Prometheus, I see the following:
Endpoint: http://123456-7890-1234-5678-12345.my-service.local:8080/metrics
State: Down
Error: context deadline exceeded
I'm not familiar with how DNS Service Discovery works with SRV records so I'm not sure where the problem lies exactly. Looking at how AWS ECS Service Discovery added the records, it looks like my-service.local maps to 123456-7890-1234-5678-12345.my-service.local:8080
However it looks like Prometheus doesn't then try to find the list of local IPs mapped to 123456-7890-1234-5678-12345.my-service.local and just tries to scrape from it directly.
Is there some configuration option that I'm missing to make this work or have I misunderstood something at a fundamental level?

Answer 1

Turns out the issue was that I needed to add a security group rule to allow my Prometheus instance to talk to my ECS cluster since both were in a public subnet.
Also scaling the desired count in the ECS cluster up creates both another SRV record and an associated A record in Route 53 (not just one additional A record as I previously thought).
Everything seems to work now.

Answer 2

A fairly good alternative to using a "proper" service discovery like Consul or ECS SD with Route 53 is relying on the AWS API. This is appropriate as long as the total number of containers / tasks stays below a few thousand, since you are limited by the AWS API request cap.
There exist a number of tools that provide this functionality in combination with Prometheus file discovery. For example https://pypi.org/project/prometheus-ecs-discoverer/ or https://github.com/teralytics/prometheus-ecs-discovery