Prometheus using multiple target - kubernetes

Question

We need to monitor several of target with prometheus, when we have a short list of targets
it was not a problem to modify, however we need to add many targets (50-70 new targets) from diffrent clusters
My question if there is a more elegant way to achieve this
instead of using it like this
- job_name: blackbox-http # To get metrics about the exporter’s targets
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- http://clusterA
- https://clusterA
- http://clusterB
- http://clusterC
- http://clusterC
...
maybe to mount additional files for each cluster , I mean to provide a file with targets for clusterA only and new file for clusterB only etc, is it possible ?
And the same for jobs, mount each job from a file

Answer 1

When you have a growing or variable list of targets the best way of managing the job definition is to use SRV records instead of static_configs.
With SRV records you only need to define a dns_sd_config with only one target that will be resolved using a DNS query, then you don't need to change the configuration every time you add a new target only add it on the DNS record
An example from the documentation here adapted to your question:
- job_name: 'myjob'
metrics_path: /probe
params:
module: [http_2xx]
dns_sd_configs:
- names:
- 'telemetry.http.srv.example.org'
- 'telemetry.https.api.srv.example.org'
You can use an internal DNS service to generate those records, and if you have targets with http and https mixed you probably need to have two records because the SRV record defines the port to use.