Graphite - Gather metrics only from active service instances - grafana

Question

Lets say my spring microservice processes data. Every time a successful processing event occurs, for metrics, I update the micrometer counter. This is registered to a Graphite Registry.
registry = new GraphiteMeterRegistry(new GraphiteConfiguration(), Clock.SYSTEM, HierarchicalNameMapper.DEFAULT);
Counter counter = Counter.builder("process").tag("status","success").register(registry);
So far, it sounds good. But what if I have to create and deploy multiple instances of my service?
How do I get the aggregated count of all successful events from all the instances?
To illustrate my case further, I log the counter.count() value on each increment. Here is what i see ->
<Instance 1> <time> <package-name> Count :122
<Instance 2> <time> <package-name> Count :53
So when I run the graphite query on graphana -
process.status.success.count
I tend to get the random count from either of these instances.
What I need is a query like -
process.service-instance.status.success.count
so that I can run a summarize() function in the end.
Update
Now I'm able to source data from all instances by getting the service instance ID. But that presents a new problem - Since I restart my services time and again, and my service-id changes every time, how do I source data from ONLY ACTIVE services?
Since process.*.status.success.count represents aggregate count of ALL services - dead or alive

Answer 1

Never use instance ids for aggregation. When instances restart, instance ids will change. (Use instance-id for logging/debugging/record-keeping purpose only.)
Use service-id for aggregation.
For micrometer, you can add service-name in common tags.
registry.config().commonTags("service", "xyz-service");
Common tags are defined at registry level and every metric associated with that registry will have common tags added to it.
And, for dead or alive situation: The metric was pushed when the instance was alive. So if you want to know how many times some step ran, you'll need to consider that count.
To source data from active instances, use time-filter. That will return data pushed by instances that were alive in that duration (Why? Because dead instances do not push metrics).