I am wondering how Prometheus behaves, if there are multiple instances of a service available.
For example there is one service which is deployed in a kubernetes cluster with three instances running.
Each instance increases its count-metric.
What happens when prometheus scrapes these instances and receives the three values?
Does prometheus sum them up?Does the last of the three values to be scraped override the first two?
Prometheus will ingest the three values from the three targets, which (unless you've done something very weird) will then exist independently.
No math will be performed on them other than math you request in PromQL.
If you want, you can sum the values of multiple instances.
If you set a rule, you can sum by application.
An example for an alert, would be:
groups:
- name: default
rules:
- alert: defaultAlert
expr: sum by (application) (expression_that_you_want)
The relevant thing here is the: sum by (application)
You can sum by others labels as well.
Related
I have a machine learning model inside a docker image. I pushed the docker image to google container registry and then deploy it inside a Kubernetes pod. There is a fastapi application that runs on Port 8000 and this Fastapi endpoint is public
(call it mymodel:8000).
The structure of fastapi is :
app.get("/homepage")
asynd def get_homepage()
app.get("/model):
aysnc def get_modelpage()
app.post("/model"):
async def get_results(query: Form(...))
User can put query and submit them and get results from the machine learning model running inside the docker. I want to limit the number of times a query can be made by all the users combined. So if the query limit is 100, all the users combined can make only 100 queries in total.
I thought of a way to do this:
Store a database that stores the number of times GET and POST method has been called. As soon as the total number of times POST has been called crosses the limit, stop accepting any more queries.
Is there an alternative way of doing this using Kubernetes limits? Such as I can define a limit_api_calls such that the total number of times mymodel:8000 is accessed is at max equal to limit_api_calls.
I looked at the documentation and I could only find setting limits for CPUs, Memory and rateLimits.
There are several approaches that could satisfy your needs.
Custom implementation: As you mentioned, keep in a persistence layer the number of API calls received and deny requests after it has been reached.
Use a service mesh: Istio (for instance) will let you limit the number of requests received and act as a circuit breaker.
Use an external Api Manager: Apigee will also let you limit and even charge your users, however if it is only for internal use (not pay per use) I definitely won't recommend it.
The tricky part is what you want to happen after the limit has been reached, if it is just a pod you may exit the application to finish and clear it.
Otherwise, if you have a deployment with its replica set and several resources associated with it (like configmaps), you probably want to use some kind of asynchronous alert or polling check to clean up everything related to your deployment. You may want to have a deep look at orchestrators like Airflow (Composer) and use several tools such as Helm for keeping deployments easy.
What is the simplest way to find out the Availability of a K8s service over a period of time, lets say 24h. Should I target a pod or find a way to calculate service reachability
I'd recommend to not approach it from a binary (is it up or down) but from a "how long does it take to serve requests" perspective. In other words, phrase your availability in terms of SLOs. You can get a very nice automatically generated SLO-based alter rules from PromTools. One concrete example rule from there, showing the PromQL part:
1 - (
sum(rate(http_request_duration_seconds_bucket{job="prometheus",le="0.10000000000000001",code!~"5.."}[30m]))
/
sum(rate(http_request_duration_seconds_count{job="prometheus"}[30m]))
)
Above captures the ratio of how long it took the service to serve non-500 (non-server errors, that is, assumed good responses) in less than 100ms to overall responses over the last 30 min with http_request_duration_seconds being the histogram, capturing the distribution of the requests of your service.
I have 2 microservices - service A and service B. For service A there is one pod running, and for service B there are two pods. Service A publishes messages on an ActiveMQ, and service B consumes it.
There are two pods for service B so I want to have a generic filter so that pod 1 of Service B consumes the message based on the given generic filter and pod 2 consumes the messages based on the given generic filter.
For Example, there is a userId in the payload and if the userId is even then I want pod 1 to consume and userId is odd then I want pod 2 to consume.
Can any one let me know that how to do this.
I understand how you could have a use case for this scenario. You can use a destinationInterceptor to perform a broker-side filter using a selector. I don't generally recommend this, since it generally requires a broker restart to make changes to the filtering rules.
If load traffic load is medium (less than 10M messages per day and/or less ~300 msg/s peak) you can readily use a Camel route to filter traffic using a content-based-router pattern (CBR) and support making changes to the filter rules (ie.. a new userId is added or one is removed).
producer -> queue://PRE.SORT -> Camel route CBR consumes
Then the Camel route doing CBR produces messages in the queue(s) based on the userId values
queue://PODA.IDS
queue://PODB.IDS
etc..
NOTE: Camel is very capable of handling over 10M messages a day and 300 msg/s. I'm providing these 'medium' sizing metrics as a guideline to where you can set it up quickly and not worry too much about performance tuning, etc.
I don't know if this is a bug/issue or question.
Proposal
Use case. Why is this important?
For monitoring multiple containers with resource/requests limits in pods on kubernetes.
Bug Report
What did you do?
I'm write a query to get a percentage of usage based on the maximum CPU usage and that we have max on the limits (resource and request) of the pod.
We have this problem affecting our query:
1. When we take a pod and it have 2 containers with configured resource/requets limits, it is not possible to take the value of resource/requests limits. 2. Show the value of the pod (resource/requests), but it can have multiple replicas.
max_over_time(sum(rate(container_cpu_usage_seconds_total{namespace="alpha",container_name!="POD", container_name!=""}[1m])) [1h:1s]) / on(pod) kube_pod_container_resource_requests_cpu_cores{namespace="alpha"}
Error executing query:found duplicate series for the match group {pod="heapster-65ddcb7b4c-vtl8j"} on the right hand-side of the operation: [{__name__="kube_pod_container_resource_requests_cpu_cores", container="heapster-nanny", instance="kubestate-alpha.internal:80", job="k8s-prod-http", namespace="alpha", node="ip-99-990-0-99.sa-east-1.compute.internal", pod="heapster-65ddcb7b4c-vtl8j"}, {__name__="kube_pod_container_resource_requests_cpu_cores", container="heapster", instance="kubestate-alpha.internal:80", job="k8s-alpha-http", namespace="alpha", node="ip-99-990-0-99.sa-east-1.compute.internal", pod="heapster-65ddcb7b4c-vtl8j"}];many-to-many matching not allowed: matching labels must be unique on one side.
We try solutions like: [Using group_left to calculate label proportions]
sum without (container) (rate(kube_pod_container_resource_requests_cpu_cores{pod="heapster-65ddcb7b4c-vtl8j"}[1m]))
But if the value is set in the container, the result of the query is 0.
For not being able to calculate.
kube_pod_container_resource_requests_cpu_cores{pod="heapster-65ddcb7b4c-vtl8j"}
kube_pod_container_resource_requests_cpu_cores{container="heapster", instance="kubestate-alpha.internal:80", job="k8s-alpha-http", namespace="alpha", node="ip-99-990-0-99.sa-east-1.compute.internal", pod="heapster-65ddcb7b4c-vtl8j"} 0.084
kube_pod_container_resource_requests_cpu_cores{container="heapster-nanny", instance="kubestate-alpha.internal:80", job="k8s-alpha-http", namespace="alpha", node="ip-99-990-0-99.sa-east-1.compute.internal", pod="heapster-65ddcb7b4c-vtl8j"} 0.05
Standard output for the kube_pod_container_resource_requests_cpu_cores command
What did you expect to see?
The sum of what is set in the containers in the pod.
What did you see instead? Under which circumstances?
Prometheus UI
Environment
System information:
Linux 4.4.0-1096-aws x86_64
Prometheus version:
v2.15.2
We are using Kubernetes and we need to do "Smart partitioning" of data. We want to split the space between 1 to 1000 between n running Pods,
And each pod should know which part of the space is his to handle (for pooling partitioned tasks).
So for example, if we have 1 pod he will handle the whole space from 1-1000.
When we scale out to 3 pods, each of them will get the same share.
Pod 1 - will handle 1-333
Pod 2 - 334-667
Pod 3 667-1000
Right now the best way that we find to handle this issue is to create a Stateful-set, that pooling the number of running pods and his instance number and decide which part of the space he needs to handle.
Is there a smarter/built-in way in Kubernetes to partition the space between nodes in this manner?
Service fabric has this feature built-in.
There are NO native tools for scaling at the partition level in K8s yet.
Only custom solutions similar to what you have came up with in your original post.
Provide another customized way for doing this for your reference. Based on this tech blog of Airbnb
Given the list of pods and their names, each pod is able to
deterministically calculate a list of partitions that it should work
on. When we add or remove pods from the ReplicaSet, the pods will
simply pick up the change, and work on the new set of partitions
instead
How do they do is based on the their repo. I summarized the key components here (Note: the repo is written in Java).
Get how many pods running in the k8s namespace, and sort by pod name (code). Code snippet
String podName = System.getenv("K8S_POD_NAME");
String namespace = System.getenv("K8S_NAMESPACE");
NamespacedKubernetesClient namespacedClient = kubernetesClient.inNamespace(namespace);
ReplicaSet replicaSet;
// see above code link to know how to get activePods, remove it here because it is too long
int podIndex = activePods.indexOf(podName);
int numPods = activePods.size();
Every time you call the above code, you will have deterministic list of podIndex and numPods. Then, using this information to calculate the range this pod is responsible for
List<Integer> partitions = new ArrayList<>();
int split = spaceRange / numPods;
int start = podIndex * split;
int end = (podIndex == numPods - 1) ? spaceRange - 1 : ((podIndex + 1) * split) - 1;
for (int i = start; i <= end; i++) {
partitions.add(i);
}
Since the number of pods will be changed anytime, you may need a executorService.scheduleWithFixedDelay to periodically update the list as here
executorService.scheduleWithFixedDelay(this::updatePartitions, 0, 30, TimeUnit.SECONDS);
This approach is not the best, since if you set scheduleWithFixedDelay with 30 seconds, any pod change won't be captured within 30 seconds. Also, it is possible in a short period of time, two pods may be responsible for the same space, and you need to handle this special case in your business logics as Airbnb tech blog does.