I have the following Prometheus query and want the right side of my substraction to only subtract the container_memory_usage_bytes for each node on the left side of the substraction.
Query:
sum(kube_node_status_capacity_memory_bytes) by (node) - on (instance) group_left container_memory_usage_bytes
The left side of the subtraction works.
But for the whole query I get
Error executing query: found duplicate series for the match group;many-to-many matching not allowed: matching labels must be unique on one side
How can I make my query to match exactly the labels node == instance from left to right side?
The RHS has no instance label, so it's trying to match all those series to one on the LHS.
You have to also add sum before listing container_memory_usage_bytes metric, - sum(container_memory_usage_bytes).
Please take a look here: prometheus-metrics.
Useful article: label-proportions.
Related
In a Grafana dashboard panel, I have two queries:
rate(container_cpu_usage_seconds_total{cloud=~"${cloud}",environment=~"${environment}",location=~"${location}",container="tlm-telemetry-service",namespace="tlm"}[5m])
and
avg(kube_pod_container_resource_limits_cpu_cores{namespace="tlm"})
which can be shown well separately:
But when I try to use the "avg" one to divide the "rate" one:
rate(container_cpu_usage_seconds_total{cloud=~"${cloud}",environment=~"${environment}",location=~"${location}",container="tlm-telemetry-service",namespace="tlm"}[5m])/avg(kube_pod_container_resource_limits_cpu_cores{namespace="tlm"})
the result shows "no data".
This query used to work in Grafana Version 6.7.5, but when I tried to move it to Grafana Cloud, this problem happened.
Any one has any idea?
This throws a No data because you have two different expression data types. On the left side, you have a range vector and the right side scalar type. The query should return the same LabelSet in each part, so you need to group the right side also with the by clause.
E.g. the following query should work:
sum by (namespace,container) (rate(container_cpu_usage_seconds_total{cloud=~"${cloud}",environment=~"${environment}",location=~"${location}",container="tlm-telemetry-service",namespace="tlm"}[5m])) /
avg by (namespace,container) (kube_pod_container_resource_limits_cpu_cores{namespace="tlm"})
I am using Grafana together with Prometheus to display data of my Pods from Kubernetes Cluster. Here I am displaying Memory Usage for each Pod by name:
sum (container_memory_working_set_bytes{namespace="namespace1", image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^$Node$"}) by (pod_name)
It gives correct result for each pod. In example:
namespace1-eventstore-1
namespace1-eventstore-0
avsandbox-X-64ff4d-rl9z6
avsandbox-X-64ff4d-ldfnx
avsandbox-Y-7d9df9ddff-asdf
avsandbox-Y-7d9df9ddff-dfas
avsandbox-Z-5957dbaf58dt-gds24
avsandbox-Z-5957dbaf58dt-g4gd7
Now I want to sum them by their respective names to receive following result or closest I can get to it
namespace1-eventstore
avsandbox-X
avsandbox-Y
avsandbox-Z
So in conclusion I want to sum everything that has same name before second -. How can I achieve that?
Edit.: Here's further example what I'm looking for (hopefully it's helpful to give practical example and general idea)
sum (container_memory_working_set_bytes{namespace="namespace1", image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^$Node$"}) by (pod_name="([a-zA-Z0-9]+-[a-zA-Z0-9])-.*")
But that's not possible because of syntax.
We have a situation where I want to add a MIB variable label to another query. This another query gives me the value result that I want but I need to add the label from first variable in order to then sort them by what I want (for example just as we do it with instance label).
E.g.
variable1{alert,env, index, instance, ..., labelneeded}
variable2{alert,env, index, instance}
For example there I wanted to get the index of both and somehow add the label-needed by
I tried the following queries but they didnt work after they are giving me both variables with its labels but not concatenated together so my question is if there is a possibility to concatenate them together?
Query example:
max by {index, instance} (variable2 * 5) or max(variable2) by (labelneeded, index, instance)
Thank you in advance :).
In order to explain how to merge labels labels from two metrics, I'll take a common case:
a value metric value{instance="foo",a_label="bar"} 42
an info metric info{instance="foo",version="1.2.3",another="bar"} 1
Info metrics (such as version, compiler, ...) have value 1 such that you can apply operators between the metrics:
value * on(instance) group_left(version) info
Result
{instance="foo",a_label="bar",version="1.2.3"} 42
The parameter(s) of on() keyword specify the criteria(s) for matching the info and the parameter(s) of group_left() operator specify the labels to pull.
It can happens that the metric you want to pull from doesn't have value 1. In that case, you can use the bool modifier with a comparison always true to obtain 1:
other_metric{instance="foo",baz="void"} 0.5555
value * on(instance) group_left(baz) other_metric != bool NaN
Result
{instance="foo",a_label="bar",baz="void"} 42
Hy there,
I'm trying to configure Kubernetes Cronjobs monitoring & alerts with Prometheus. I found this helpful guide
But I always get a many-to-many matching not allowed: matching labels must be unique on one side error.
For example, this is the PromQL query which triggers this error:
max(
kube_job_status_start_time
* ON(job_name) GROUP_RIGHT()
kube_job_labels{label_cronjob!=""}
) BY (job_name, label_cronjob)
The queries by itself result in e.g. these metrics
kube_job_status_start_time:
kube_job_status_start_time{app="kube-state-metrics",chart="kube-state-metrics-0.12.1",heritage="Tiller",instance="REDACTED",job="kubernetes-service-endpoints",job_name="test-1546295400",kubernetes_name="kube-state-metrics",kubernetes_namespace="monitoring",kubernetes_node="REDACTED",namespace="test-develop",release="kube-state-metrics"}
kube_job_labels{label_cronjob!=""}:
kube_job_labels{app="kube-state-metrics",chart="kube-state-metrics-0.12.1",heritage="Tiller",instance="REDACTED",job="kubernetes-service-endpoints",job_name="test-1546295400",kubernetes_name="kube-state-metrics",kubernetes_namespace="monitoring",kubernetes_node="REDACTED",label_cronjob="test",label_environment="test-develop",namespace="test-develop",release="kube-state-metrics"}
Is there something I'm missing here? The same many-to-many error happens for every query I tried from the guide.
Even constructing it by myself from ground up resulted in the same error.
Hope you can help me out here :)
In my case I don't get this extra label from Prometheus when installed via helm (stable/prometheus-operator).
You need to configure it in Prometheus. It calls: honor_labels: false
# If honor_labels is set to "false", label conflicts are resolved by renaming
# conflicting labels in the scraped data to "exported_<original-label>" (for
# example "exported_instance", "exported_job") and then attaching server-side
# labels.
So you have to configure your prometheus.yaml file - config with option honor_labels: false
# Setting honor_labels to "true" is useful for use cases such as federation and
# scraping the Pushgateway, where all labels specified in the target should be
# preserved
Anyway if I have it like this (I have now exported_jobs), still can't do proper query, but I guess is still because of my LHS.
Error executing query: found duplicate series for the match group
{exported_job="kube-state-metrics"} on the left hand-side of the operation:
[{__name__=
I ran into the same issue when I followed that article, but for me, I actually get duplicate job names but in different namespaces.
Ex. When running kube_job_status_start_time:
kube_job_status_start_time{instance="REDACTED",job="kube-state-metrics",job_name="job-abc-123",namespace="us"}
kube_job_status_start_time{instance="REDACTED",job="kube-state-metrics",job_name="job-abc-123",namespace="ca"}
So I had to either add a filter for the namespace or add namespace into the ON/BY clauses to get it to be unique.
e.g. for one of the subqueries I had to do this:
max(
kube_job_status_start_time
* ON(namespace, job_name) GROUP_RIGHT()
kube_job_labels{label_cronjob!=""}
) BY (namespace, label_cronjob)
Essentially had to apply that principle to all the rest of the queries for it to work for me. Not sure if that applies in your case.
Replacing kube_job_status_start_time with max(kube_job_status_start_time) by (job_name) will aggregate out any duplicates and should resolve the error.
The resulting query will look like this
max(
max(kube_job_status_start_time) by (job_name)
* ON(job_name) GROUP_RIGHT()
kube_job_labels{label_cronjob!=""}
) BY (job_name, label_cronjob)
I dug into this issue a bit more, and I guess the root cause of it is within this one-to-many vector matching expression:
kube_job_status_start_time * ON(job_name) GROUP_RIGHT() kube_job_labels{label_cronjob!=""}
where the group modifier "GROUP_RIGHT()" suggests, that each vector element from the left side (kube_job_status_start_time) can match with multiple elements on the right side (kube_job_labels), based on common label (job_name). The thing is that we are really dealing here with many-to-many matching, as each vector element from right side can match also multiple elements from left vector as well:
I think that what we are missing here is the way to uniquely identify exported Job objects from K8S by Prometheus. The author of this blog post, mentions about this feature in his setup:
...Prometheus resolves this collision of label names by including the
raw metric’s label as an exported_job label...
In my case I don't get this extra label from Prometheus when installed via helm (stable/prometheus-operator).
Regarding the missing labels - make sure that your kube-state-metrics is configured with a --metric-labels-allowlist. This is "new" since kube-state-metrics v2. See https://kubernetes.io/blog/2021/04/13/kube-state-metrics-v-2-0/#what-is-new-in-v2-0
By default, the metric contains only name and namespace labels.
But... the original guide is not woking with newer kube-state-metrics anyway. I can recommend this guide, which is a rework and does not need the labels.
I have two metrics (relating to memory usage in my Kubernetes pods) defined as follows:
kube_pod_container_resource_limits_memory_bytes{app="kube-state-metrics",container="foo",instance="10.244.0.7:8080",job="kubernetes-endpoints",kubernetes_name="kube-state-metrics",kubernetes_namespace="monitoring",namespace="test",node="aks-nodepool1-25518080-0",pod="foo-cb9bc5fb5-2bghz"}
container_memory_working_set_bytes{agentpool="nodepool1",beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="Standard_A2",beta_kubernetes_io_os="linux",container_name="foo",failure_domain_beta_kubernetes_io_region="westeurope",failure_domain_beta_kubernetes_io_zone="1",id="/kubepods/burstable/pod5b0099a9-eeff-11e8-884b-ca2011a99774/eeb183b21e2b3226a32de41dd85d7a2e9fc8715cf31ea7109bfbb2cae7c00c44",image="#sha256:6d6003ba86a0b7f74f512b08768093b4c098e825bd7850db66d11f66bc384870",instance="aks-nodepool1-25518080-0",job="kubernetes-cadvisor",kubernetes_azure_com_cluster="MC_test.planned.bthbygg.se_bthbygg-test_westeurope",kubernetes_io_hostname="aks-nodepool1-25518080-0",kubernetes_io_role="agent",name="k8s_foo_foo-cb9bc5fb5-2bghz_test_5b0099a9-eeff-11e8-884b-ca2011a99774_0",namespace="test",pod_name="foo-cb9bc5fb5-2bghz",storageprofile="managed",storagetier="Standard_LRS"}
I want to combine these two into a percentage, by doing something like
container_memory_working_set_bytes{namespace="test"}
/ kube_pod_container_resource_limits_memory_bytes{namespace="test"}
but that gives me no data back, presumably because there are no matching labels to join the data sets on. As you can see, I do have matching label values, but the label names don't match.
Is there somehow I can formulate my query to join these on e.g. pod == pod_name, without having to change the metrics at the other end (where they are exported)?
You can use PromQL label_replace function to create a new matching label from the original labels.
For instance, you can use the below expression to add a container_name="foo" label to the first metric which can be used to do the join:
label_replace(
kube_pod_container_resource_limits_memory_bytes,
"container_name", "$1", "container", "(.*)")
You can use the above patern to create new labels that can be used for the matching.