How does one add a where clause in PromQL?
I'm trying to construct a query that displays when an application running in Kubernetes has been up for more than one minute but I want to filter by namespace.
This is what my query looks like at the moment
100 * (count(up == 1) BY (job, namespace, service) ) > 1
This works fine but it gives me additional information that I don't need.
{job="prometheus-grafana", namespace="monitor", service="prometheus-grafana"}
{job="jenkins", namespace="jenkins", service="jenkins"}
{job="kube-state-metrics", namespace="monitor", service="prometheus-kube-state-metrics"}
{job="node-exporter", namespace="monitor", service="prometheus-prometheus-node-exporter"}
{job="kubelet", namespace="kube-system", service="prometheus-kube-prometheus-kubelet"}
{job="apiserver", namespace="default", service="kubernetes"}
What I'm trying to accomplish is to get results for only the jenkins and default namespace.
{job="apiserver", namespace="default", service="kubernetes"}
{job="jenkins", namespace="jenkins", service="jenkins"}
I've tried doing
100 * (count(up == 1) BY (job, namespace, service) ) > 1 and ON {namespace="jenkins"}
But I get an invalid parameter "query": 1:65: parse error: unexpected "{" in grouping opts, expected "(" error.
You would have to filter the metric "up" by the labels you want (namespaces) in your case it should look something like this:
100 * count(up{namespace=~"default|jenkins"} == 1) > 1
You can try this too. In Kubernetes all resources uses pod. So if you take pod status metrics and minus current time with 60, which gives post 1 min pods running status.
time()-60 > (kube_pod_start_time)
Prometheus provides the following ways for filtering the data in queries:
Time series selectors. They allow filtering time series by metrics and labels. For example, up{namespace=~"default|jenkins"} is a series selector, which returns only time series with the name up, which contain label namespace matching the given given regular expression: default|jenkins. This is roughly equivalent to the following SQL:
SELECT * FROM table WHERE name = 'up' and namespace ~ '^(default|jenkins)$'
Comparison operators, which allow filtering time series by values. For example, up == 0 returns time series with up name, which have 0 value. This is roughly equivalent to the following SQL:
SELECT * FROM table WHERE name = 'up' and value == 0
Time series matching via binary operators. This allows performing join-like queries. For example, up * on(instance) group_left(name) node_os_info joins up metric with node_os_info metric via instance label and selects additional name label from node_os_info metric. This is roughly equivalent to the following SQL:
SELECT up.*, node_os_info.name
FROM up LEFT JOIN node_os_info ON (instance)
Related
Prometheus has metrics such as container_cpu_usage_seconds_total. However, they are only grouped by pod. How can I group them by deployment/cronjobs/etc?
I was able to handle this with the following query:
((label_replace((rate(container_cpu_usage_seconds_total{image!=""}[2m]) * on(pod) group_left(owner_name) (sum without (instance) (kube_pod_owner))), "replicaset", "$1", "owner_name", "(.*)")) * on(replicaset) group_left(owner_name) (sum without (instance) (kube_replicaset_owner{})))
Here is the explanation:
Join container_cpu_usage_seconds_total with kube_pod_owner on pod
Copy over the owner_name from kube_pod_owner
Use label_replace to rename kube_pod_owner's owner_name to replicaset
Join that with kube_replicaset_owner on replicaset
Copy over the owner_name from kube_replicaset_owner (this value is your deployment etc)
The without (instance) are used to remove the instance field from the joined sets. Because there can be multiple instances for a single deployment, this can cause issues.
Lastly, the rate function is called on container_cpu_usage_seconds_total directly at the innermost area because otherwise Prometheus complains about parse error: ranges only allowed for vector selectors. Placing it in the innermost area is a workaround.
In a Microstrategy report is there a way to create an IF statement off of a condition on the attribute not a metric so that it looks like the table below? If(Unique ID Attribute = "Test1", 0, 1)
Unique_ID Attribute
Metric Output from If Statement
Test1
0
Test2
1
Test3
1
You have to convert the attribute to a metric first. You can use "Max([Unique_ID Attribute]){~+}" for this, and call it "MAX UNIQUE_ATTR"
With this metric you can then use your IF statement: IF(([MAX UNIQUE_ATTR] = "Test1"), 1, 0)
You don't have to use two metrics, you could write this inside a single metric too. Depends on whether you want to re-use said MAX-metric. Also think about subtotals/dynamic aggregation and what it should show there (sum/avg).
All the other options for metrics still apply, you can use filters, manually set the levels of the metric etc.
I've included a screenshot I did on a testproject.
I have the following query in pgsql to extract a tabular view that has the 3 columns - node, value and time. The column node is created and if I try to filter for a specific node, I get a message - "node is not a column in the table metrics" so do I need to use regex or a specific where node like xyz.
Query:
select label->>'machine' as node, value, time
from metrics
where metrics.name='total_bytes' and time<'09-22-2020'
group by node, metrics.value, metrics.time
order by node;`
Thanks for any help.
You can write the clause this way:
where label->>'machine' like 'xyz'
To use node directly, you could do this:
with my_metrics as (
select label->>'machine' as node, value, time
from metrics
where metrics.name='total_bytes' and time<'09-22-2020'
)
select node, value, time
from my_metrics
where node like 'xyz'
group by node, value, time
order by node;
We have a situation where I want to add a MIB variable label to another query. This another query gives me the value result that I want but I need to add the label from first variable in order to then sort them by what I want (for example just as we do it with instance label).
E.g.
variable1{alert,env, index, instance, ..., labelneeded}
variable2{alert,env, index, instance}
For example there I wanted to get the index of both and somehow add the label-needed by
I tried the following queries but they didnt work after they are giving me both variables with its labels but not concatenated together so my question is if there is a possibility to concatenate them together?
Query example:
max by {index, instance} (variable2 * 5) or max(variable2) by (labelneeded, index, instance)
Thank you in advance :).
In order to explain how to merge labels labels from two metrics, I'll take a common case:
a value metric value{instance="foo",a_label="bar"} 42
an info metric info{instance="foo",version="1.2.3",another="bar"} 1
Info metrics (such as version, compiler, ...) have value 1 such that you can apply operators between the metrics:
value * on(instance) group_left(version) info
Result
{instance="foo",a_label="bar",version="1.2.3"} 42
The parameter(s) of on() keyword specify the criteria(s) for matching the info and the parameter(s) of group_left() operator specify the labels to pull.
It can happens that the metric you want to pull from doesn't have value 1. In that case, you can use the bool modifier with a comparison always true to obtain 1:
other_metric{instance="foo",baz="void"} 0.5555
value * on(instance) group_left(baz) other_metric != bool NaN
Result
{instance="foo",a_label="bar",baz="void"} 42
I have two metrics (relating to memory usage in my Kubernetes pods) defined as follows:
kube_pod_container_resource_limits_memory_bytes{app="kube-state-metrics",container="foo",instance="10.244.0.7:8080",job="kubernetes-endpoints",kubernetes_name="kube-state-metrics",kubernetes_namespace="monitoring",namespace="test",node="aks-nodepool1-25518080-0",pod="foo-cb9bc5fb5-2bghz"}
container_memory_working_set_bytes{agentpool="nodepool1",beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="Standard_A2",beta_kubernetes_io_os="linux",container_name="foo",failure_domain_beta_kubernetes_io_region="westeurope",failure_domain_beta_kubernetes_io_zone="1",id="/kubepods/burstable/pod5b0099a9-eeff-11e8-884b-ca2011a99774/eeb183b21e2b3226a32de41dd85d7a2e9fc8715cf31ea7109bfbb2cae7c00c44",image="#sha256:6d6003ba86a0b7f74f512b08768093b4c098e825bd7850db66d11f66bc384870",instance="aks-nodepool1-25518080-0",job="kubernetes-cadvisor",kubernetes_azure_com_cluster="MC_test.planned.bthbygg.se_bthbygg-test_westeurope",kubernetes_io_hostname="aks-nodepool1-25518080-0",kubernetes_io_role="agent",name="k8s_foo_foo-cb9bc5fb5-2bghz_test_5b0099a9-eeff-11e8-884b-ca2011a99774_0",namespace="test",pod_name="foo-cb9bc5fb5-2bghz",storageprofile="managed",storagetier="Standard_LRS"}
I want to combine these two into a percentage, by doing something like
container_memory_working_set_bytes{namespace="test"}
/ kube_pod_container_resource_limits_memory_bytes{namespace="test"}
but that gives me no data back, presumably because there are no matching labels to join the data sets on. As you can see, I do have matching label values, but the label names don't match.
Is there somehow I can formulate my query to join these on e.g. pod == pod_name, without having to change the metrics at the other end (where they are exported)?
You can use PromQL label_replace function to create a new matching label from the original labels.
For instance, you can use the below expression to add a container_name="foo" label to the first metric which can be used to do the join:
label_replace(
kube_pod_container_resource_limits_memory_bytes,
"container_name", "$1", "container", "(.*)")
You can use the above patern to create new labels that can be used for the matching.