I'm creating a panel to show the error count in logs for canary instances. First, I need to find whether the instance is canary or not. So, if the instance is canary then I have to show the error log count for that instance.
To filter the canary instance - I have stack label so if the stack contains one instance then it should be a canary instance.
The expression should check the instance count of each and every stack, if a stack has one instance then it needs to search for the keyword in the log.
How do achieve this? I am looking for an expression something like below.
sum(count_over_time({component="stack-blue.*" ,cloud=~"${cloud}" ,environment=~"${environment}" ,location=~"${location}" } |= "Unable to record" [$__interval]))
and
(count(count by(hostname)(count_over_time({component="stack-blue.*",cloud=~"${cloud}" ,environment=~"${environment}" ,location=~"${location}" } [$__interval]))) == 1)
You can combine two separate queries A and B with a Math expression $A && $B.
Note that you can decide whether a query or expression is displayed in the panel by clicking on the eye symbol.
Related
I want to have a panel in Grafana which displays what pods are currently running in a host.
For the host variable I have the following query (the job variable is just label_values(node_uname_info, job).):
label_values(node_uname_info{job="$job"}, instance)
This gives me an array of sockets: host_ip:port
I can get the pod names from kube_pod_info{job="$job", host_ip="$host_ip"}, but in order to get the IP I need to remove the port part of the socket:
label_replace(node_uname_info{job="$job", instance="$node"}, "host_ip", "$1", "instance", "(.*):.*")
I haven't found how to use the new host_ip label in the pod query to eventually get all the pod label values of kube_pod_info. I don't want to put the label_replace in Prometheus to avoid data duplication - is there a way to use the new host_ip label in the pod query?
Edit:
I added the host_ip variable with the regex as shan1024 showed in his answer and changed the panel's query to:
sum by (pod) (kube_pod_info{job="$job", host_ip="$host_ip"})
Then I changed the panel's visualization to table and added column styles to Time and Value (chose type Hidden). This allows me to display the host's running pods in a list-like fashion.
This is actually quite easy to do in Grafana and no need to change labels in Prometheus. You just need to add a regex in the instance variable (when we add a regex with a capturing group, the value(s) of the 1st captured group will be the value(s) of the variable).
e.g.
Variable definition without Regex (you get host_ip:port)-
Variable definition with Regex (you only get host_ip)-
Then you can add a new variable with value kube_pod_info{ host_ip="$instance" } to get all pods in the selected host.
AKS = 1.17.9
Prometheus = 2.16.0
kube-state-metrics = 1.8.0
My use case: I want to alert when 1 of my persistent volumes are not in a "Bound" phase and only when this falls within a predefined set of namespaces.
This got me to my first attempt at joining Prometheus metrics - so, please bear with me : )
I opted to use the following to obtain the pv phase:
kube_persistentvolume_status_phase{phase="Bound",job="kube-state-metrics"}
Renders:
kube_persistentvolume_status_phase{instance="10.147.5.110:8080",job="kube-state-metrics",persistentvolume="pvc-33197ae6-d42a-777e-b8ca-efbd66a8750d",phase="Bound"} 1
kube_persistentvolume_status_phase{instance="10.147.5.110:8080",job="kube-state-metrics",persistentvolume="pvc-165d5006-erd4-481e-8acc-eed4a04a3bce",phase="Bound"} 1
This worked well, except for the fact that it does not include the namespace.
So I managed to determine the persistentvolumeclaim namespaces with this:
kube_persistentvolumeclaim_info{namespace=~"monitoring|vault"}
Renders:
kube_persistentvolumeclaim_info{instance="10.147.5.110:8080",job="kube-state-metrics",namespace="vault",persistentvolumeclaim="vault-file",storageclass="default",volumename="pvc-33197ae6-d42a-777e-b8ca-efbd66a8750d"} 1
kube_persistentvolumeclaim_info{instance="10.147.5.110:8080",job="kube-state-metrics",namespace="monitoring",persistentvolumeclaim="prometheus-prometheus-db-prometheus-prometheus-0",storageclass="default",volumename="pvc-165d5006-erd4-481e-8acc-eed4a04a3bce"} 1
So my idea was to join these sets with the matching values in the following fields:
(kube_persistentvolume_status_phase)persistentvolume
on
(kube_persistentvolumeclaim_info)volumename
BUT, if I understood it correctly you are only able to join two metrics sets on labels that match exactly (text and their values). I hence opted for the "instance" and "job" labels as these were common on both sides and matching.
kube_persistentvolume_status_phase{phase!="Bound",job="kube-state-metrics"} * on(instance,job) group_left(namespace) kube_persistentvolumeclaim_info{namespace=~"monitoring|vault"}
Renders:
Error executing query: found duplicate series for the match group {instance="10.147.5.110:8080" , job="kube-state-metrics"} on the right hand-side of the operation: [{__name__="kube_persistentvolumeclaim_info", instance="10.147.5.110:8080", job="kube-state-metrics", namespace="monitoring", persistentvolumeclaim="alertmanager-prometheusam-db-alertmanager-prometheusam-0", storageclass="default", volumename="pvc-b8406fb8-3262-7777-8da8-151815e05d75"}, {__name__="kube_persistentvolumeclaim_info", instance="10.147.5.110:8080", job="kube-state-metrics", namespace="vault", persistentvolumeclaim="vault-file", storageclass="default", volumename="pvc-33197ae6-d42a-777e-b8ca-efbd66a8750d"}];many-to-many matching not allowed: matching labels must be unique on one side
So in all fairness, the query does communicate well on what the problem is - so I attempted to solve this with the "ignoring" option - attempting to keep only the matching labels and values (instance and job) and "excluding/ignoring" the non-matching ones on both sides. This did not work either - resulting in a parsing error. Which in turn nudged me to take a step back and reassess what I am doing.
I am just a bit concerned that I am perhaps barking up the wrong tree here.
My question is: Is this at all possible and if so how? or is there perhaps another, more prudent way to achieve this?
Thanks in advance!
An Icinga2 plugin (written by myself) returns performance data with metrics named /var(avail_MB), /var(total_MB) and similar. Data is forwarded to an InfluxDB with Grafana as Frontend.
I'm using "GROUP BY" "tag(metric)" and "ALIAS BY" "$tag_metric" in a dashboard's panel query.
The metric names are displayed correctly below the graph then.
However when I try to override series by specifying "alias or regex" /var(avail_MB) it does not seem to work, and when going back from panel configuration to dashboard, I get an error message saying "Panel rendering error '/var(avail_MB)' is not a valid regular expression.".
I tried to put a backslash in front of ( and ), but that didn't help.
To make matters worse, the whole graph disappeared, and when trying to open the "Query Inspector", the frontend seems to take forever (Query never appears).
What is the problem, and how could I fix it?
I'm new to Icinga2, Grafana and InfluxDB (I'm just a "user" not administrator of those).
The color change is not applied to the graph.
Here is an example of plugin output:
OK: /var: 3114/5632MB (55.30%), slope is NaN|/var(total_MB)=5631.56MB;;;0 /var(avail_pct)=55.30%;25;5;0;100 /var(avail_MB)=3114.12MB;10;5;0;5632 /var(est_avail_MB)=nanMB;10;5;0;5632
(The "nanMB" was a bug in the plugin that has been fixed already, but that data wasn't from the machine in question.)
The problems seems to be the beginning of the string ("/var").
Grafana seems to treat every string starting with / as regular expression, and it expects any regular expression to start with /, too (it seems).
So the fix was to add a trailing /, and escape the literal / as \/.
Unfortunately this only removes the error message, but doesn't make the override work (match).
It is also required to backslash-escape the parentheses and the slash(es):
Instead of /var(total_MB) you need to write /\/var\(total_MB\).
The original problem has two origins:
The monitoring plugin specification at https://www.monitoring-plugins.org/doc/guidelines.html#AEN201 states: "2. label can contain any characters except the equals sign or single quote (')") states that any character except = and ' is allowed as metric name.
Grafana v6.7.3 proposes the incorrect (i.e.: unescaped) values for "alias or regex".
That is how I had created the problem.
I have several metrics with the label "service". I want to get a list of all the "service" levels that begin with "abc" and end with "xyz". These will be the values of a grafana template variable.
This is that I have tried:
label_values(service) =~ "abc.*xyz"
However this produces a error Template variables could not be initialized: parse error at char 13: could not parse remaining input "(service_name) "...
Any ideas on how to filter the label values?
This should work (replacing up with the metric you mention):
label_values(up{service=~"abc.*xyz"}, service)
Or, in case you actually need to look across multiple metrics (assuming that for some reason some metrics have some service label values and other metrics have other values):
label_values({__name__=~"metric1|metric2|metric3", service=~"abc.*xyz"}, service)
We have a bug at this CKEditor plugin, and a generic solution is like to this, applying a generic string-filter only to terminal text nodes of the DOM.
QUESTION: how (need code example) to traverse a selection node (editor.getSelection()) by CKEditor-4 tools, like CKEDITOR.dom.range?
First step will be to get ranges from current selection. To do this just call:
var ranges = editor.getSelection().getRanges();
This gives you an array of ranges, because theoretically (and this theory is proved only by Firefox) one selection can contain many ranges. But in 99% of real world cases you can just handle the first one and ignore other. So you've got range.
Now, the easiest way to iterate over each node in this range is to use CKEDITOR.dom.walker. Use it for example this way:
var walker = new CKEDITOR.dom.walker( range ),
node;
while ( ( node = walker.next() ) ) {
// Log values of every text node in range.
console.log( node.type == CKEDITOR.NODE_TEXT && node.getText() );
}
However, there's a problem with text nodes at the range's boundaries. Consider following range:
<p>[foo<i>bar</i>bo]m</p>
This will log: foo, bar, and bom. Yes - entire bom, because it is a single node and walker does not modify DOM (there's a bug in documentation).
Therefore you should check every node you want to transform if it equals range's startContainer or endContainer and if yes - transform only that part of it which is inside range.
Note: I don't know walker's internals and I'm not sure whether you can modify DOM while iterating over it. I'd recommend first caching nodes and then making changes.