Host's pods panel in Grafana - kubernetes

I want to have a panel in Grafana which displays what pods are currently running in a host.
For the host variable I have the following query (the job variable is just label_values(node_uname_info, job).):
label_values(node_uname_info{job="$job"}, instance)
This gives me an array of sockets: host_ip:port
I can get the pod names from kube_pod_info{job="$job", host_ip="$host_ip"}, but in order to get the IP I need to remove the port part of the socket:
label_replace(node_uname_info{job="$job", instance="$node"}, "host_ip", "$1", "instance", "(.*):.*")
I haven't found how to use the new host_ip label in the pod query to eventually get all the pod label values of kube_pod_info. I don't want to put the label_replace in Prometheus to avoid data duplication - is there a way to use the new host_ip label in the pod query?
Edit:
I added the host_ip variable with the regex as shan1024 showed in his answer and changed the panel's query to:
sum by (pod) (kube_pod_info{job="$job", host_ip="$host_ip"})
Then I changed the panel's visualization to table and added column styles to Time and Value (chose type Hidden). This allows me to display the host's running pods in a list-like fashion.

This is actually quite easy to do in Grafana and no need to change labels in Prometheus. You just need to add a regex in the instance variable (when we add a regex with a capturing group, the value(s) of the 1st captured group will be the value(s) of the variable).
e.g.
Variable definition without Regex (you get host_ip:port)-
Variable definition with Regex (you only get host_ip)-
Then you can add a new variable with value kube_pod_info{ host_ip="$instance" } to get all pods in the selected host.

Related

JSON path semantics different in kubectl and additional printer columns in custom resource definition

I use kubectl to list Kubernetes custom resources of a kind mykind with an additional table column LABEL that contains the value of a label a.b.c.com/key if present:
kubectl get mykind -o=custom-columns=LABEL:.metadata.labels.'a\.b\.c\.com/key'
This works, i.e., the label value is properly displayed.
Subsequently, I wanted to add a corresponding additional printer column to the custom resource definition of mykind:
- description: Label value
jsonPath: .metadata.labels.'a\.b\.c\.com/key'
name: LABEL
type: string
Although the additional column is added to kubectl get mykind, it is empty and no label value is shown (in contrast to above kubectl command). My only suspicion were problems with escaping of the special characters - but no variation helped.
Are you aware of any difference between the JSON path handling in kubectl and additional printer columns? I expected strongly that they are exactly the same.
mdaniel's comment works!
- description: Label value
jsonPath: '.metadata.labels.a\.b\.c\.com/key'
name: LABEL
type: string
You need to use \. instead of . and use single quotes ' '. It doesn't work with double quotes for the reasons I don't understand

Pulum DigitalOcean: use outputs

I want to create some servers on DigitalOcean using Pulumi. I have the following code:
for i in range(0, amount):
name = f"droplet-{i+1}"
droplet = digitalocean.Droplet(
name,
image=_image,
region=_region,
size=_size,
)
pulumi.export(f"droplet-ip-{i+1}", droplet.ipv4_address)
This is correctly outputting the IP address of the servers on the console.
However I would like to use the IP addresses elsewhere in my Python script. Therefor I had added the droplets to a list as follows:
droplets = []
for i in range(0, amount):
name = f"droplet-{i+1}"
droplet = digitalocean.Droplet(
name,
image=_image,
region=_region,
size=_size,
)
pulumi.export(f"droplet-ip-{i+1}", droplet.ipv4_address)
droplets.append(droplet)
to then loop over the droplets as follows:
for droplet in droplets:
print(droplet.ipv4_address)
In the Pulumi output, I see the following:
Diagnostics:
pulumi:pulumi:Stack (Pulumi_DigitalOcean-dev):
<pulumi.output.Output object at 0x105086b50>
<pulumi.output.Output object at 0x1050a5ac0>
I realize that while the droplets are still being created, the IP address is unknown but I'm adding the droplets to the list after the creation.
Is there a way to know the IP addresses at some point so it can be used elsewhere in the Python script.
The short answer is that because these values are Outputs, if you want the strings, you'll need to use .apply:
https://www.pulumi.com/docs/intro/concepts/inputs-outputs/#apply
To access the raw value of an output and transform that value into a new value, use apply. This method accepts a callback that will be invoked with the raw value, once that value is available.
You can print these IPs by iterating over the list and calling the apply method on the ipv4_address output value:
...
pulumi.export(f"droplet-ip-{i+1}", droplet.ipv4_address)
droplets.append(droplet)
...
for droplet in droplets:
droplet.ipv4_address.apply(lambda addr: print(addr))
$ pulumi up
...
Diagnostics:
pulumi:pulumi:Stack (so-71888481-dev):
143.110.157.64
137.184.92.205
Outputs:
droplet-ip-1: "137.184.92.205"
droplet-ip-2: "143.110.157.64"
Depending on how you plan to use these strings in your program, this particular may may not be perfect, but in general, if you want the unwrapped value of pulumi.Output, you'll need to use .apply().
The pulumi.Output.all() also comes in handy if you want to wait for several output values to resolve before using them:
https://www.pulumi.com/docs/intro/concepts/inputs-outputs/#all
If you have multiple outputs and need to join them, the all function acts like an apply over many resources. This function joins over an entire list of outputs. It waits for all of them to become available and then provides them to the supplied callback.
Hope that helps!

Terraform EKS specify node-role.kubernetes.io label on node group

In the terraform aws_eks_node_group resource I can't set :
labels = {
"node-role.kubernetes.io/others" = "other"
}
as AWS complains labels key should not contains kubernetes.io.
Error: error creating EKS Node Group (my-cluster:others): InvalidParameterException: Label cannot contain reserved labels kubernetes.io/
{
ClusterName: "my-cluster",
Message_: "Label cannot contain reserved labels kubernetes.io/",
NodegroupName: "others"
}
Also, EC2 instances spawned have no name and I have no clue on how to specify a Name for my instances based on their node group.
Any idea on how to achieve this ?
As per the documentation you can't use specific labels, regarding labels:
The kubernetes.io/ and k8s.io/ prefixes are reserved for Kubernetes core components. Valid label values must be 63 characters or less and must be empty or begin and end with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between.
Regarding your specific label. There are many issues since k8s 1.15 or 1.16 where some change in the core kubernetes don’t allow that label. one detailed issue
As for naming of EC2 instances created by EKS Node Group. Currently, there is no way to pass "Name" tag. This question is a duplicate of this where you can also find the answer how to name your instances for time being.
node-role.kubernetes.io and kubernetes.io - this is DIFFERENT prefixes

How can I filter the result of label_values(label) to get a list of labels that match a regex?

I have several metrics with the label "service". I want to get a list of all the "service" levels that begin with "abc" and end with "xyz". These will be the values of a grafana template variable.
This is that I have tried:
label_values(service) =~ "abc.*xyz"
However this produces a error Template variables could not be initialized: parse error at char 13: could not parse remaining input "(service_name) "...
Any ideas on how to filter the label values?
This should work (replacing up with the metric you mention):
label_values(up{service=~"abc.*xyz"}, service)
Or, in case you actually need to look across multiple metrics (assuming that for some reason some metrics have some service label values and other metrics have other values):
label_values({__name__=~"metric1|metric2|metric3", service=~"abc.*xyz"}, service)

Prometheus OR when using rate()

Summary
I'm trying to figure out how to properly use the OR | operator in a Prometheus query because my imported Grafana dashboard is not working.
Long version
I'm trying to debug a Grafana dashboard based on some data scraped from my Kubernetes pods running AppMetrics/Prometheus; the dashboard is here. Basically what happens is that when the value "All" for the server is selected on the Grafana dashboard (server is an individual pod in this case), no data appears. However, when I select an individual pod, then data does appear.
Here's an example of the same metric scraped from the two pods:
# HELP application_httprequests_transactions
# TYPE application_httprequests_transactions summary
application_httprequests_transactions_sum{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test"} 5.006965628
application_httprequests_transactions_count{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test"} 1367
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.5"} 0.000202825
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.75"} 0.000279318
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.95"} 0.000329862
application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.99"} 0.055584233
# HELP application_httprequests_transactions
# TYPE application_httprequests_transactions summary
application_httprequests_transactions_sum{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test"} 6.10214788
application_httprequests_transactions_count{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test"} 1363
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.5"} 0.000218548
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.75"} 0.000277483
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.95"} 0.033821094
application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.99"} 0.097113234
I ran the Query inspector in Grafana to find out which query it is calling, and then ran the PromQL query in Prometheus itself. Basically, when I execute the following PromQL queries individually, they return data:
rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-l9tdv"}[15m])*60
rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-jdq78"}[15m])*60
However, when I try to use PromQL's | operator to combine them, I don't get data back:
rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-l9tdv|myapp-test-58d94bf78d-jdq78"}[15m])*60
Here's the raw output from Grafana's query inspector:
xhrStatus:"complete"
request:Object
method:"GET"
url:"api/datasources/proxy/56/api/v1/query_range?query=rate(application_httprequests_transactions_count%7Benv%3D%22test%22%2Capp%3D%22MyApp%22%2Cserver%3D%22myapp-test-58d94bf78d-jdq78%7Cmyapp-test-58d94bf78d-l9tdv%7Cmyapp-test-5b8c9845fb-7lklm%7Cmyapp-test-5b8c9845fb-8jf7n%7Cmyapp-test-5b8c9845fb-d9x5c%7Cmyapp-test-5b8c9845fb-fw4gj%7Cmyapp-test-5b8c9845fb-vtl9z%7Cmyapp-test-5b8c9845fb-vv7xv%7Cmyapp-test-5b8c9845fb-wq9bs%7Cmyapp-test-5b8c9845fb-xqfrt%7Cmyapp-test-69999d58b5-549vd%7Cmyapp-test-69999d58b5-lmp8x%7Cmyapp-test-69999d58b5-nbvt9%7Cmyapp-test-69999d58b5-qphj2%7Cmyapp-test-6b8dcc5ffb-gjjvj%7Cmyapp-test-6b8dcc5ffb-rxfk2%7Cmyapp-test-7fdf446767-bzhm2%7Cmyapp-test-7fdf446767-hp46w%7Cmyapp-test-7fdf446767-rhqhq%7Cmyapp-test-7fdf446767-wxmm2%22%7D%5B1m%5D)*60&start=1540574190&end=1540574505&step=15"
response:Object
status:"success"
data:Object
resultType:"matrix"
result:Array[0] => []
I opened a GitHub issue for this as well; it has a quick GIF screen recording showing what I mean: AppMetrics/Prometheus#43
| is for regular expressions, PromQL doesn't have a | operator (but it does have an or operator). You need to specify that the matcher is a regex rather than an exact match with =~:
rate(application_httprequest_transactions_count{env="test",app="MyApp",server=~"myapp-test-58d94bf78d-l9tdv|myapp-test-58d94bf78d-jdq78"}[15m])*60