Is there a way to easily query Kubernetes resources in an intuitive way? Basically I want to run queries to extract info about objects which match my criteria. Currently I face an issue where my match labels isn't quite working and I would like to run the match labels query manually to try and debug my issue.
Basically in a pseudo code way:
Select * from pv where labels in [red,blue,green]
Any third party tools who do something like this? Currently all I have to work with is the search box on the dashboard which isn't quite robust enough.
You could use kubectl with JSONPath (https://kubernetes.io/docs/reference/kubectl/jsonpath/). More information on JSONPath: https://github.com/json-path/JsonPath
It allows you to query any resource property, example:
kubectl get pods -o=jsonpath='{$.items[?(#.metadata.namespace=="default")].metadata.name}'
This would list all pod names in namespace "default". Your pseudo code would be something along the lines:
kubectl get pv -o=jsonpath='{$.items[?(#.metadata.label in ["red","blue","green"])]}'
Related
I have a CR in the cluster. I know that there is a filed specialField somewhere in this CRD. I do not know where it is. Right now I am randomly exploring the CRD using kubectl explain path.to.some.filed to try to fiend the filed. Is there some way to explain the entire nested structure using kubectl explain?
What I am looking for is something like this
kubectl explain-magic my-crd
my-crd
a
b
c
other
field
more
fields
very
nested
field
or as an alternative
my-crd
a
a.b
a.b.c
other
other.field
more
more.fields
very
very.nested
very.nested.field
Add the --recursive flag to display all of the fields at once without descriptions. Information about each field is
retrieved from the server in OpenAPI format.
from kubectl help explain.
Just do: kubectl explain --recursive my-crd
I am trying to understand the difference between pods and annotations.
Standard documentation says that annotations captures "non-identifying information".
While on labels, selectors can be applied. Labels are used to organise objects in kubernetes cluster.
If this is the case then why istio use pod annotations instead of labels for various different settings : https://istio.io/latest/docs/reference/config/annotations/
Isn't label is good approach ?
Just trying to understand what advantages does annotations provide, if istio developers chose to use annotations.
As Extending the Burak answer,
Kubernetes labels and annotations are both ways of adding metadata to
Kubernetes objects. The similarities end there, however. Kubernetes
labels allow you to identify, select and operate on Kubernetes
objects. Annotations are non-identifying metadata and do none of these
things.
Labels are mostly used to attach with the resources like POD, Replica set, etc. it also get used to route the traffic and routing deployment to service and other.
Labels are getting stored in the ETCD database so you can search using it.
Annotation is mostly to store metadata and config-if any.
Metadata like : owner details, last helm release if using helm, side car injection
You can store owner details in labels, K8s use labels for traffic routing from service to deployment and labels should be the same on both resources (deployment & service) to route traffic.
What will you do in that case to match labels for resources? Use service owner name same inside all deployment & service? when you are running multiple distributed services managed by diff team and service owners.
If you notice some of annotation of istio is just for storing metadata like : install.operator.istio.io/chart-owner, install.operator.istio.io/owner-generation
Read more at : https://istio.io/latest/docs/reference/config/annotations/
You should also check once syntax of both label and annotation.
We have multiple people ( with admin access ) doing the deployments in kubernetes cluster. We are finding it difficult to manage who has modified which object.
We can control the access and privileges using RBAC with roles and role bindings. We are planning to implement well defined roles and rolebindings for different groups.
We would also want to list all objects modified in N days in a specific namespace. Is there a way to display the objects using kubectl? please let me know
This probably can't be done easily just with kubectl.
But you might look into Kubernetes auditing. It causes the API server to record all requests to the API, and you can query them in different ways. For example, it should be possible to query the audit logs for all the objects that have been specified in the last N days.
I am trying to put a dropdown for each API end point which will show the QPS and Latency of http requests (RED metrics).
I used Grafana's templating and used the following prometheus query.
label_values(http_duration_milliseconds_count, api_path)
But the problem here is sort order. It shows some longtail api requests like /admin/phpMyAdmin all.
I want to do only the top 10 endpoints by count to be shown in this drop down. How do I achieve this?
Attached an image for reference on my first dashboard.
We can use query_result to achieve this.
https://grafana.com/docs/grafana/latest/datasources/prometheus/template-variables/#use-query-variables
query_result(topk(10, sort_desc(sum(http_tt_ms_count) by (api_path))))
http_tt_ms_count - is my metric timeseries of Prometheus with time taken.
api_path - is my label name
This query_result will give three-tuple value like this.
{api_path="/search/query"} 25704195 1507641522000
used the Regex field in query path to get only the api names.
*api_path="(.*)".*
This looks like a long way but
label_values((topk(10, sort_desc(sum(http_tt_ms_count) by (api_path)))), api_path)
is not working in Grafana which made me to go into this path.
I annotate my Kubernetes objects with things like version and whom to contact when there are failures. How would I relay this information to Prometheus, knowing that these annotation values will frequently change? I can't capture this information in Prometheus labels, as they serve as the primary key for a target (e.g. if the version changes, it's a new target altogether, which I don't want). Thanks!
I just wrote a blog post about this exact topic! https://www.weave.works/aggregating-pod-resource-cpu-memory-usage-arbitrary-labels-prometheus/
The trick is Kubelet/cAdvisor doesn't expose them directly, so I run a little exporter which does, and join this with the pod name in PromQL. The exporter is: https://github.com/tomwilkie/kube-api-exporter
You can do a join in Prometheus like this:
sum by (namespace, name) (
sum(rate(container_cpu_usage_seconds_total{image!=""}[5m])) by (pod_name, namespace)
* on (pod_name) group_left(name)
k8s_pod_labels{job="monitoring/kube-api-exporter"}
)
Here I'm using a label called "name", but it could be any label.
We use the same trick to get metrics (such as error rate) by version, which we then use to drive our continuous deployment system. kube-api-exporter exports a bunch of useful meta-information about Kubernetes objects to Prometheus.
Hope this helps!