I know how to use the API to perform simple request such as display node information selecting node by labels value.
For example : curl http://localhost:8080/api/v1/nodes?labelSelector=kubernetes.io/role%3Dworker3
Display informations about node whose role is worker3.
Is there a way to perform the same request using a JSON query ?
looked on the web to find a such example but did not find one.
You can query with kubectl by label.
The Roles of the node are just labels.
To return in yaml format
kubectl get nodes -l node-role.kubernetes.io/worker -o yaml
To return in json format
kubectl get nodes -l node-role.kubernetes.io/worker -o json
Update
Querying the api with json you can do like so:
curl http://localhost:8080/api/v1/nodes?{"node.kubernetes.io/worker01":"worker01"}
This in my case returns this:
{
"kind": "NodeList",
"apiVersion": "v1",
"metadata": {
"resourceVersion": "317238"
},
"items": [
{
"metadata": {
"name": "worker01",
"uid": "a2bec224-361f-49e9-8bba-b3b172816d6e",
"resourceVersion": "316653",
"creationTimestamp": "2022-12-24T11:04:43Z",
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/os": "linux",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "worker01",
"kubernetes.io/os": "linux",
"microk8s.io/cluster": "true",
"node.kubernetes.io/microk8s-worker": "microk8s-worker"
},
............
As you can see it works, but you must analyse 2 things generally.
the api version (can be different to v1, depends on the kubernetes version)
the labels and property name.
The example above comes from microk8s, here i havent even Roles defined.
kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready <none> 17d v1.25.4
worker01 Ready <none> 17d v1.25.4
So i looked for some label that could extract the required data.
Related
I have grafana Dashboards, Pods drop down coming None within namespace, however we have pods running in namespace and pulling data prometheus.
Screenshot:
Query:
"datasource": "Prometheus",
"definition": "",
"description": null,
"error": null,
"hide": 0,
"includeAll": false,
"label": "Pod",
"multi": false,
"name": "pod",
"options": [],
"query": {
"query": "query_result(sum(container_memory_working_set_bytes{namespace=\"$namespace\"}) by (pod_name))",
"refId": "Prometheus-pod-Variable-Query"
},
"refresh": 1,
"regex": "/pod_name=\\\"(.*?)(\\\")/",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
I am imported json code:
https://grafana.com/grafana/dashboards/6879
Edit your dashboard's JSON:
Rename "pod_name" to "pod" in the 2 places (and save)
Looks like this grafana dashboard was created with older kubernetes version,
and metrics internals since changed.
Probably will also need similar edits for "container_name" changing to "container" in these older dashboards
This might not be a full answer, but I cannot yet comment.
The linked dashboard imports and works for me fine. So I suspect one of these:
Prometheus scraping is not running (correctly). You could enter directly into the Prometheus app and check whether if the container_memory_working_set_bytes metric has any value at all, anywhere.
The kube_system system namespace might be restricted with respect to scraping and such. If another namespace works and only this one doesn't, then this is the case.
i am trying to experiment with scaling one of my application pods running on my raspberry pi kubernetes cluster using HPA + custom metrics but ran into several issues which despite reading the documentations on https://github.com/DirectXMan12/k8s-prometheus-adapter and troubleshooting for the past 2 days, i am still having difficulties grasping why some problems are happening.
Firstly, i built an ARM-compatible image of k8s-prometheus-adapter and install it using helm. I can confirm its running properly by checking the pod logs.
I have also set up a script which sends raspberry pis temperature to pushgateway and i can query via this Prometheus query node_temp, which will return the following series
node_temp{job="kube4"} 42
node_temp{job="kube1"} 44
node_temp{job="kube2"} 39
node_temp{job="kube3"} 40
Now i want to be able to scale one of my application pods using the above temperature values as an experiment to understand better how it works.
Below is my k8s-prometheus-adapter helm values.yml file
image:
repository: jaanhio/k8s-prometheus-adapter-arm
tag: latest
logLevel: 7
prometheus:
url: http://10.17.0.12
rules:
default: false
custom:
- seriesQuery: 'etcd_object_counts'
resources:
template: <<.Resource>>
name:
as: "etcd_object"
metricsQuery: count(etcd_object_counts)
- seriesQuery: 'node_temp'
resources:
template: <<.Resource>>
name:
as: "node_temp"
metricsQuery: count(node_temp)
After installing via helm, i ran kubectl get apiservices and can see v1beta1.custom.metrics.k8s.io listed.
i then ran kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq and got the following
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "custom.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "jobs.batch/node_temp",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "jobs.batch/etcd_object",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
]
i then tried to query the value of the registered node_temp metrics using kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp but got the following response
Error from server (InternalError): Internal error occurred: unable to list matching resources
Questions:
Why is the node_temp metrics associated with jobs.batch resource type?
Why am i not able to retrieve the value of metrics via kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp?
What is a definitive way of figuring the path of the query? e.g /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp i kinda trial and error until i got see somewhat of a response. i also see some other path with namespaces in the query e.g /apis/custom.metrics.k8s.io/v1beta1/namespaces/*/metrics/foo_metrics
Any help and advice will be greatly appreciate!
Why is the node_temp metrics associated with jobs.batch resource type?
It picks the labels attached to the prometheus metrics and tries to interpret them, in this case u have clearely "job-kube4"
Why am i not able to retrieve the value of metrics via kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp?
Metrics are namespaced, see the "namespaced:true" so you'll need "/apis/custom.metrics.k8s.io/v1beta1/namespaces//jobs//node_temp"
What is a definitive way of figuring the path of the query? e.g /apis/custom.metrics.k8s.io/v1beta1/jobs//node_temp i kinda trial and error until i got see somewhat of a response. i also see some other path with namespaces in the query e.g /apis/custom.metrics.k8s.io/v1beta1/namespaces//metrics/foo_metrics
Check https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/custom-metrics-api.md#api-paths
I call the below rest api with post body to create a namespace in kubernetes
http://kuberneteshot/api/v1/namespaces/
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"name": "testnamespace"
}
}
In response i get the http status 201 created and the below json response
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "testnamespace",
"selfLink": "/api/v1/namespaces/testnamespace",
"uid": "701ff75e-5781-11e6-a48a-74dbd1a0fb73",
"resourceVersion": "57825525",
"creationTimestamp": "2016-08-01T00:46:52Z",
},
"spec": {
"finalizers": [
"kubernetes"
]
},
"status": {
"phase": "Active"
}
}
Does the status in response with phase as Active mean the namespace is successfully created and active ?
Is there any other rest api to check if the namespace exists and is active ?
The reason i would like to know if the namespace is created is because i get an error message if i fire create pod before the namespace is actually created:
Error from server: error when creating "./pod.json": pods "my-pod" is
forbidden: service account username/default was not found, retry after
the service account is created
The below works fine if i give a sleep of 5 seconds between create namespace and create pod command
kubectl delete namespace testnamepsace;kubectl create namespace
testnamepsace;sleep 5;kubectl create -f ./pod.json
--namespace=testnamepsace
If i don't give the sleep of 5 seconds i see the error message mentioned above
Apparently your Pod has a hard dependency on the default ServiceAccount, so you probably want to check it's been created instead of looking only at the namespace state. The existence of the namespace doesn't guarantee the immediate availability of your default ServiceAccount.
Some API endpoints you might want to query:
GET /api/v1/namespaces/foo/serviceaccounts/default
returns 200 with the object description if the ServiceAccount default exists in the namespace foo, 404 otherwise
GET /api/v1/serviceaccounts?fieldSelector=metadata.namespace=foo,metadata.name=default
returns 200 and a list of all ServiceAccount items in the namespace foo with name default (empty list if no object matches)
Yes, the namespace is persisted prior to being returned from the create API call. The returned object shows the status of the namespace as Active.
You can also do GET http://kubernetehost/api/v1/namespaces/testnamespace to retrieve the same information.
kubernetes version:1.02
REST api
DELETE /api/v1/namespaces/default/replicationcontrollers/test
body
{
"apiVersion": "v1",
"kind": "ReplicationController",
"gracePeriodSeconds": 0}
}
Fail
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "converting to : type names don't match (ReplicationController, DeleteOptions), and no conversion 'func (v1.ReplicationController, api.DeleteOptions) error' registered.",
"code": 500
}
if setting body is empty, delete success, but pod is exist.
kubectl get rc, rc is deleted
kubectl get pod, pod is existting
why?
How can I delete rc with all pods by api delete method?
API requests are designed to be able to be fulfilled immediately. Tasks like reaping/recursively deleting are typically handled by a client by combining multiple API requests. In this case, you can do what kubectl does when running kubectl delete rc/test (which you can see by adding --v=8):
Set the spec.replicas of rc/test to 0
Watch until status.replicas of rc/test is also 0
Delete rc/test
I am trying to configure php phabricator example from kubernetes but after creating the replication controller. POD is not showing in ready state ever. It shows in below state:
NAME READY STATUS RESTARTS AGE
phabricator-controller-z0nk3 0/1 CrashLoopBackOff 5 2m
Below is the controller yaml:
{
"kind": "ReplicationController",
"apiVersion": "v1",
"metadata": {
"name": "phabricator-controller",
"labels": {
"name": "phabricator"
}
},
"spec": {
"replicas": 1,
"selector": {
"name": "phabricator"
},
"template": {
"metadata": {
"labels": {
"name": "phabricator"
}
},
"spec": {
"containers": [
{
"name": "phabricator",
"image": "fgrzadkowski/example-php-phabricator",
"ports": [
{
"name": "http-server",
"containerPort": 80
}
]
}
]
}
}
}
}
Can someone please suggest me how to fix this?
This Pod is crash-looping. You can tell because the number of restarts is greater than zero.
kubectl describe pods <pod-name>
Should give further details to help debug. As will
kubectl logs <pod-name>
Actually tracking issues with kubectl describe pods <pod-name> and kubectl logs <pod-name> is indeed the default way to track issues, unfortunately in my case it WASN'T helpful (at first.) All logs were nice or at least were giving no error or clue that something goes wrong.
Readiness and Liveness probes were however showing the app is not passing through...
So where the devil were hiding? In my case increasing values for "initialDelaySeconds" and/or "timeoutSeconds" for Readiness and Liveness probes did the thing.
My first assumption was the app has not enough time to reach "Ready status". However app was still not ready and failed in fact... !!!BUT!!! extending those values increased deployment attempt time and thus I've been able to reach more logs. And what I got??? "Database connection failed attempt due to the timeout". So no connection to the database, and the app is died in fact. Tricky moment is - timeouts are not appearing quickly and you need to wait a bit more ... at least default values for "initialDelaySeconds" and/or "timeoutSeconds" were unable to give me needed time to see the "database connectivity timeout".
When firewall rule was set to allow app talk to the database, issue has gone!