I'm trying to update a service using :
kubectl update service my-service \
--patch='{ "apiVersion":"v1", "spec": { "selector": { "build":"2"} } }'
I receive the following Error :
Error from server: service "\"apiVersion\":\"v1\"," not found
I have tried the following :
moving the service name to the end
Removing the apiVersion
Maybe the kubectl update is not available for service ?
For now I was making my updates by simply stoping and restarting my service. But sometime, the corresponding forwarding-port changes. So it seems to not be the good choice ...
PS:
v0.19
api_v1
I am not sure if patch is 100% working yet, but if you are going to do this, you at least need to put apiVersion inside metadata, like so:
--patch='{ metadata:{ "apiVersion":"v1" }, "spec": { "selector": { "build":"2"} } }'
Related
As part of kubernetes 1.19, structured logging has been implemented.
I've read that kubernetes log's engine is klog and structured logs are following this format :
<klog header> "<message>" <key1>="<value1>" <key2>="<value2>" ...
Cool ! But even better, you apparently can pass a --logging-format=json flag to klog so logs are generated in json directly !
{
"ts": 1580306777.04728,
"v": 4,
"msg": "Pod status updated",
"pod":{
"name": "nginx-1",
"namespace": "default"
},
"status": "ready"
}
Unfortunately, I haven't been able to find out how and where I should specify that --logging-format=json flag.
Is it a kubectl command? I'm using Azure's aks.
--logging-format=json is a flag which need to be set on all Kuberentes System Components ( Kubelet, API-Server, Controller-Manager & Scheduler). You can check all flags here.
Unfortunately you cant do it right now with AKS as you have the managed control plane from Microsoft.
I am working on Opersator-SDK. In my operator controller, I want to perform CRUD operation on a Custom Resource (say ExampleCR) for which go api module is not available
Suppose ExampleCR does not have go api (I have access to crd definition in yaml). I am watching Deployment object and whenever Deployment object is created or updated. I want to perform following operation on ExampleCR in my controller code.
kubectl create on ExampleCR
kubectl update on ExampleCR
kubectl get on ExampleCR
I was able to solve this using unstructured.Unstructured type.
Using the following sample, you can watch the CR (ExampleCR) in the controller (Ref).
// You can also watch unstructured objects
u := &unstructured.Unstructured{}
u.SetGroupVersionKind(schema.GroupVersionKind{
Kind: "ExampleCR",
Group: "",
Version: "version", // set version here
})
//watch for primary resource
err = c.Watch(&source.Kind{Type: u}, &handler.EnqueueRequestForObject{})
//watch for secondary resource
err = c.Watch(&source.Kind{Type: u}, &handler.EnqueueRequestForOwner{
IsController: true,
OwnerType: &ownerVersion.OwnerType{}})
Once you have done this, controller will receive the reconciliation request.
CRUD operation will remain same as we do it for other kinds (for example Pod).
creation of object can be done using following
func newExampleCR() (*unstructured.Unstructured)
&unstructured.Unstructured{
Object: map[string]interface{}{
"apiVersion": "version", //set version here
"kind": "ExampleCR", // set CR name here
"metadata": map[string]interface{}{
"name": "demo-deployment",
},
"spec": map[string]interface{}{
//spec goes here
},
},
}
}
Complete example can be found here for deployment object
NOTE: You have to make sure that the CRD is registered with the scheme before the manager is started.
i am trying to experiment with scaling one of my application pods running on my raspberry pi kubernetes cluster using HPA + custom metrics but ran into several issues which despite reading the documentations on https://github.com/DirectXMan12/k8s-prometheus-adapter and troubleshooting for the past 2 days, i am still having difficulties grasping why some problems are happening.
Firstly, i built an ARM-compatible image of k8s-prometheus-adapter and install it using helm. I can confirm its running properly by checking the pod logs.
I have also set up a script which sends raspberry pis temperature to pushgateway and i can query via this Prometheus query node_temp, which will return the following series
node_temp{job="kube4"} 42
node_temp{job="kube1"} 44
node_temp{job="kube2"} 39
node_temp{job="kube3"} 40
Now i want to be able to scale one of my application pods using the above temperature values as an experiment to understand better how it works.
Below is my k8s-prometheus-adapter helm values.yml file
image:
repository: jaanhio/k8s-prometheus-adapter-arm
tag: latest
logLevel: 7
prometheus:
url: http://10.17.0.12
rules:
default: false
custom:
- seriesQuery: 'etcd_object_counts'
resources:
template: <<.Resource>>
name:
as: "etcd_object"
metricsQuery: count(etcd_object_counts)
- seriesQuery: 'node_temp'
resources:
template: <<.Resource>>
name:
as: "node_temp"
metricsQuery: count(node_temp)
After installing via helm, i ran kubectl get apiservices and can see v1beta1.custom.metrics.k8s.io listed.
i then ran kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq and got the following
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "custom.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "jobs.batch/node_temp",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "jobs.batch/etcd_object",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
]
i then tried to query the value of the registered node_temp metrics using kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp but got the following response
Error from server (InternalError): Internal error occurred: unable to list matching resources
Questions:
Why is the node_temp metrics associated with jobs.batch resource type?
Why am i not able to retrieve the value of metrics via kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp?
What is a definitive way of figuring the path of the query? e.g /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp i kinda trial and error until i got see somewhat of a response. i also see some other path with namespaces in the query e.g /apis/custom.metrics.k8s.io/v1beta1/namespaces/*/metrics/foo_metrics
Any help and advice will be greatly appreciate!
Why is the node_temp metrics associated with jobs.batch resource type?
It picks the labels attached to the prometheus metrics and tries to interpret them, in this case u have clearely "job-kube4"
Why am i not able to retrieve the value of metrics via kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp?
Metrics are namespaced, see the "namespaced:true" so you'll need "/apis/custom.metrics.k8s.io/v1beta1/namespaces//jobs//node_temp"
What is a definitive way of figuring the path of the query? e.g /apis/custom.metrics.k8s.io/v1beta1/jobs//node_temp i kinda trial and error until i got see somewhat of a response. i also see some other path with namespaces in the query e.g /apis/custom.metrics.k8s.io/v1beta1/namespaces//metrics/foo_metrics
Check https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/custom-metrics-api.md#api-paths
As a QA in our company I am daily user of kubernetes, and we use kubernetes job to create performance tests pods. One advantage of job, according to the docs, is
to create one Job object in order to reliably run one Pod to completion
But in our tests this feature will create infinite pods if previous ones fail, which will occupy resources of our team's shared cluster, and deleting such pods will take a lot of time. see this image:
Currently the job manifest is like this:
{
"apiVersion": "batch/v1",
"kind": "Job",
"metadata": {
"name": "upgradeperf",
"namespace": "ntg6-grpc26-tts"
},
"spec": {
"template": {
"spec": {
"containers": [
{
"name": "upgradeperfjob",
"image":
"mycompany.com:5000/ncs-cd-qa/upgradeperf:0.1.1",
"command": [
"python",
"/jmeterwork/jmeter.py",
"-gu",
"git#gitlab-pri-eastus2.dev.mycompany.net:mobility-ncs-tools/tts-cdqa-tool.git",
"-gb",
"upgradeperf",
"-t",
"JMeter/testcases/ttssvc/JMeterTestPlan_ttssvc_cmpsize.jmx",
"-JtestDataFile",
"JMeter/testcases/ttssvc/testData/avaml_opus.csv",
"-JthreadNum",
"3",
"-JthreadLoopCount",
"1500",
"-JresultsFile",
"results_upgradeperf_cavaml_opus_t3_l1500.csv",
"-Jhost",
"mtl-blade32-03.mycompany.com",
"-Jport",
"28416"
]
}
],
"restartPolicy": "Never",
"imagePullSecrets": [
{
"name": "docker-registry-secret"
}
]
}
}
}
}
In some cases, such as misconfiguring of ip/ports, 'reliably run one Pod to completion' is impossible and recreating pods is waste of time and resource.
So is it possible, and how to limit kubernetes job to create a maxium number(say 3) of pods if always fail?
Depending on your kubernetes version, you can resolve this problem with these methods:
set the option: restartPolicy: OnFailure, then the failed container will be restarted in the same Pod, so you will not get lots of failed Pods, instead you will get a Pod with lots of restart.
From Kubernetes 1.8 on, There is a parameter backoffLimit to control the restart policy of failed job. This parameter defines the retry times of the job before treating the job to be failed, default 6 times. For this parameter to work you must set the parameter restartPolicy: Never .
You probably didn't set restartPolicy: Never in your pod spec, add that and I would expect it matches your expected behaviors better.
I call the below rest api with post body to create a namespace in kubernetes
http://kuberneteshot/api/v1/namespaces/
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"name": "testnamespace"
}
}
In response i get the http status 201 created and the below json response
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "testnamespace",
"selfLink": "/api/v1/namespaces/testnamespace",
"uid": "701ff75e-5781-11e6-a48a-74dbd1a0fb73",
"resourceVersion": "57825525",
"creationTimestamp": "2016-08-01T00:46:52Z",
},
"spec": {
"finalizers": [
"kubernetes"
]
},
"status": {
"phase": "Active"
}
}
Does the status in response with phase as Active mean the namespace is successfully created and active ?
Is there any other rest api to check if the namespace exists and is active ?
The reason i would like to know if the namespace is created is because i get an error message if i fire create pod before the namespace is actually created:
Error from server: error when creating "./pod.json": pods "my-pod" is
forbidden: service account username/default was not found, retry after
the service account is created
The below works fine if i give a sleep of 5 seconds between create namespace and create pod command
kubectl delete namespace testnamepsace;kubectl create namespace
testnamepsace;sleep 5;kubectl create -f ./pod.json
--namespace=testnamepsace
If i don't give the sleep of 5 seconds i see the error message mentioned above
Apparently your Pod has a hard dependency on the default ServiceAccount, so you probably want to check it's been created instead of looking only at the namespace state. The existence of the namespace doesn't guarantee the immediate availability of your default ServiceAccount.
Some API endpoints you might want to query:
GET /api/v1/namespaces/foo/serviceaccounts/default
returns 200 with the object description if the ServiceAccount default exists in the namespace foo, 404 otherwise
GET /api/v1/serviceaccounts?fieldSelector=metadata.namespace=foo,metadata.name=default
returns 200 and a list of all ServiceAccount items in the namespace foo with name default (empty list if no object matches)
Yes, the namespace is persisted prior to being returned from the create API call. The returned object shows the status of the namespace as Active.
You can also do GET http://kubernetehost/api/v1/namespaces/testnamespace to retrieve the same information.