I'm new to Kubernetes and I wanted to know if there is there a way I can add '--record=true' inside the deployment yaml file, so I do not have to type it on the command line!
I know it goes like this: kubectl apply -f deployfile.yml --record
I am asking this because we work on a team, and not everyone is using --record=true at the end of the command when deploying files to kubernetes!
Thank you in advance,
As far as I'm aware there is no feature like --record=true flag in kubectl that you can add to Manifest.
The command which was used to start the Deployment is being stored in the kubernetes.io/change-cause annotation. This is being used for Rollout history which is described here.
First, check the revisions of this Deployment:
kubectl rollout history deployment.v1.apps/nginx-deployment
The output is similar to this:
deployments "nginx-deployment"
REVISION CHANGE-CAUSE
1 kubectl apply --filename=https://k8s.io/examples/controllers/nginx-deployment.yaml --record=true
2 kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.9.1 --record=true
3 kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.91 --record=true
CHANGE-CAUSE is copied from the Deployment annotation kubernetes.io/change-cause to its revisions upon creation. You can specify the CHANGE-CAUSE message by:
Annotating the Deployment with kubectl annotate deployment.v1.apps/nginx-deployment kubernetes.io/change-cause="image updated to 1.9.1"
Append the --record flag to save the kubectl command that is making changes to the resource.
Manually editing the manifest of the resource.
To see the details of each revision, run:
kubectl rollout history deployment.v1.apps/nginx-deployment --revision=2
The output is similar to this:
deployments "nginx-deployment" revision 2
Labels: app=nginx
pod-template-hash=1159050644
Annotations: kubernetes.io/change-cause=kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.9.1 --record=true
Containers:
nginx:
Image: nginx:1.9.1
Port: 80/TCP
QoS Tier:
cpu: BestEffort
memory: BestEffort
Environment Variables: <none>
No volumes.
For the command history I would use $ history or check user bash_history
$ tail /home/username/.bash_history
Create an alias in your bashrc or zshrc as below
alias kubectl=kubectl --record and then do kubectl apply -f deployfile.yml
or
alias kr=kubectl --record and kr apply -f deployfile.yml
Related
After a successful
kubectl rollout restart deployment/foo
the
kubectl rollout undo deployment/foo
or
kubectl rollout undo deployment/foo --to-revision=x
are not having effect. I mean, the pods are replaced by new ones and a new revision is created which can be checked with
kubectl rollout history deployment foo
but when I call the service, the rollback had no effect.
I also tried to remove the imagePullPolicy: Always, guessing that it was always pulling even in the rollback, with no success because probably one thing is not related to the other.
Edited: The test is simple, I change the health check route of the http api to return something different in the json, and it doesn't.
Edited:
Maybe a typo, but not: I was executing with ... undo deployment/foo ..., and now tried with ... undo deployment foo .... It also gives me deployment.apps/foo rolled back, but no changes in the live system.
More tests: I changed again my api route to test what would happen if I executed a rollout undo to every previous revision one by one. I applied the last 10 revisions, and nothing.
To be able to rollback to a previous version don't forget to append the --record parameter to your kubectl command, for example:
kubectl apply -f DEPLOYMENT.yaml --record
Then you should be able to see the history as you know with:
kubectl rollout history deployment DEPLOYMENT_NAME
And your rollback will work properly
kubectl rollout undo deployment DEPLOYMENT_NAME --to-revision=CHOOSEN_REVISION_NUMBER
Little example:
consider my nginx deployment manifest "nginx-test.yaml" here:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
lets create it:
❯ kubectl apply -f nginx-test.yaml --record
Flag --record has been deprecated, --record will be removed in the future
deployment.apps/nginx-deployment created
lets check the image of this deployment, as expected from the manifest:
❯ k get pod nginx-deployment-74d589986c-k9whj -o yaml | grep image:
- image: nginx
image: docker.io/library/nginx:latest
now lets modify the image of this deployment to "nginx:1.21":
#"nginx=" correspond to the name of the container inside the pod create by the deployment.
❯ kubectl set image deploy nginx-deployment nginx=nginx:1.21.6
deployment.apps/nginx-deployment image updated
we can optionnaly check the rollout status:
❯ kubectl rollout status deployment nginx-deployment
deployment "nginx-deployment" successfully rolled out
we can check the rollout history with:
❯ kubectl rollout history deploy nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
1 kubectl apply --filename=nginx-test.yaml --record=true
2 kubectl apply --filename=nginx-test.yaml --record=true
lets check the image of this deployment, as expected:
❯ k get pod nginx-deployment-66dcfc79b5-4pk7w -o yaml | grep image:
- image: nginx:1.21.6
image: docker.io/library/nginx:1.21.6
Oh, no, i don't like this image ! Lets rollback:
❯ kubectl rollout undo deployment nginx-deployment --to-revision=1
deployment.apps/nginx-deployment rolled back
creating:
> kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-deployment-66dcfc79b5-4pk7w 1/1 Running 0 3m41s 10.244.3.4 so-cluster-1-worker3 <none> <none>
pod/nginx-deployment-74d589986c-m2htr 0/1 ContainerCreating 0 13s <none> so-cluster-1-worker2 <none> <none>
after few seconds:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/nginx-deployment-74d589986c-m2htr 1/1 Running 0 23s 10.244.4.10 so-cluster-1-worker2 <none> <none>
as you can see it worked:
❯ k get pod nginx-deployment-74d589986c-m2htr -o yaml | grep image:
- image: nginx
image: docker.io/library/nginx:latest
lets recheck the history:
❯ kubectl rollout history deploy nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
1 kubectl apply --filename=nginx-test.yaml --record=true
2 kubectl apply --filename=nginx-test.yaml --record=true
you can change the rollout history's CHANGE-CAUSE with the "kubernetes.io/change-cause" annotation:
❯ kubectl annotate deploy nginx-deployment kubernetes.io/change-cause="update image from 1.21.6 to latest" --reco
rd
Flag --record has been deprecated, --record will be removed in the future
deployment.apps/nginx-deployment annotated
lets recheck the history:
❯ kubectl rollout history deploy nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
2 kubectl apply --filename=nginx-test.yaml --record=true
3 update image from 1.21.6 to latest
lets describe the deployment:
❯ kubectl describe deploy nginx-deploy
Name: nginx-deployment
Namespace: so-tests
CreationTimestamp: Fri, 06 May 2022 00:56:09 -0300
Labels: app=nginx
Annotations: deployment.kubernetes.io/revision: 3
kubernetes.io/change-cause: update image from latest to latest
...
hope this has helped you, bguess.
I experienced a similar situation and it was more an own mistake rather than a configuration issue with my K8s manifests.
At my Docker image creation workflow, I forgot to update the version of my Docker image. I have a GitHub Action workflow that pushes the image to DockerHub. Not updating the image version, overwrote the current image with latest changes in the application.
Then my kubectl rollout undo command was pulling the correct image but that image had the most recent changes in the application. In other words, image 1.1 was the same as 1.0. Running undo had no effect on the application state.
Stupid mistake but that was my experience in case helps someone.
Along with the container image in kubernetes, I would like to update the sidecar image as well.
What will be the kubectl command for this process?
kubernetes have the command set image that allow you to update an image to the expected version
the syntax is
kubectl set image deployment/{deployment-name} {container name}:{image:version}
with a sample it look like
kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1
you can found the documentation of this command here https://kubernetes.io/fr/docs/concepts/workloads/controllers/deployment/#mise-%C3%A0-jour-d-un-d%C3%A9ploiement
Assumed you have a deployment spec look like this:
...
kind: Deployment
metadata:
name: mydeployment
...
spec:
...
template:
...
spec:
...
containers:
- name: application
image: nginx:1.14.0
...
- name: sidecar
image: busybox:3.15.0
...
kubectl set image deployment mydeployment application=nginx:1.16.0 sidecar=busybox:3.18.0
Besides what other users advise to use kubectl set image you can also patch your resource (pod, deployment, etc.).
In Kubernetes Patch documentation you have some examples there:
# Update a container's image; spec.containers[*].name is required because it's a merge key
$ kubectl patch pod valid-pod -p '{"spec":{"containers":[{"name":"kubernetes-serve-hostname","image":"new image"}]}}'
# Disable a deployment livenessProbe using a json patch with positional arrays
kubectl patch deployment valid-deployment --type json -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/livenessProbe"}]'
Also you can edit your configuration YAML and apply it.
kubectl apply -f <yamlfile with new image>
I am trying to set up Argo CD on Google Kubernetes Engine Autopilot and each pod/container is defaulting to the default resource request (0.5 vCPU and 2 GB RAM per container). This is way more than the pods need and is going to be too expensive (13GB of memory reserved in my cluster just for Argo CD). I am following the Getting Started guide for Argo CD and am running the following command to add Argo CD to my cluster:
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
How do I specify the resources for each pod when I am using someone else's yaml template? The only way I have found to set resource requests is with my own yaml file like this:
apiVersion: v1
kind: Pod
metadata:
name: memory-demo
namespace: mem-example
spec:
containers:
- name: memory-demo-ctr
image: polinux/stress
resources:
limits:
memory: "200Mi"
requests:
memory: "100Mi"
But I don't understand how to apply this type of configuration to Argo CD.
Thanks!
So right now you are just using kubectl with the manifest from github and you cannot edit it. What you need to do is
1 Download the file with wget
https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
2 Use an editor like nano or vim to edit the file with requests as
explained in my comments above.
3 Then use kubectl apply -f newfile.yaml
You can dump the yaml of argocd, then customize your resource request, and then apply the modified yaml.
$ kubectl get deployment -n argocd -o yaml > argocd_deployment.yaml
$ kubectl get sts -n argocd -o yaml > argocd_statefulset.yaml
$ # modify resource
$ vim argocd_deployment.yaml
$ vim argocd_statefulset.yaml
$ kubectl apply -f argocd_deployment.yaml
$ kubectl apply -f argocd_statefulset.yaml
Or modify deplopyment and statefulset directly by kubectl edit
$ kubectl edit deployment -n argocd
$ kubectl edit sts -n argocd
I tried to automate the rolling update when the configmap changes are made. But, I am confused about how can I verify if the rolling update is successful or not. I found out the command
kubectl rollout status deployment test-app -n test
But I guess this is used when we are performing the rollback rather than for rolling update. What's the best way to know if the rolling update is successful or not?
I think it is fine,
kubectl rollout status deployments/test-app -n test can be used to verify the rollout deployment as well.
As an additional step, you can run,
kubectl rollout history deployments/test-app -n test as well.
if you need further clarity,
kubectl get deployments -o wide and check the READY and IMAGE fields.
ConfigMap generation and rolling update
I tried to automate the rolling update when the configmap changes are made
It is a good practice to create new resources instead of mutating (update in-place). kubectl kustomize is supporting this workflow:
The recommended way to change a deployment's configuration is to
create a new configMap with a new name,
patch the deployment, modifying the name value of the appropriate configMapKeyRef field.
You can deploy using Kustomize to automatically create a new ConfigMap every time you want to change content by using configMapGenerator. The old ones can be garbage collected when not used anymore.
Whith Kustomize configMapGenerator can you get a generated name.
Example
kind: ConfigMap
metadata:
name: example-configmap-2-g2hdhfc6tk
and this name get reflected to your Deployment that then trigger a new rolling update, but with a new ConfigMap and leave the old unchanged.
Deploy both Deployment and ConfigMap using
kubectl apply -k <kustomization_directory>
When handling change this way, you are following the practices called Immutable Infrastructure.
Verify deployment
To verify a successful deployment, you are right. You should use:
kubectl rollout status deployment test-app -n test
and when leaving the old ConfigMap unchanged but creating a new ConfigMap for the new ReplicaSet it is clear which ConfigMap belongs to which ReplicaSet.
Also rollback will be easier to understand since both old and new ReplicaSet use its own ConfigMap (on change of content).
Your command is fine to check if an update went through.
Now, a ConfigMap change eventually gets applied to the Pod. There is no need to do a rolling update for that. Depending on what you have passed in the ConfigMap, you probably could have restarted the service and that's it.
What's the best way to know if the rolling update is successful or not?
To check if you rolling update was executed correctly, you command works fine, you could check also if you replicas is running.
I tried to automate the rolling update when the configmap changes are made.
You could use Reloader to perform your rolling updates automatically when a configmap/secret changed.
Reloader can watch changes in ConfigMap and Secret and do rolling upgrades on Pods with their associated DeploymentConfigs, Deployments, Daemonsets and Statefulsets.
Let's explore how Reloader works in a pratical way using a nginx deployment as exxample.
First install Reloader in your cluster:
kubectl apply -f https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml
You will see a new container named reloader-reloader-... this container is responsible to 'watch' your deployments and make the rolling updates when necessary.
Create a configMap with your values, in my case I'll create a my-config and set a variable called myvar.value with value hello:
kubectl create configmap my-config --from-literal=myvar.value=hello
Now, let's create a simple deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-example
labels:
app: nginx
metadata:
annotations:
configmap.reloader.stakater.com/reload: my-config
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
env:
- name: MYVAR
valueFrom:
configMapKeyRef:
name: my-config
key: myvar.value
In this example, the nginx image will be used getting the value from my configMap and set in a variable called MYVAR.
To Reloader works, you must specify the name of your configMap in annotations, in the example above it will be:
metadata:
annotations:
configmap.reloader.stakater.com/reload: my-config
Apply the deployment example with kubectl apply -f mydeplyment-example.yaml and check the variable from the new pod.
$ kubectl exec -it $(kubectl get pods -l=app=nginx --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}') env | grep MYVAR
MYVAR=hello
Now let's change the value of the variable:
Edit configmap with kubectl edit configmap my-config, change the value of myvar.value to hi, save and close.
After save, Reloader will recreate your container and get the new value from configmap.
To check if the rolling update was executed successfully:
kubectl rollout status deployment deployment-example
Check the new value:
$ kubectl exec -it $(kubectl get pods -l=app=nginx --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}') env | grep MYVAR
MYVAR=hi
That's all!
Check the Reloader github see more options to use.
I hope it helps!
Mounted ConfigMaps are updated automatically reference
To Check roll-out history used below steps
ubuntu#dlv-k8s-cluster-master:~$ kubectl rollout history deployment busybox
REVISION CHANGE-CAUSE
1 kubectl create --filename=busybox.yaml --record=true
Update image on deployment as below
$ kubectl set image deployment.app/busybox *=busybox --record
deployment.apps/busybox image updated
Check new rollout history which will list the new change cause for rollout
$ kubectl rollout history deployment busybox
REVISION CHANGE-CAUSE
1 kubectl create --filename=busybox.yaml --record=true
2 kubectl set image deployment.app/busybox *=busybox --record=true
To Rollback Deployment : use undo flag along rollout command
ubuntu#dlv-k8s-cluster-master:~$ kubectl rollout undo deployment busybox deployment.apps/busybox rolled back
ubuntu#dlv-k8s-cluster-master:~$ kubectl rollout history deployment busybox
REVISION CHANGE-CAUSE
2 kubectl set image deployment.app/busybox *=busybox --record=true
3 kubectl create --filename=busybox.```
I have created a replicaset with wrong container image with below configuration.
apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
name: rs-d33393
namespace: default
spec:
replicas: 4
selector:
matchLabels:
name: busybox-pod
template:
metadata:
labels:
name: busybox-pod
spec:
containers:
- command:
- sh
- -c
- echo Hello Kubernetes! && sleep 3600
image: busyboxXXXXXXX
name: busybox-container
Pods Information:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
rs-d33393-5hnfx 0/1 InvalidImageName 0 11m
rs-d33393-5rt5m 0/1 InvalidImageName 0 11m
rs-d33393-ngw78 0/1 InvalidImageName 0 11m
rs-d33393-vnpdh 0/1 InvalidImageName 0 11m
After this, i try to edit the image inside replicaset using kubectl edit replicasets.extensions rs-d33393 and update image as busybox.
Now, i am expecting pods to be recreated with proper image as part of replicaset.
This has not been the exact result.
Can someone please explain, why it is so?
Thanks :)
With ReplicaSets directly you have to kill the old pod, so the new ones will be created with the right image.
If you would be using a Deployment, and you should, changing the image would force the pod to be re-created.
Replicaset does not support updates. As long as required number of pods exist matching the selector labels, replicaset's jobs is done. You should use Deployment instead.
https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
From the docs:
To update Pods to a new spec in a controlled way, use a Deployment, as
ReplicaSets do not support a rolling update directly.
Deployment is a higher-level concept that manages ReplicaSets and provides declarative updates to Pods. Therefore, it is recommend to use Deployments instead of directly using ReplicaSets unless you don’t require updates at all. ( i.e. one may never need to manipulate ReplicaSet objects when using a Deployment)
Its easy to perform rolling updates and rollbacks when deployed using deployments.
$ kubectl create deployment busybox --image=busyboxxxxxxx --dry-run -o yaml > busybox.yaml
$ cat busybox.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: busybox
name: busybox
spec:
replicas: 1
selector:
matchLabels:
app: busybox
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: busybox
spec:
containers:
- image: busyboxxxxxxx
name: busyboxxxxxxx
ubuntu#dlv-k8s-cluster-master:~$ kubectl create -f busybox.yaml --record=true
deployment.apps/busybox created
Check rollout history
ubuntu#dlv-k8s-cluster-master:~$ kubectl rollout history deployment busybox
deployment.apps/busybox
REVISION CHANGE-CAUSE
1 kubectl create --filename=busybox.yaml --record=true
Update image on deployment
ubuntu#dlv-k8s-cluster-master:~$ kubectl set image deployment.app/busybox *=busybox --record
deployment.apps/busybox image updated
ubuntu#dlv-k8s-cluster-master:~$ kubectl rollout history deployment busybox
deployment.apps/busybox
REVISION CHANGE-CAUSE
1 kubectl create --filename=busybox.yaml --record=true
2 kubectl set image deployment.app/busybox *=busybox --record=true
Rollback Deployment
ubuntu#dlv-k8s-cluster-master:~$ kubectl rollout undo deployment busybox
deployment.apps/busybox rolled back
ubuntu#dlv-k8s-cluster-master:~$ kubectl rollout history deployment busybox
deployment.apps/busybox
REVISION CHANGE-CAUSE
2 kubectl set image deployment.app/busybox *=busybox --record=true
3 kubectl create --filename=busybox.yaml --record=true
You could use
k scale rs new-replica-set --replicas=0
and then
k scale rs new-replica-set --replicas=<Your number of replicas>
Edit the replicaset(assuming its called replicaset.yaml) file with command:
kubectl edit rs replicaset
edit the image name in the editor
save the file
exit the editor
Now , you will need to either delete the replica sets or delete the existing pods:
kubectl delete rs new-replica-set
kubectl delete pod pod_1 pod_2 pod_2 pod_4
replicaset should spin up new pods with new image.