This question already has answers here:
Kubernetes: kubectl apply does not update pods when using "latest" tag
(4 answers)
Closed 3 years ago.
Problem
A frequent question that comes up on Slack and Stack Overflow is how to trigger an update to a Deployment/RS/RC when the image tag hasn't changed but the underlying image has.
Consider:
There is an existing Deployment with image foo: latest
The user builds a new image foo: latest
The user pushes foo: latest to their registry
The user wants to do something here to tell the Deployment to pull the new image and do a rolling-update of existing pods
For Example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: worker-deployment
spec:
replicas: 1
selector:
matchLabels:
component: worker
template:
metadata:
labels:
component: worker
spec:
containers:
- name: worker
image: stephengrider/multi-worker
env:
- name: REDIS_HOST
value: redis-cluster-ip-service
- name: REDIS_PORT
value: "6379"
And now I have made changes in my multiWorker image
and redeploys the deployment, but it doesn't fetch the latest image?
The default pull policy is IfNotPresent which causes the Kubelet to skip pulling an image if it already exists. Refer Link
If you would like to always force a pull, you can do one of the following:
1: set the imagePullPolicy of the container to Always.
2: omit the imagePullPolicy and use :latest as the tag for the image to use.
3: omit the imagePullPolicy and the tag for the image to use.
4: enable the AlwaysPullImages admission controller.
Note that you should avoid using :latest tag, we should avoid using the :latest tag when deploying containers in production as it is harder to track which version of the image is running and more difficult to roll back properly.
Various imagePullPolicy and the tag of the image affect when the kubelet attempts to pull the specified image listed below. Refer Link for Details
1) imagePullPolicy: IfNotPresent: the image is pulled only if it is not already present locally.
2) imagePullPolicy: Always: the image is pulled every time the pod is started.
3) imagePullPolicy is omitted and either the image tag is :latest or it is omitted: Always is applied.
4) imagePullPolicy is omitted and the image tag is present but not :latest: IfNotPresent is applied.
5) imagePullPolicy: Never: the image is assumed to exist locally. No attempt is made to pull the image.
Related
I want that every time I create a new image with the tag latest Kubernetes automatically pull the new image. I added imagePullPolicy: Always in pod spec but it doesn't update the old image with new image.
apiVersion: apps/v1
kind: Deployment
metadata:
name: node
namespace: dev
labels:
app: my-node-app
spec:
replicas: 2
selector:
matchLabels:
app: my-node-app
template:
metadata:
labels:
app: my-node-app
spec:
hostNetwork: true
securityContext:
fsGroup: 1000
containers:
- name: node
imagePullPolicy: Always
image: gcr.io/my-repo/my-node-app:latest
ports:
- containerPort: 3000
envFrom:
- configMapRef:
name: my-configmap
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 2
memory: 8Gi
restartPolicy: Always
imagePullPolicy is only taken into account by Kubernetes when a POD is created or re-started. It is NOT taken into account while a POD is running, which means it does NOT check for image updates at any time while a POD is running.
Even if another POD with the same image would be scheduled onto the same Kubernetes node, the already running POD is not affected, even though Kubernetes does a pull and then uses the new image for the new POD.
If you want the desired functionality, you will have to implement your own solution. You could do this by implementing a sidecar that regularly checks the Docker Repository for changes to the given tag. When it detects such a change, it could trigger a restart of the POD, which would then force the image to be re-pulled.
A restart of the POD can either be triggered by simply exiting the sidecar or by utilizing the Kubernetes API inside the sidecar. The latter solution however gets more complicated as you will also need service accounts and RBAC rules to get proper permissions inside the sidecar container. It also has security implications you'd have to give the whole POD escalated permissions.
Setting imagePullPolicy: Always does not mean an image will be pulled automatically without any trigger.
I would recommend to use tagged image with semvar. Since you are using deployment you can perform rolling update of the pod which will pull new image and rollout the change across all the replica pod one by one in a graceful way without causing any downtime.
Let's say initially the image is gcr.io/my-repo/my-node-app:v1 and you want to update it to v2
kubectl set image deployment/node nginx=gcr.io/my-repo/my-node-app:v2 --record
Check the rollout history
kubectl rollout history deployment.v1.apps/node
In case of any issue rollback to previous version
kubectl rollout undo deployment.v1.apps/node
Also if you want to be more advanced you could do GitOps using FluxCD which supports triggering a rollout automatically whenever a new version of an image is pushed to a container registry.
Kubernetes will pull image only upon Pod creation which means it does not check for image updates while a POD is in running state.
I would recommend to use Semantic Versioning for the image tag and use a CI/CD pipeline which build, tag, and push to your registry. Then use a CD tool such as keel to re-create your pods in the last step of the pipeline.
I have an image pushed to Google Container Registry, named gcr.io/$(PROJECT_ID)/img-name:46d49ab.
In my replication controller I have:
apiVersion: v1
kind: ReplicationController
metadata:
name: go-server-rc
spec:
replicas: 3
selector:
name: go-server
version: v8
template:
metadata:
labels:
name: go-server
version: v8
spec:
containers:
- name: go-server
image: gcr.io/$(PROJECT_ID)/img-name:46d49ab
ports:
- containerPort: 5000
This works, but it doesn't when I remove the commit hash tag 46d49ab. I don't want to have to change the tag every single time I commit.
I have also set up a trigger on Google Container Builder to pull the master branch of my repository after every commit, and create an image gcr.io/$(PROJECT_ID)/img-name:$(COMMIT_HASH).
How can I edit my replication controller file to just get the most recent? What workflows do people use?
It's possible to use the latest tag to ensure Kubernetes pulls the image each time it runs. Every time you create a new image tag it with latest and push it to the container registry. However, I would not recommend this.
You won't know which pods are running which version of your code. I do exactly as you mention in your question. I find it's better to update your deployment object each time your image updates. This will ensure the deployment is in the state you expect and troubleshooting will be clearer when looking at images.
I'm using a CI to update my kubernetes cluster whenever there's an update to an image. Whenever the image is pushed and has the latest tag it kubectl apply's the existing deployment but nothing gets updated.
this is what runs
$ kubectl apply --record --filename /tmp/deployment.yaml
My goal is when the apply is ran that a rolling deployment gets executed.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: api
spec:
replicas: 1
template:
metadata:
labels:
app: api
spec:
containers:
- name: api
image: us.gcr.io/joule-eed41/api:latest
imagePullPolicy: Always
ports:
- containerPort: 1337
args:
- /bin/sh
- -c
- echo running api;npm start
env:
- name: NAMESPACE
valueFrom:
configMapKeyRef:
name: config
key: NAMESPACE
As others suggested, have a specific tag.
Set new image using following command
kubectl set image deployment/deployment_name deployment_name=image_name:image_tag
In your case it would be
kubectl set image deployment/api api=us.gcr.io/joule-eed41/api:0.1
As #ksholla20 mentionedm using kubectl set image is a good option for many (most?) cases.
But if you can't change the image tag consider using:
1 ) kubectl rollout restart deployment/<name>
(reference).
2 ) kubectl patch deployment <name> -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"version\":\"$CURRENT_BUILD_HASH_OR_DATE\"}}}}}}" (reference)
(*) Notice that the patch command allow you to change specific properties in the deployment (or any other object chosen) like the label selector and the pod label or other properties like the value of the NAMESPACE environment variable in your example.
I've run into the same problem and none of the solutions posted so far will help. The solution is easy, but not easy to see or predict. The applied yaml will generate both a deployment and a replicaset the first time it's run. Unfortunately, applying changes to the manifest likely only replaces the replicaset, while the deployment will remain unchanged. This is a problem because some changes need to happen at the deployment level, but the old deployment hangs around. To have best results, delete the deployment and ensure all previous deployments and replicasets are deleted. Then apply the updated manifest.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: test
spec:
replicas: 1
template:
metadata:
labels:
app: test
spec:
containers:
- name: test
image: xxx:latest
ports:
- containerPort: 80
imagePullPolicy: Always
imagePullSecrets:
- name: aaaa
I use the tag "latest". When I update the image,and the new image is still "latest". When I "kubectl set image deployments/test test=xxx:latest",nothing happend. What should I do?
a RollingUpdate is always triggered when the PodTemplateSpec under template is changed.
While using the :latest tag is not suggested it can still work when using imagePullPolicy: Always and a label which is changed with every image adjustment. Sth like this:
kubectl patch deployment test -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"$(date +%s)\"}}}}}"
Rolling update depends on docker image tags. If you're using the latest tag in your deployment, you need to cut a new image with a new version.
The deployment resource can't determine if the images has changed if you're always using the latest tag. As far as k8s is concerned, you're already running an image with the tag latest, so it doesn't have anything to do.
It's highly recommended not to use latest for deployments for this reason. You'll have a much easier time if you version your docker images correctly.
I have a Replication Controller with one replica using a secret. How can I update or recreate its (lone) pod—without downtime—with latest secret value when the secret value is changed?
My current workaround is increasing number of replicas in the Replication Controller, deleting the old pods, and changing the replica count back to its original value.
Is there a command or flag to induce a rolling update retaining the same container image and tag? When I try to do so, it rejects my attempt with the following message:
error: Specified --image must be distinct from existing container image
A couple of issues #9043 and #13488 describe the problem reasonably well, and I suspect a rolling update approach will eventuate shortly (like most things in Kubernetes), though unlikely for 1.3.0. The same issue applies with updating ConfigMaps.
Kubernetes will do a rolling update whenever anything in the deployment pod spec is changed (eg. typically image to a new version), so one suggested workaround is to set an env variable in your deployment pod spec (eg. RESTART_)
Then when you've updated your secret/configmap, bump the env value in your deployment (via kubectl apply, or patch, or edit), and Kubernetes will start a rolling update of your deployment.
Example Deployment spec:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: test-nginx
spec:
replicas: 2
template:
metadata:
spec:
containers:
- name: nginx
image: "nginx:stable"
ports:
- containerPort: 80
- mountPath: /etc/nginx/conf.d
name: config
readOnly: true
- mountPath: /etc/nginx/auth
name: tokens
readOnly: true
env:
- name: RESTART_
value: "13"
volumes:
- name: config
configMap:
name: test-nginx-config
- name: tokens
secret:
secretName: test-nginx-tokens
Two tips:
your environment variable name can't start with an _ or it magically disappears somehow.
if you use a number for your restart variable you need to wrap it in quotes
If I understand correctly, Deployment should be what you want.
Deployment supports rolling update for almost all fields in the pod template.
See http://kubernetes.io/docs/user-guide/deployments/