When my ConfigMap changes, I'd like my CronJob to restart.
There's a trick in Helm to make this happen for Deployments using an annotation that contains the SHA of the ConfigMap (see here or here).
This works for my Deployment, doing kubectl get deployments show the AGE is just a couple minutes.
But not my CronJob, doing kubectl get cronjobs, the "AGE" is still hours old implying they haven't restarted.
So, it turns out CronJobs don't need to be restarted when the ConfigMap changes anyway. Everytime a Job is created, the Pod that spins up to execute the Job seems to use the latest ConfigMap values anyway.
Related
A pod can be created by Deployment or ReplicaSet or DaemonSet, if I am updating a pod's container specs, is it OK for me to simply modify the yaml file that created the pod? Would it be erroneous once I have done that?
Brief Question:
Is kubectl apply -f xxx.yml the silver bullet for all pod update?
...if I am updating a pod's container specs, is it OK for me to simply modify the yaml file that created the pod?
The fact that the pod spec is part of the controller spec (eg. deployment, daemonset), to update the container spec you naturally start with the controller spec. Also, a running pod is largely immutable, there isn't much you can change directly unless you do a replace - which is what the controller already doing.
you should not make changes to the pods directly, but update the spec.template.spec section of the deployment used to create the pods.
reason for this is that the deployment is the controller that manages the replicasets and therefore the pods that are created for your application. that means if you apply changes to the pods manifest directly, and something like a pod rescheduling/restart happens, the changes made to the pod will be lost because the replicaset will recreate the pod according to its own specification and not the specification of the last running pod.
you are safe to use kubectl apply to apply changes to existing resources but if you are unsure, you can always extract the current state of the deployment from kubernetes and pipe that output into a yaml file to create a backup:
kubectl get deploy/<name> --namespace <namespace> -o yaml > deploy.yaml
another option is to use the internal rollback mechanism of kubernetes to restore a previous revision of your deployment. see https://learnk8s.io/kubernetes-rollbacks for more infos on that.
I have a rather peculiar use case. Specifically, prior to the deployment of my statefulset I am deploying a ConfigMap which contains an environment variable setting (namely RECREATE_DATADIR) which instructs the pod's container to create a new data structure on the file system.
However, during the typical lifetime of the container the data structure should NOT be recreated. Hence, right after the pod is successfully running, I am changing the ConfigMap and then reapply it. Hence - if the pod ever fails, it won't recreate the data directory structure when it respawns.
How can I achieve this same result using Helm charts?
You can create a job as part of your helm chart, with the post-install helm hook which will have configmap edit permissions, will use a kubectl image (bitnami/kubectl for example), and it will patch the configmap to false using kubectl commands.
Deployment resource object is still not supported in our cluster and not enabled.
We are using Pod resource object Yaml file. something like below:
apiVersion: v1
kind: Pod
metadata:
name: sample-test
namespace: default
spec:
automountServiceAccountToken: false
containers:
I have explored patch and Put rest api for Pod(Kubectl patch and replace) - it will update to new image version and pod restarts.
I need help in below:
When the image version is same, it will not update and pod will not restart.
How can i acheive Pod restart, is there any API for this or any alternate
approach for this. Because My pod also refers configmap and secret. After i
make changes to secret, i want to restart pod so that it can take updated
value.
Suppose when patch applied with new container image and it fails status is failed, I want to rollback to previous version, How can i acheive this with standalone pod without using deployment. Is there any alternate approach.
Achieving solutions for your scenario, can be handled like this:
When the image version is same, it will not update and pod will not restart. How can i acheive Pod restart, is there any API for this or any alternate approach for this. Because My pod also refers configmap and secret. After i make changes to secret, i want to restart pod so that it can take updated value
Create a new secret/configmap each time and update the pod yaml to use the new configmap/secret rather than the old name.
Suppose when patch applied with new container image and it fails status is failed, I want to rollback to previous version, How can i acheive this with standalone pod without using deployment. Is there any alternate approach
Before you do a Pod update, get the current Pod yaml using kubectl like this,
kubectl get pod <pod-name> -o yaml -n <namespace>
After getting the yaml, generate the new pod yaml and apply it. In case of failure, clean up the new resources created(configmaps & secrets) and apply the older version of pod to achieve rollback
On an occasional basis I need to perform a rolling replace of all Pods in my StatefulSet such that all PVs are also recreated from scratch. The reason to do so is to get rid of all underlying hard drives that use old versions of encryption key. This operation should not be confused with regular rolling upgrades, for which I still want volumes to survive Pod terminations. The best routine I figured so far to do that is following:
Delete the PV.
Delete the PVC.
Delete the Pod.
Wait until all deletions complete.
Manually recreate the PVC deleted in step 2.
Wait for the new Pod to finish streaming data from other Pods in the StatefulSet.
Repeat from step 1. for the next Pod.
I'm not happy about step 5. I wish StatefulSet recreated the PVC for me, but unfortunately it does not. I have to do it myself, otherwise Pod creation fails with following error:
Warning FailedScheduling 3s (x15 over 15m) default-scheduler persistentvolumeclaim "foo-bar-0" not found
Is there a better way to do that?
I just recently had to do this. The following worked for me:
# Delete the PVC
$ kubectl delete pvc <pvc_name>
# Delete the underlying statefulset WITHOUT deleting the pods
$ kubectl delete statefulset <statefulset_name> --cascade=false
# Delete the pod with the PVC you don't want
$ kubectl delete pod <pod_name>
# Apply the statefulset manifest to re-create the StatefulSet,
# which will also recreate the deleted pod with a new PVC
$ kubectl apply -f <statefulset_yaml>
This is described in https://github.com/kubernetes/kubernetes/issues/89910. The workaround proposed there, of deleting the new Pod which is stuck pending, works and the second time it gets replaced a new PVC is created. It was marked as a duplicate of https://github.com/kubernetes/kubernetes/issues/74374, and reported as potentially fixed in 1.20.
It seems like you're using "Persistent" volume in a wrong way. It's designed to keep the data between roll-outs, not to delete it. There are other different ways to renew the keys. One can use k8s Secret and ConfigMap to mount the key into the Pod. Then you just need to recreate a Secret during a rolling update
I am modify config maps environment from DEV to FAT, and now I want to make it works in all my pods in dabai-fat name space.How to restart all pods in the namespace? If I modify one by one it is too slow, and my deployment service have more than 20 now. How to enable the config the easy way?
You should prefer mounted config maps for your solution where you will not need POD restart.
Kubelet is checking whether the mounted ConfigMap is fresh on every periodic sync.
Total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the pod can be as long as kubelet sync period (1 minute by default) + ttl of ConfigMaps cache (1 minute by default) in kubelet. You can trigger an immediate refresh by updating one of the pod’s annotations. Important to remember that container using a ConfigMap as a subPath volume will not receive ConfigMap updates.
How to Add ConfigMap data to a Volume
You should not edit already existing ConfigMap.
This question Restart pods when configmap updates in Kubernetes? is the best possible answer to your question.
First, use Deployments so it's easy to scale everything.
Second, create new ConfigMap and point Deployment to it. If new ConfigMap is broken the Deployment won't scale and if it's correct, the Deployment will scale to 0 and reschedule new pods that will be using new ConfigMap.