I'm running Kubernetes in a GKE cluster and need to run a DB migration script on every deploy. For staging this is easy: we have a permanent, separate MySQL service with its own volume. For production however we make use of GCE SQL, resulting in the job having two containers - one more for the migration, and the other for Cloud Proxy.
Because of this new container, the job always shows as 1 active when running kubectl describe jobs/migration and I'm at a complete loss. I have tried re-ordering the containers to see if it checks one by default but that made no difference and I cannot see a way to either a) kill a container or b) check the status of just one container inside the Job.
Any ideas?
I know it's a year too late, but best practice would be to run single cloudsql proxy service for all app's purposes, and then configure DB access in app's image to use this service as a DB hostname.
This way you will not require putting cloudsql proxy container into every pod which uses DB.
each Pod can be configured with a init container which seems to be a good fit for your issue. So instead of having a Pod with two containers which have to run permanently, you could rather define a init container to do your migration upfront. E.g. like this:
apiVersion: v1
kind: Pod
metadata:
name: init-container
annotations:
pod.beta.kubernetes.io/init-containers: '[
{
"name": "migrate",
"image": "application:version",
"command": ["migrate up"],
}
]'
spec:
containers:
- name: application
image: application:version
ports:
- containerPort: 80
You haven't posted enough details about your specific problem. But I'm taking a guess based on experience.
TL;DR: Move your containers into separate jobs if they are independent.
--
Kubernetes jobs keep restarting till the job succeeds.
A kubernetes job will succeed only if every container within succeeds.
This means that your containers should be return in a restart proof way. Once a container sucessfully runs, it should return a success even if it runs again. Otherwise, say container1 is successful, container2 fails. Job restarts. Then, container1 fails (because it has already been successful). Hence, Job keeps restarting.
The reason is the container/process never terminates.
One possible work around is: move the cloud-sql-proxy to it's own deployment - and add a service in front of that. Hence your job won't be responsible for running the long running cloud-sql-proxy and hence will terminate / complete.
Related
Does anyone know why restartPolicy field is defined on the Pod level instead of the container level?
It would seem that this setting is more closely related to the container, not the Pod.
Then how to controll restart policy of single container in multi-container Pod?
I think restart policy is part of the POD spec.
apiVersion: v1
kind: Pod
metadata:
name: test
spec:
containers:
- name: 1st
image: image-1
command: ["./bash", "-test1"]
- name: 2nd
image: image-2
command: ["./bash", "-test2"]
restartPolicy: Never
Restart policy gets set the at POD spec level, and get applied to all the containers in POD even if init container is there.
If there are multi containers inside the POD, we have to consider those as tightly coupled.
Official documents says something like this : link
Pods that run multiple containers that need to work together. A Pod can encapsulate an application composed of multiple co-located
containers that are tightly coupled and need to share resources. These
co-located containers form a single cohesive unit of service—for
example, one container serving data stored in a shared volume to the
public, while a separate sidecar container refreshes or updates those
files. The Pod wraps these containers, storage resources, and an
ephemeral network identity together as a single unit.
Note: Grouping multiple co-located and co-managed containers in a
single Pod is a relatively advanced use case. You should use this
pattern only in specific instances in which your containers are
tightly coupled.
If you want to restart the single container in POD you won't be able to do it, you have keep that container out of POD then by POD design.
Even if you will see the container restart policy it's talk about the : POD spec restart policy only.
I have an application that I want to scale in parallel but would love a way to have a brief pause between each new pod getting created.
I need this because during pod creation I update a file, and if the file isn't closed by the time the next pod tries to open it and update it, it will fail.
So I just need a 1-second buffer between pods opening and closing this file.
Right now if a scale happens and more than 1 new pod is added, they hit the file at the same time. So one pod will work, the other will fail, and I have to wait for the pod to timeout for k8s to kill it and recreate it, at which point it will work, but wastes a lot of valuable time during scaling events when I need new pods as quickly as possible to handle the load.
Not sure if I'm not wording my searches correctly, but not able to find this on the k8s website. Any ideas would be helpful.
(additional note, I know on StatefulSet it will scale 1 pod at a time by default, However, that method requires all pods to be healthy to continue to add that one node at a time, and if any pod becomes unhealthy it will stop scaling until all pods are healthy again. Which during high load situations, wouldn't be ideal)
You will need to do some work in order to achieve it.
Few "hacks/solutions"
1. Init Container
Add an init container to your deployment
The init container will write/delete a destination file like lock.txt
The init container will check to see if this file exist, if so he will wait until the file is removed
Once the file is removed the init container will exit so the pod will "start"
2. Probe 'PostStart'
Add a PostStart probe to your pods which will read the current number of the desired pods vs the actual ones and if required execute a scale
apiVersion: v1
kind: Pod
metadata:
name: lifecycle-demo
spec:
containers:
- name: ...
image: ...
lifecycle:
postStart:
exec:
command: [<get the current number of pods from the replica>
<check to see if you need to restart a new pod>
<scale a new pod if required>]
3. CronJob
Scale your pods with CronJob, Use a file or config map to set the desired number of pods.
Apply a Cron job that scales your pods and when you reached the desired pods remove the CronJob
I have a very simple docker-compose app with nginx and php-fpm services.
I would like to create a Helm chart that would deploy this exact same thing as a unit (a POD I guess?) but I would like to be able to scale the php-fpm service.
How can I achieve that?
My idea is to have these containers in a single deployment so I don't have a lot of deployments scattered.
i.e
App1:
- Container 1 (php-fpm autoscaled or manually scaled)
- Container 2 (nginx)
- Container 3 (Redis)
- Container 4 (something else that this app needs)
App2
- Container 1
- Container 2
- Container 3 etc...
I could do different deployments but then it would be like
app1_php-fpm
app1_nginx
app1_redis
app2_container1
app2_container2
app2_container3
Instead of a single pod.
The "one-container-per-Pod" model is the most common Kubernetes use
case; in this case, you can think of a Pod as a wrapper around a
single container; Kubernetes manages Pods rather than managing the
containers directly.
Nginx, Redis should run as separate services, so that they can scale independent of your application in a decoupled manner.
Nginx can reverse proxy multiple applications so you don't need to have a separate Nginx instance for every app.
https://kubernetes.io/docs/concepts/workloads/pods/#using-pods
It's best practice to keep the single container inside the pod however we can run the multiple containers also.
Keep the Redis & Nginx containers as separate deployment and you can add the HPA on that they can scale up and down based on load.
By keeping the deployment separate would also help in doing any maintenance if required or deployment without downtime.
However, if you don't have an option to keep Nginx aside you can keep that also with fpm container as it's also a light weight container.
For example here I have one fpm and Nginx containers running inside single POD : https://github.com/harsh4870/Kubernetes-wordpress-php-fpm-nginx/blob/master/wordpress-deployment.yaml
if possible divide all the applications as a single deployment so that it will be very easy to manage the deployments and maintenance and scaling also.
Before I start all the services I need a pod that would do some initialization. But I dont want this pod to be running after the init is done and also all the other pods/services should start after this init is done. I am aware of init containers in a pod, but I dont think that would solve this issue, as I want the pod to exit after initialization.
You are recommended to let Kubernetes handle pods automatically instead of manual management by yourself, when you can. Consider Job for run-once tasks like this:
apiVersion: batch/v1
kind: Job
metadata:
name: myjob
spec:
template:
spec:
restartPolicy: Never
containers:
- name:
image:
$ kubectl apply -f job.yml
Kubernetes will create a pod to run that job. The pod will complete as soon as the job exits. Note that the completed job and its pod will be released from consuming resources but not removed completely, to give you a chance to examine its status and log. Deleting the job will remove the pod completely.
Jobs can do advanced things like restarting on failure with exponential backoff, running tasks in parallelism, and limiting the time it runs.
It depend on the Init task, but the init container is the best option you have (https://kubernetes.io/docs/concepts/workloads/pods/init-containers/).
Kubernetes will create the initContainer before the other container, and when it accomplished it's task, it will exit.
Make the InitContainer exit gracefully (with a code 0) so that k8s will understand that the container completed the task it was meant to do and do no try to restart it.
You init task will be done, and the container will no longer exist
You can try attaching handlers to Container lifecycle events. Kubernetes supports the postStart and preStop events. Kubernetes sends the postStart event immediately after a Container is started, and it sends the preStop event immediately before the Container is terminated.
https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/
https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
I have a container that runs some data fetching from a MySQL database and simply displays the result in console.log(), and want to run this as a cron job in GKE. So far I have the container working on my local machine, and have successfully deployed this to GKE (in terms of there being no errors thrown so far as I can see).
However, the pods that were created were just left as Running instead of stopping after completion of the task. Are the pods supposed to stop automatically after executing all the code, or do they require explicit instruction to stop and if so what is the command to terminate a pod after creation (by the Cron Job)?
I'm reading that there is supposedly some kind of termination grace period of ~30s by default, but after running a minutely-executed cronjob for ~20minutes, all the pods were still running. Not sure if there's a way to terminate the pods from inside the code, otherwise it would be a little silly to have a cronjob generating lots of pods left running idly..My cronjob.yaml below:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: test
spec:
schedule: "5 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: test
image: gcr.io/project/test:v1
# env:
# - name: "DELAY"
# value: 15
restartPolicy: OnFailure
A CronJob is essentially a cookie cutter for jobs. That is, it knows how to create jobs and execute them at a certain time. Now, that being said, when looking at garbage collection and clean up behaviour of a CronJob, we can simply look at what the Kubernetes docs have to say about this topic in the context of jobs:
When a Job completes, no more Pods are created, but the Pods are not deleted either. Keeping them around allows you to still view the logs of completed pods to check for errors, warnings, or other diagnostic output. The job object also remains after it is completed so that you can view its status. It is up to the user to delete old jobs after noting their status. Delete the job with kubectl (e.g. kubectl delete jobs/pi or kubectl delete -f ./job.yaml).
Adding a process.kill(); line in the code to explicitly end the process after the code has finished executing allowed the pod to automatically stop after execution
A job in Kubernetes is intended to run a single instance of a pod and ensure it runs to completion. As another answer specifies, a CronJob is a factory for Jobs which knows how and when to spawn a job according to the specified schedule.
Accordingly, and unlike a service which is intended to run forever, the container(s) in the pod created by the pod must exit upon completion of the job. There is a notable problem with the sidecar pattern which often requires manual pod lifecycle handling; if your main pod requires additional pods to provide logging or database access, you must arrange for these to exit upon completion of the main pod, otherwise they will remain running and k8s will not consider the job complete. In such circumstances, the pod associated with the job will never terminate.
The termination grace period is not applicable here: this timer applies after Kubernetes has requested that your pod terminate (e.g. if you delete it). It specifies the maximum time the pod is afforded to shutdown gracefully before the kubelet will summarily terminate it. If Kubernetes never considers your job to be complete, this phase of the pod lifecycle will not be entered.
Furthermore, old pods are kept around after completion for some time to allow perusal of logs and such. You may see pods listed which are not actively running and so not consuming compute resources on your worker nodes.
If your pods are not completing, please provide more information regarding the code they are running so we can assist in determining why the process never exits.