I have a pod which embed an initContainer named initdb. Is there a kubectl command which returns true if initdb is started or else false? I need it to display logs of initdb in Github Action CI (kubectl log <pod> -c initdb crashes if initd is not started yet).
If you have a single init container in the Pod, you could so something like the following:
k get pod pod-name --output="jsonpath={.status.initContainerStatuses[0].ready}"
This will return true if the init container is in status Ready, but this means only that the init container is ready, it could be already terminated (because it completed the execution) or still running. I'm not completely sure but if an init container is ready, requesting its logs should work without errors)
You can use jsonpath to select specific sections of Pods definitions exactly for the scope of automating certain checks.
To see the full definition of your Pod, just use:
k get pod pod-name -oyaml
and maybe select what you are interested from there. If you want to wait until the init container is terminated or started, you could check its state section which explains the current state in detail and basically create a more finer check on what you are expecting to see.
Related
I know I can use kubectl wait to check if a pod is Ready but is there an easy way to check whether the pod is gone or in Terminating state? I'm running some tests and I only want to continue when the pod (or the namespace for that matter) is completely gone.
Also a timeout option would come in handy.
It's actually part of the wait command.
kubectl wait --for=delete pod/busybox1 --timeout=60s
You can check with kubectl wait --help to see this example and some more. For example
--for='': The condition to wait on: [delete|condition=condition- name|jsonpath='{JSONPath expression}'=JSONPath
Condition]. The default status value of condition-name is true, you > can set false with condition=condition-name=false.
If you execute kubectl delete po' <pod name>, the command will automatically wait until the pod is deleted. This is thanks to the finalizers feature that keeps the resource (the Pod in this case) from being deleted until the dependent resources (the containers of the pod for example) are cleaned up by the kubelet.
I have a Deployment in k8s which contains a single container, which exits when it completes its work. By default, k8s then restarts the container.
I would like to recreate the container (or the whole Pod) on exit instead. The work involves a number of temporary files and other changes in the container, which should be discarded when it exits and a fresh container created from the image.
How can I configure the Deployment (or another workload) so that an exited container is either recreated, or causes the whole pod to exit and be recreated?
There is a restartPolicy setting, but it's forced to Always for Deployments. There appears to be a maxRetries setting, but I can't find the documentation or any examples for it. I'm not sure what else to search for.
Using Job API Object may help. If you use restartPolicy=Never in a Job then the Pods managed by the Job will restart each time the process running in the Pod's container exits with error code. Using restartPolicy=Always will only restart/recreate the container without recreating the pod.
In addition, when a container restart in Kubernetes, it means it is re-created.: all the file created in the container are removed, except the ones which where created on attached persistent storage (for example PersistenceVolume). There is no equivalent to docker start/stop in Kubernetes.
Pods on our k8s cluster are scheduled with Airflow's KubernetesExecutor, which runs all Tasks in a new pod.
I have a such a Task for which the pod instantly (after 1 or 2 seconds) crashes, and for which of course I want to see the logs.
This seems hard. As soon the pod crashes, it gets deleted, along with the ability to retrieve crash logs. I already tried all of:
kubectl logs -f <pod> -p: cannot be used since these pods are named uniquely
(courtesy of KubernetesExecutor).
kubectl logs -l label_name=label_value: I
struggle to apply the labels to the pod (if this is a known/used way of working, I'm happy to try further)
An shared nfs is mounted on all pods on a fixed log directory. The failing pod however, does not log to this folder.
When I am really quick I run kubectl logs -f -l dag_id=sample_dag --all-containers (dag_idlabel is added byAirflow)
between running and crashing and see Error from server (BadRequest): container "base" in pod "my_pod" is waiting to start: ContainerCreating. This might give me some clue but:
these are only but the last log lines
this is really backwards
I'm basically looking for the canonical way of retrieving logs from transient pods
You need to enable remote logging. Code sample below is for using S3. In airflow.cfg set the following:
remote_logging = True
remote_log_conn_id = my_s3_conn
remote_base_log_folder = s3://airflow/logs
The my_s3_conn can be set in airflow>Admin>Connections. In the Conn Type dropdown, select S3.
I would like to suspend the main process in a docker container running in a kubernetes pod. I have attempted to do this by running
kubectl exec <pod-name> -c <container-name> kill -STOP 1
but the signal will not stop the container. Investigating other approaches, it looks like docker stop --signal=SIGSTOP or docker pause might work. However, as far as I know, kubectl exec always runs in the context of a container, and these commands would need to be run in the pod outside the context of the container. Does kubectl's interface allow for anything like this? Might I achieve this behavior through a call to the underlying kubernetes API?
You could set the replicaset to 0 which would set the number of working deployments to 0. This isn't quite a Pause but it does Stop the deployment until you set the number of deployments to >0.
kubectl scale --replicas=0 deployment/<pod name> --namespace=<namespace>
So kubernetes does not support suspending pods because it's a VM kinda behavior, and since starting a new one is cheaper it just schedules a new pod in case of failure. In effect your pods should be stateless. And any application that needs to store state, should have a persistent volume mounted inside the pod.
The simple mechanics(and general behavior) of Kubernetes is if the process inside the contaiener fails kuberentes will restart it by creating a new pod.
If you also comment what you are trying to achieve as an end goal I think I can help you better.
Container getting killed at node after pod creation
Issue was raised at github and asked me to move to SO
https://github.com/kubernetes/kubernetes/issues/24241
However i am briefing my issue here.After creating pod it doesnt run since i have to mention the container name in the kubelet args under --pod-infra-container-image as mentioned below.
I have solved the issue of Pods Status Container Creating by adding the container name in "--pod-infra-container-image= then pod creation was successful.
However I want to resolve this issue some other way instead of adding containers name in kubelet args. Kindly let me know how do I get this issue fixed.
Also after the pod creation is done. The containers keep on restarting. However if I check the logs via kubectl logs output shows the container expected output.
But the container restarts often. For restarting of pod what i did i have made the restartPolicy: never in spec file of pod and then it didnt restarted however container doesnt run. Kindly help me.
Your description is very confusing, can you please reply to this answer with:
1. What error you get when you do docker pull gcr.io/google_containers/pause:2.0
2. What cmd you're running in your container
For 2 you need a long running command, like while true; do echo SUCCESS; done, otherwise it'll just exit and get restarted by the kubelet with a RestartPolicy of Always.