I'm wondering for a batch distributed job I need to run. Is there a way in K8S if I use a Job/Stateful Set or whatever, a way for the pod itself(via ENV var or whatever) to know its 1 of X pods run for this job?
I need to chunk up some data and have each process fetch the stuff it needs.
--
I guess the statefulset hostname setting is one way of doing it. Is there a better option?
This is planned but not yet implemented that I know of. You probably want to look into higher order layers like Argo Workflows or Airflow instead for now.
You could write some infrastructure as code using Ansible that will perform the following tasks in order:
kubectl create -f jobs.yml
kubectl wait --for=condition=complete job/job1
kubectl wait --for=condition=complete job/job2
kubectl wait --for=condition=complete job/job3
kubectl create -f pod.yml
kubectl wait can be used in situations like this to halt progress until an action has been performed. In this case, a job has completed its run.
Here is a similar question that someone asked on StackOverflow before.
Related
What would be the best way to set up a GCP monitoring alert policy for a Kubernetes CronJob failing? I haven't been able to find any good examples out there.
Right now, I have an OK solution based on monitoring logs in the Pod with ERROR severity. I've found this to be quite flaky, however. Sometimes a job will fail for some ephemeral reason outside my control (e.g., an external server returning a temporary 500) and on the next retry, the job runs successfully.
What I really need is an alert that is only triggered when a CronJob is in a persistent failed state. That is, Kubernetes has tried rerunning the whole thing, multiple times, and it's still failing. Ideally, it could also handle situations where the Pod wasn't able to come up either (e.g., downloading the image failed).
Any ideas here?
Thanks.
First of all, confirm the GKE’s version that you are running. For that, the following commands are going to help you to identify the GKE’s
default version and the available versions too:
Default version.
gcloud container get-server-config --flatten="channels" --filter="channels.channel=RAPID" \
--format="yaml(channels.channel,channels.defaultVersion)"
Available versions.
gcloud container get-server-config --flatten="channels" --filter="channels.channel=RAPID" \
--format="yaml(channels.channel,channels.validVersions)"
Now that you know your GKE’s version and based on what you want is an alert that is only triggered when a CronJob is in a persistent failed state, GKE Workload Metrics was the GCP’s solution that used to provide a fully managed and highly configurable solution for sending to Cloud Monitoring all Prometheus-compatible metrics emitted by GKE workloads (such as a CronJob or a Deployment for an application). But, as it is right now deprecated in GKE 1.24 and was replaced with Google Cloud Managed Service for Prometheus, then this last is the best option you’ve got inside of GCP, as it lets you monitor and alert on your workloads, using Prometheus, without having to manually manage and operate Prometheus at scale.
Plus, you have 2 options from the outside of GCP: Prometheus as well and Ranch’s Prometheus Push Gateway.
Finally and just FYI, it can be done manually by querying for the job and then checking it's start time, and compare that to the current time, this way, with bash:
START_TIME=$(kubectl -n=your-namespace get job your-job-name -o json | jq '.status.startTime')
echo $START_TIME
Or, you are able to get the job’s current status as a JSON blob, as follows:
kubectl -n=your-namespace get job your-job-name -o json | jq '.status'
You can see the following thread for more reference too.
Taking the “Failed” state as the medullary point of your requirement, setting up a bash script with kubectl to send an email if you see a job that is in “Failed” state can be useful. Here I will share some examples with you:
while true; do if `kubectl get jobs myjob -o jsonpath='{.status.conditions[?(#.type=="Failed")].status}' | grep True`; then mail email#address -s jobfailed; else sleep 1 ; fi; done
For newer K8s:
while true; do kubectl wait --for=condition=failed job/myjob; mail#address -s jobfailed; done
I need a job to be run once every time I apply my configuration to Kubernetes. I tried to use a job but it does not allow to update image version and requires deletion of previous job. I tried to use initContainer but it seems to be overkill since I don't need this to be run every time pod is started.
Essentially I have simple deployment with single pod and need to run a job before deployment is happening every time I update image in deployment and confused what is the best way to achieve it. Job seems to be perfect for this but issue is that I can not submit the same config second time with update image tag for job.
I'm afraid there is no other option. I recently had a similar situation and this is how I'm doing the job recreation:
kubectl get <job name> -o json | \
jq -r '.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration"' | \
kubectl replace --save-config --force -f -
It uses the "kubectl.kubernetes.io/last-applied-configuration" annotation, that holds the initial job config that was applied, without the auto-generated/cluster fields. Then the kubectl replace will completely replace the current one with a copy of itself, simulating the "rerun" that you are looking for.
You can then add to the snippet above some code replacing the image name/tag, before the kubectl replace.
I am building deploy pipeline. I Need a "kubectl" command that would tell me that rollout is completed to all the pods then I can deploy to next stage.
The Deployment documentation suggests kubectl rollout status, which among other things will return a non-zero exit code if the deployment isn't complete. kubectl get deployment will print out similar information (how many replicas are expected, available, and up-to-date), and you can add a -w option to watch it.
For this purpose you can also consider using one of the Kubernetes APIs. You can "get" or "watch" the deployment object, and get back something matching the structure of a Deployment object. Using that you can again monitor the replica count, or the embedded condition list, and decide if it's ready or not. If you're using the "watch" API you'll continue to get updates as the object status changes.
The one trick here is detecting failed deployments. Say you're deploying a pod that depends on a database; usual practice is to configure the pod with the hostname you expect the database to have, and just crash (and get restarted) if it's not there yet. You can briefly wind up in CrashLoopBackOff state when this happens. If your application or deployment is totally wrong, of course, you'll also wind up in CrashLoopBackOff state, and your deployment will stop progressing. There's not an easy way to tell these two cases apart; consider an absolute timeout.
I using Kubernetes 1.12. I have a service (e.g. pod) which may have multiple instances (e.g. replicas > 1)
My goal is to perform a maintenance task (e.g. create\upgrade database, generate certificate, etc) before any of service instances are up.
I was considering to use Init Container, but at least as I understand, Init Container will be executed anytime additional replica (pod) is created and worse - that might happen in parallel. In that case, multiple Init Containers might work in parallel and thus corrupt my database and everything else.
I need a clear solution to perform a bootstrap maintenance task only once per deployment. How you would suggest to do that?
I encountered the same problem running db migrations before each deployment. Here's a solution based on a Job resource:
kubectl apply -f migration-job.yml
kubectl wait --for=condition=complete --timeout=60s job/migration
kubectl delete job/migration
kubectl apply -f deployment.yml
migration-job.yml defines a Job configured with restartPolicy: Never and a reasonably low activeDeadlineSeconds. Using kubectl wait ensures that any errors or timeout in migration-job.yml causes the script to fail and thus prevent applying deployment.yml.
One of the ways you could use to retain startup sequence controll would be to use StatefulSet. With sequential startup, next pod will not start untill previous is done, removing parallel init risk.
Personally I would prefer this init to have its own locking mechanism and stick to regular Deploymants.
Remember that you need to take into account not only first startup on Deployment creation, but also cases for rolling releases, scaling, outages etc.
Quick question regarding Kubernetes job status.
Lets assume I submit my resource to 10 PODS and want to check if my JOB is completed successfully.
What is the best available options that we can use from KUBECTL commands.
I think of kubectl get jobs but the problem here is you have only two codes 0 & 1. 1 for completion 0 for failed or running, we cannot really depend on this
Other option is kubectl describe to check the PODS status like out of 10 PODS how many are commpleted/failed.
Any other effective way of monitoring the PODs? Please let me know
Anything that can talk to the Kubernetes API can query for Job object and look at the JobStatus field, which has info on which pods are running, completed, failed, or unavailable. kubectl is probably the easiest, as you mentioned, but you could write something more specialized using any client library if you wanted/needed to.
I was able to able to achieve it by running the following, which will show the status of the container:
kubectl describe job