I have a k8s cronjob run my docker image transaction-service.
It starts and gets its job done successfully. When it's over, I expect the pod to terminate but... istio-proxy still lingers there:
And that results in:
Nothing too crazy, but I'd like to fix it.
I know I should call curl -X POST http://localhost:15000/quitquitquit
But I don't know where and how. I need to call that quitquitquit URL only when transaction-service is in a completed state. I read about preStop lifecycle hook, but I think I need more of a postStop one. Any suggestions?
You have a few options here:
On your job/cronjob spec, add the following lines and your job immediately after:
command: ["/bin/bash", "-c"]
args:
- |
trap "curl --max-time 2 -s -f -XPOST http://127.0.0.1:15020/quitquitquit" EXIT
while ! curl -s -f http://127.0.0.1:15020/healthz/ready; do sleep 1; done
echo "Ready!"
< your job >
Disable Istio injection at the Pod level in your Job/Cronjob definition:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
...
spec:
...
jobTemplate:
spec:
template:
metadata:
annotations:
# disable istio on the pod due to this issue:
# https://github.com/istio/istio/issues/11659
sidecar.istio.io/inject: "false"
Note: The annotation should be on the Pod's template, not on the Job's template.
You can use TTL mechanism for finished Jobs mentioned in kubernetes doc which help removing the whole pod.
In my Dockerfile I put
ADD ./entrypoint.sh /entrypoint.sh
RUN ["chmod", "+x", "/entrypoint.sh"]
RUN apk --no-cache add curl
ENTRYPOINT ["/entrypoint.sh"]
My entrypoint.sh looks like this:
#!/bin/sh
/app/myapp && curl -X POST http://localhost:15000/quitquitquit
It works.
Related
Is this a valid imperative command for creating job?
kubectl create job my-job --image=busybox
I see this in https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands. But the command is not working. I am getting error as bellow:
Error: unknown flag: --image
What is the correct imperative command for creating job?
Try this one
kubectl create cronjob my-job --schedule="0,15,30,45 * * * *" --image=busy-box
What you have should work, though it not recommended as an approach anymore. I would check what version of kubectl you have, and possibly upgrade it if you aren't using the latest.
That said, the more common approach these days is to write a YAML file containing the Job definition and then run kubectl apply -f myjob.yaml or similar. This file-driven approach allowed for more natural version control, editing, review, etc.
Using correct value for --restart field on "kubectl run" will result run command to create an deployment or job or cronjob
--restart='Always': The restart policy for this Pod. Legal values [Always, OnFailure, Never]. If set to 'Always'
a deployment is created, if set to 'OnFailure' a job is created, if set to 'Never', a regular pod is created. For the
latter two --replicas must be 1. Default 'Always', for CronJobs `Never`.
Use "kubectl run" for creating basic kubernetes job using imperatively command as below
master $ kubectl run nginx --image=nginx --restart=OnFailure --dry-run -o yaml > output.yaml
Above should result an "output.yaml" as below example, you can edit this yaml for advance configurations as needed and create job by "kubectl create -f output.yaml or if you just need basic job then remove --dry-run option from above command and you will get basic job created.
apiVersion: batch/v1
kind: Job
metadata:
creationTimestamp: null
labels:
run: nginx
name: nginx
spec:
template:
metadata:
creationTimestamp: null
labels:
run: nginx
spec:
containers:
- image: nginx
name: nginx
resources: {}
restartPolicy: OnFailure
status: {}
On my test cluster, starting a pod seems to consistently take ~12 seconds, give or take one. I would like to know if that is reasonable, or if I am doing something wrong, either in configuring the pod, or in measuring the time, or configuring the cluster.
According to https://github.com/kubernetes/kubernetes/issues/3952 and https://medium.com/google-cloud/profiling-gke-startup-time-9052d81e0052, I believe what I am getting is excessively slow.
The way I measure startup is by running the following script and counting how many times it prints "Pending", and that is my startup time in seconds. Since due to the sleep command, I get almost exactly one "Pending" per second.
id=mypod1
tee job.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
name: clusterrunner-build-${id}
spec:
containers:
- name: clusterrunner-slave
image: jdanekrh/clusterrunner-slave
command: ["bash", "-c", "echo bof; sleep 5; echo lek"]
restartPolicy: Never
EOF
kubectl create -f job.yaml
while kubectl get pod/clusterrunner-build-${id} -o jsonpath='{.status.phase}' | grep Pending; do
sleep 1
done
kubectl logs -f po/clusterrunner-build-${id}
kubectl delete -f job.yaml
I'm doing a prototype where one service depends on an availability of other. Scenario:
Service A is assumed to be already available in a local network. It was either deployed by K8S or manually (or even a managed one provided by AWS etc.).
Service B depends on environment variable SERVICE_A_IP and won't start without it. It's treated as a black box and can't be modified.
I want to pass Service A IP to Service B through K8S YAML configuration file. Perfect syntax for this occasion:
...
env:
- name: SERVICE_A_IP
valueFrom:
k8sDeployment:
name: service_a
key: deploymentIP
...
During the prototyping stage Service A is an another K8S deployment but it might not be so in a production environment. Thus I need to decouple from SERVICE_A_SERVICE_IP that will be available to Service B (given it's deployed after Service A). I'm not into DNS discovery as well as it would require container modification which is far from a perfect solution.
If I would do it manually with kubectl (or with a shell script) it would be like the following:
$ kubectl run service_a --image=service_a:latest --port=8080
$ kubectl expose deployment service_a
$ SERVICE_A_IP="$(kubectl describe service service_a | \
grep IP: | \
cut -f2 -d ':' | \
xargs)"
$ kubectl run service_b --image=service_b:latest --port=8080 \
--env="SERVICE_A_IP=${SERVICE_A_IP}"
It works. Though I want to do the same using YAML configuration without injecting SERVICE_A_IP into configuration file with shell (basically modifying the file).
Is there any way to do so? Please take the above setting as set in stone.
UPDATE
Not the best way though still:
$ kubectl create -f service_a.yml
deployment "service_a" created
service "service_a" created
$ SERVICE_A_IP="$(kubectl describe service service_a | \
grep IP: | \
cut -f2 -d ':' | \
xargs)"
$ kubectl create configmap service_a_meta \
--from-literal="SERVICE_A_IP=${SERVICE_A_IP}"
And then in service_b.yml:
...
env:
- name: SERVICE_A_IP
valueFrom:
configMapKeyRef:
name: service_a_meta
key: SERVICE_A_IP
...
That will work but still involves some shell and generally feels way too hax.
You can use attach handlers to lifecycle events for update your environment variables on start.
Here is an example:
apiVersion: v1
kind: Pod
metadata:
name: appB
spec:
containers:
- name: appB
image: nginx
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "export SERVICE_B_IP=$(host <SERVICE_B>.<SERVICE_B_NAMESPACE>.svc.cluster.local)"]
Kubernetes will run preStart script each time when pod with your appB container is starting right in appB container before execution of the main application.
But, because of that description:
PostStart
This hook executes immediately after a container is created. However, there is no guarantee that the hook will execute before the container ENTRYPOINT. No parameters are passed to the handler.
You need to add some sleep for your main app before the real start just to be sure that hook will be finished before application will be started.
I am trying to develop a Helm chart for an application to ease release management and deployment of the application to kubernetes. In order to do so, i have written a pre-install hook in the Helm chart.
apiVersion: batch/v1
kind: Job
metadata:
name: px-etcd-preinstall-hook
labels:
heritage: {{.Release.Service | quote }}
release: {{.Release.Name | quote }}
chart: "{{.Chart.Name}}-{{.Chart.Version}}"
annotations:
"helm.sh/hook": pre-install
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": hook-succeeded, hook-failed
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: pre-install-job
imagePullPolicy: Always
image: "hrishi/px-etcd-preinstall-hook:v1"
command: ['/bin/sh']
args: ['/usr/bin/etcdStatus.sh',"{{ .Values.etcdEndPoint }}"]
This docker container just checks if an ETCD endpoint is accessible or not. Idea is for it to wait for a few seconds and a few tries and then exit.
Here is the initial shell script which runs as part of this container.
set -x
echo "Initializing..."
svcname=$1
echo $svcname
etcdURL=$(echo "$svcname" | awk -F: '{ st = index($0,":");print substr($0,st+1)}')
echo $etcdURL
response=$(curl --write-out %{http_code} --silent --output /dev/null "$etcdURL/version")
echo $response
if [[ "$response" != 200 ]]
then
echo "Provided etcd url is not reachable. Exiting.."
exit 1
fi
All is well and fine if the ETCD url is accessible, but if the etcd url is inaccessible then I get an error stating Error: Job failed: BackoffLimitExceeded”
I want to check if there is a way of setting a user friendly error message stating that the url isnt accessible or something like that.
Seems there isnt a way to do it right now, not that i know of. I tried this to just be a Pod instead of a Job and that doesnt work either.
Looked up at the docs for Helm but couldnt seem to find any information regarding this.
I don't think is possible. But I'd take a different approach.
If your application requires ETCD, why don't you check if ETCD is accesible as one of your Pod probes, like liveness or readiness? That way, if there is no connectivity between your application and ETCD, your application won't start and you'll know that the probe failed when describing your Pod, in a more kubernetes way.
Furthermore, you can even make helm install to wait until all the Pod's are Ready, meaning that the command helm install would fail if your application didn't connect to ETCD.
We are using elasticsearch/kibana instead of gcp for logging (based on what is described here).
To have fluentd-elsticsearch pod's launched we've set LOGGING_DESTINATION=elasticsearch and ENABLE_NODE_LOGGING="true" in the "Compute Instance Template" -> "Custom metadata" -> "kube-env".
While this works fine when done manually it gets overwritten with every gcloud container clusters upgrade as a new Instance Template with defaults (LOGGING_DESTINATION=gcp ...) is created.
My question is: How do I persist this kind of configuration for GKE/GCE?
I thought about adding a k8s-user-startup-script but that's also defined in the Instance Template and therefore is overwritten by gcloud container clusters upgrade.
I've also tried to add a k8s-user-startup-script to the project metadata but that is not taken into account.
//EDIT
Current workaround (without recreating Instance Template and Instances) for manually switching back to elasticsearch is:
for node in $(kubectl get nodes -o name | cut -f2 -d/); do
gcloud compute ssh $node \
--command="sudo cp -a /srv/salt/fluentd-es/fluentd-es.yaml /etc/kubernetes/manifests/; sudo rm /etc/kubernetes/manifests/fluentd-gcp.yaml";
done
kubelet will pick that up, kill fluentd-gcp and start fluentd-es.
//EDIT #2
Now running a "startup-script" DaemonSet for this:
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
name: startup-script
namespace: kube-system
labels:
app: startup-script
spec:
template:
metadata:
labels:
app: startup-script
spec:
hostPID: true
containers:
- name: startup-script
image: gcr.io/google-containers/startup-script:v1
securityContext:
privileged: true
env:
- name: STARTUP_SCRIPT
value: |
#! /bin/bash
set -o errexit
set -o pipefail
set -o nounset
# Replace Google-Cloud-Logging with EFK
if [[ ! -f /etc/kubernetes/manifests/fluentd-es.yaml ]]; then
if [[ -f /home/kubernetes/kube-manifests/kubernetes/fluentd-es.yaml ]]; then
# GCI images
cp -a /home/kubernetes/kube-manifests/kubernetes/fluentd-es.yaml /etc/kubernetes/manifests/
elif [[ -f /srv/salt/fluentd-es/fluentd-es.yaml ]]; then
# Debian based GKE images
cp -a /srv/salt/fluentd-es/fluentd-es.yaml /etc/kubernetes/manifests/
fi
test -f /etc/kubernetes/manifests/fluentd-es.yaml && rm /etc/kubernetes/manifests/fluentd-gcp.yaml
fi
There isn't a fully supported way to reconfigure the kube-env in GKE. As you've found, you can hack the instance template, but this isn't guaranteed to work across upgrades.
An alternative is to create your cluster without gcp logging enabled and then create a DaemonSet that places a fluentd-elasticsearch pod on each of your nodes. Using this technique you don't need to write a (brittle) startup script or rely on the fact that the built-in startup script happens to work when setting LOGGING_DESTINATION=elasticsearch (which may break across upgrades even if it wasn't getting overwritten).