Let's say I have a Pod with 2 containers: App and Database. I want to run a Pod that executes a command in App and then terminates.
I have set up my App container to run that command, and then it succesully runs and terminates which is great. But now my Database container is still running, so the Pod is not marked as complete.
How can I get the Pod to be marked as complete when the App container is completed?
You can make a call to the Kubernetes API server to accomplish this. Consider the following example:
---
apiVersion: v1
kind: Pod
metadata:
name: multi-container-completion
spec:
containers:
- name: long-running-process
image: fbgrecojr/office-hours:so-47848488
command: ["sleep", "1000"]
- name: short-running-process
image: fbgrecojr/office-hours:so-47848488
command: ["sleep", "1"]
lifecycle:
preStop:
exec:
command: ["/pre-stop.sh"]
pre-stop.sh
#!/bin/bash
curl \
-X DELETE \
-H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
--cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
https://kubernetes.default.svc.cluster.local/api/v1/namespaces/$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)/pods/$HOSTNAME
Dockerfile for fbgrecojr/office-hours:so-47848488
FROM centos:latest
COPY pre-stop.sh /
RUN chmod +x /pre-stop.sh
NOTE: I was not able to properly test this because preStop hooks do not seem to be working for my local Minikube setup. In case this issue is not localized to me, the corresponding issue can be tracked here.
Related
I'm looking for a way to quickly run/restart a Job/Pod from the command line and override the command to be executed in the created container.
For context, I have a Kubernetes Job that gets executed as a part of our deploy process. Sometimes that Job crashes and I need to run certain commands inside the container the Job creates to debug and fix the problem (subsequent Jobs then succeed).
The way I have done this so far is:
Copy the YAML of the Job, save into a file
Clean up the YAML (delete Kubernetes-managed fields)
Change the command: field to tail -f /dev/null (so that the container stays alive)
kubectl apply -f job.yaml && kubectl get all && kubectl exec -ti pod/foobar bash
Run commands inside the container
kubectl delete job/foobar when I am done
This is very tedious. I am looking for a way to do something like the following
kubectl restart job/foobar --command "tail -f /dev/null"
# or even better
kubectl run job/foobar --exec --interactive bash
I cannot use the run command to create a Pod:
kubectl run --image xxx -ti
because the Job I am trying to restart has certain volumeMounts and other configuration I need to reuse. So I would need something like kubectl run --from-config job/foobar.
Is there a way to achieve this or am I stuck with juggling the YAML definition file?
Edit: the Job YAML looks approx. like this:
apiVersion: batch/v1
kind: Job
metadata:
name: database-migrations
labels:
app: myapp
service: myapp-database-migrations
spec:
backoffLimit: 0
template:
metadata:
labels:
app: myapp
service: myapp-database-migrations
spec:
restartPolicy: Never
containers:
- name: migrations
image: registry.example.com/myapp:977b44c9
command:
- "bash"
- "-c"
- |
set -e -E
echo "Running database migrations..."
do-migration-stuff-here
echo "Migrations finished at $(date)"
imagePullPolicy: Always
volumeMounts:
- mountPath: /home/example/myapp/app/config/conf.yml
name: myapp-config-volume
subPath: conf.yml
- mountPath: /home/example/myapp/.env
name: myapp-config-volume
subPath: .env
volumes:
- name: myapp-config-volume
configMap:
name: myapp
imagePullSecrets:
- name: k8s-pull-project
The commands you suggested don't exist. Take a look at this reference where you can find all available commands.
Based on that documentation the task of the Job is to create one or more Pods and continue retrying execution them until the specified number of successfully terminated ones will be achieved. Then the Job tracks the successful completions. You cannot just update the Job because these fields are not updatable. To do what's you want you should delete current job and create one once again.
I recommend you to keep all your configurations in files. If you have a problem with configuring job commands, practice says that you should modify these settings in yaml and apply to the cluster - if your deployment crashes - by storing the configuration in files, you have a backup.
If you are interested how to improve this task, you can try those 2 examples describe below:
Firstly I've created several files:
example job (job.yaml):
apiVersion: batch/v1
kind: Job
metadata:
name: test1
spec:
template:
spec:
containers:
- name: test1
image: busybox
command: ["/bin/sh", "-c", "sleep 300"]
volumeMounts:
- name: foo
mountPath: "/script/foo"
volumes:
- name: foo
configMap:
name: my-conf
defaultMode: 0755
restartPolicy: OnFailure
patch-file.yaml:
spec:
template:
spec:
containers:
- name: test1
image: busybox
command: ["/bin/sh", "-c", "echo 'patching test' && sleep 500"]
and configmap.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
name: my-conf
data:
test: |
#!/bin/sh
echo "skrypt test"
If you want to automate this process you can use plugin
A plugin is a standalone executable file, whose name begins with kubectl-. To install a plugin, move its executable file to anywhere on your PATH.
There is no plugin installation or pre-loading required. Plugin executables receive the inherited environment from the kubectl binary. A plugin determines which command path it wishes to implement based on its name.
Here is the file that can replace your job
A plugin determines the command path that it will implement based on its filename.
kubectl-job:
#!/bin/bash
kubectl patch -f job.yaml -p "$(cat patch-job.yaml)" --dry-run=client -o yaml | kubectl replace --force -f - && kubectl wait --for=condition=ready pod -l job-name=test1 && kubectl exec -it $(kubectl get pod -l job-name=test1 --no-headers -o custom-columns=":metadata.name") -- /bin/sh
This command uses an additional file (patch-job.yaml, see this link) - within we can put our changes for job.
Then you should change the permissions of this file and move it:
sudo chmod +x .kubectl-job
sudo mv ./kubectl-job /usr/local/bin
It's all done. Right now you can use it.
$ kubectl job
job.batch "test1" deleted
job.batch/test1 replaced
pod/test1-bdxtm condition met
pod/test1-nh2pv condition met
/ #
As you can see Job has been replaced (deleted and created).
You can also use single-line command, here is the example:
kubectl get job test1 -o json | jq "del(.spec.selector)" | jq "del(.spec.template.metadata.labels)" | kubectl patch -f - --patch '{"spec": {"template": {"spec": {"containers": [{"name": "test1", "image": "busybox", "command": ["/bin/sh", "-c", "sleep 200"]}]}}}}' --dry-run=client -o yaml | kubectl replace --force -f -
With this command you can change your job entering parameters "by hand". Here is the output:
job.batch "test1" deleted
job.batch/test1 replaced
As you can see this solution works as well.
I was trying to add a poststart hook for my pod using curl, say sending a message to my slack channel
in shell, the command looks like this
curl -d "text=Hi I am a bot that can post messages to any public channel." -d "channel=C1234567" -H "Authorization: Bearer xoxb-xxxxxxxxxxxxxxxx" -X POST https://slack.com/api/chat.postMessage
and in my pod definition, i tried sth like this
apiVersion: v1
kind: Pod
metadata:
name: dapi-test-pod
spec:
containers:
- name: test-container
image: curlimages/curl
env:
- name: TOKEN
valueFrom:
configMapKeyRef:
name: my-config
key: token
command: ["sleep"]
args: ["3000"]
lifecycle:
postStart:
exec:
command:
- "sh"
- "-c"
- |
curl -d "text=Hi going to start." -d "channel=C1234567" -H "Authorization: Bearer $(TOKEN)" -X POST https://slack.com/api/chat.postMessage
Unlike the container->command, it has args parameter which i could pass multi line command with quote, but in lifecycle->poststart->exec->command it doesn't support args parameter
I also tried sth like but no luck
command: ["curl","-d","text=Hi going to start.",....]
but i never got my slack message
My question is, how can i pass long curl command with quote in lifecycle->poststart->exec->command?
it finally solved by replacing () with {}
to use a env variable in command, it should be ${TOKEN}
I am deploying my application in kubernetes using helm chart with 2 sub-charts app and test.
I have the pod of app chart properly running.
But test pod will be running only if it can properly authenticate to app container.
That means, i have to generate an auth_token using a curl request to app service and then add that token as Environment variable AUTH_TOKEN for test container.
I tried different ways to achieve this:
Added an init-container generate-token for test pod, that will generate the token and will save it in a shared volume. And test container will have access to that volume. But the problem here is, the test container doesn't have a code to set env for the container by reading from the shared volume.
Added a sidecar-container sidecar-generate-token instead of an init-container for the same setup as mentioned above. Here also problem is, the test container doesn't have a code to set env for the container by reading from the shared volume. And also, the test pod got into a crashloopbackoff state. If you check the content of volume by getting into the container, there are multiple tokens in the volume file which are generated on each pod restart of crashloopbackoff.
Third plan was that an init-container generate-token should create a kubernetes secret in the cluster, after generating the auth_token. Then the main container test can set Environment variable from that secret. For that, the init container generate-token should have a kubectl setup in it first.
If i am proceeding with the third plan, How can i setup and use kubectl from init-container to generate secret in the cluster?
Is there any other alternative plan to achieve this goal?
EDIT:
This is the yaml part for the first option:
initContainers:
- name: generate-service-token
image: app.mycr.io/alpine-network-troubleshooting:dev-latest
command:
- /bin/sh
- -c
- |
BEARER_TOKEN=$(curl -k -X POST -H "Content-Type:application/json" --data '{"user":"dynizer","password":"xxxx"}' "https://app:50051/api/v2/login" | jq -r '.jwt')
SERVICE_TOKEN=$(curl -k -X GET -H 'Accept: application/json' -H "Authorization: Bearer ${BEARER_TOKEN}" "https://app:50051/api/v2/servicetoken/issue" | jq -r '.token')
echo $SERVICE_TOKEN
mkdir -p /vol
touch /vol/token.txt
echo $SERVICE_TOKEN >> /vol/token.txt
volumeMounts:
- mountPath: /vol
name: token-vol
containers:
- name: nginx-container
image: nginx
volumeMounts:
- name: token-vol
mountPath: /vol
volumes:
- name: token-vol
emptyDir: {}
Trying to answer your question:
But still the same problem of container not having the code to set env by reading from the shared volume, will be there.
Let's try to read this env from other container. Here Is what I have come up with.
First you need to know what command your container is running. In case of nginx that is /docker-entrypoint.sh nginx -g "daemon off;" (source code)
Then you use command field where you read the token value from file and use env to set it and run the actual applciation.
Example:
initContainers:
- name: generate-service-token
image: app.mycr.io/alpine-network-troubleshooting:dev-latest
command:
- /bin/sh
- -c
- |
BEARER_TOKEN=$(curl -k -X POST -H "Content-Type:application/json" --data '{"user":"dynizer","password":"xxxx"}' "https://app:50051/api/v2/login" | jq -r '.jwt')
SERVICE_TOKEN=$(curl -k -X GET -H 'Accept: application/json' -H "Authorization: Bearer ${BEARER_TOKEN}" "https://app:50051/api/v2/servicetoken/issue" | jq -r '.token')
echo $SERVICE_TOKEN
mkdir -p /vol
touch /vol/token.txt
echo $SERVICE_TOKEN >> /vol/token.txt
volumeMounts:
- mountPath: /vol
name: token-vol
containers:
- name: nginx-container
image: nginx
command:
- sh
- -c
- exec env SERVICE_TOKEN=$(cat /vol/token.txt) /docker-entrypoint.sh nginx -g "daemon off;"
volumeMounts:
- name: token-vol
mountPath: /vol
volumes:
- name: token-vol
emptyDir: {}
More general example:
command:
- sh
- -c
- exec env SERVICE_TOKEN=$(cat /vol/token.txt) <<any command>>
I am not sure if this is the best example, but I hope that at least it gives you an idea how you can approach this problem.
Whenever I need to register my EKS services and required routes with kong, I have to manually execute CURL method( post/get ) commands for same, Services and routes get register successfully, but my requirement is to build or automate above multiple configurations with KONG, some way like producing a YAML file for all service registrations and routes for KONG and then executing at once.
I explored all the sources, even KONG official documentation, but couldn't find any way which ease my requirement
###################### Adding Svc ##########################################
curl -k -i -X POST \
--url https://localhost:7001/services/ \
--data 'name=hello-world1' \
--data 'host=service-helloworld' \
--data 'port=80'
###################### Adding Route ##########################################
curl -k -i -X POST --url https://localhost:7001/services/hello-world/routes --data 'paths=/hello-world' --data 'methods[]=GET'
Some way to automate above CURL commands
If I understand you correctly those are some of the ways you are looking for:
Container Lifecycle Hooks
In your case you would want to use PostStart
This hook executes immediately after a container is created. However, there is no guarantee that the hook will execute before the container ENTRYPOINT. No parameters are passed to the handler.
Hook handler implementations
Containers can access a hook by implementing and registering a handler for that hook. There are two types of hook handlers that can be implemented for Containers:
Exec - Executes a specific command, such as pre-stop.sh, inside the cgroups and namespaces of the Container. Resources consumed by the command are counted against the Container.
HTTP - Executes an HTTP request against a specific endpoint on the Container.
Your pod might look like the following example:
apiVersion: v1
kind: Pod
metadata:
name: lifecycle-demo
spec:
containers:
- name: lifecycle-demo-container
image: nginx
lifecycle:
postStart:
exec:
command:
- "sh"
- "-c"
- >
curl -k -i -X POST --url https://localhost:7001/services/ --data 'name=hello-world1' --data 'host=service-helloworld' --data 'port=80';
curl -k -i -X POST --url https://localhost:7001/services/hello-world/routes --data 'paths=/hello-world' --data 'methods[]=GET'
Init Containers
A Pod can have multiple containers running apps within it, but it can also have one or more init containers, which are run before the app containers are started.
Init containers are exactly like regular containers, except:
Init containers always run to completion.
Each init container must complete successfully before the next one starts.
And here is an example from docs:
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: busybox:1.28
command: ['sh', '-c', 'echo The app is running! && sleep 3600']
initContainers:
- name: init-myservice
image: busybox:1.28
command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice; sleep 2; done;']
- name: init-mydb
image: busybox:1.28
command: ['sh', '-c', 'until nslookup mydb; do echo waiting for mydb; sleep 2; done;']
What is the best practice for backing up a Postgres database running on Google Cloud Container Engine?
My thought is working towards storing the backups in Google Cloud Storage, but I am unsure of how to connect the Disk/Pod to a Storage Bucket.
I am running Postgres in a Kubernetes cluster using the following configuration:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: postgres-deployment
spec:
replicas: 1
template:
metadata:
labels:
app: postgres
spec:
containers:
- image: postgres:9.6.2-alpine
imagePullPolicy: IfNotPresent
env:
- name: PGDATA
value: /var/lib/postgresql/data
- name: POSTGRES_DB
value: my-database-name
- name: POSTGRES_PASSWORD
value: my-password
- name: POSTGRES_USER
value: my-database-user
name: postgres-container
ports:
- containerPort: 5432
volumeMounts:
- mountPath: /var/lib/postgresql
name: my-postgres-volume
volumes:
- gcePersistentDisk:
fsType: ext4
pdName: my-postgres-disk
name: my-postgres-volume
I have attempted to create a Job to run a backup:
apiVersion: batch/v1
kind: Job
metadata:
name: postgres-dump-job
spec:
template:
metadata:
labels:
app: postgres-dump
spec:
containers:
- command:
- pg_dump
- my-database-name
# `env` value matches `env` from previous configuration.
image: postgres:9.6.2-alpine
imagePullPolicy: IfNotPresent
name: my-postgres-dump-container
volumeMounts:
- mountPath: /var/lib/postgresql
name: my-postgres-volume
readOnly: true
restartPolicy: Never
volumes:
- gcePersistentDisk:
fsType: ext4
pdName: my-postgres-disk
name: my-postgres-volume
(As far as I understand) this should run the pg_dump command and output the backup data to stdout (which should appear in the kubectl logs).
As an aside, when I inspect the Pods (with kubectl get pods), it shows the Pod never gets out of the "Pending" state, which I gather is due to there not being enough resources to start the Job.
Is it correct to run this process as a Job?
How do I connect the Job to Google Cloud Storage?
Or should I be doing something completely different?
I'm guessing it would be unwise to run pg_dump in the database Container (with kubectl exec) due to a performance hit, but maybe this is ok in a dev/staging server?
As #Marco Lamina said you can run pg_dump on postgres pod like
DUMP
// pod-name name of the postgres pod
// postgres-user database user that is able to access the database
// database-name name of the database
kubectl exec [pod-name] -- bash -c "pg_dump -U [postgres-user] [database-name]" > database.sql
RESTORE
// pod-name name of the postgres pod
// postgres-user database user that is able to access the database
// database-name name of the database
cat database.sql | kubectl exec -i [pod-name] -- psql -U [postgres-user] -d [database-name]
You can have a job pod that does run this command and exports this to a file storage system such as AWS s3.
I think running pg_dump as a job is a good idea, but connecting directly to your DB's persistent disk is not. Try having pg_dump connect to your DB over the network! You could then have a second disk onto which your pg_dump command dumps the backups. To be on the safe side, you can create regular snapshots of this second disk.
The reason for the Jobs POD to stay in Pending state is that it forever tries to attach/mount the GCE persistent disk and fails to do so because it is already attached/mounted to another POD.
Attaching a persistent disk to multiple PODs is only supported if all of them attach/mount the volume in ReadOnly mode. This is of course no viable solution for you.
I never worked with GCE, but it should be possible to easily create a snapshot from the PD from within GCE. This would not give a very clean backup, more like something in the state of "crashed in the middle, but recoverable", but this is probably acceptable for you.
Running pg_dump inside the database POD is a viable solution, with a few drawbacks as you already noticed, especially performance. You'd also have to move out the resulting backup from the POD afterwards, e.g. by using kubectl cp and another exec to cleanup the backup in the POD.
You can use Minio Client
First of all use simple dockerfile to make docker image contains postgres along with minio client (let name this image postgres_backup):
FROM postgres
RUN apt-get update && apt-get install -y wget
RUN wget https://dl.min.io/client/mc/release/linux-amd64/mc
RUN chmod +x mc
RUN ./mc alias set gcs https://storage.googleapis.com BKIKJAA5BMMU2RHO6IBB V8f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12
Now you can use postgres_backup image in your CronJob (I assumed you made backups bucket in your Google storage):
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: backup-job
spec:
# Backup the database every day at 2AM
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: postgres-backup
image: postgres_backup
env:
- name: POSTGRES_HOST_AUTH_METHOD
value: trust
command: ["/bin/sh"]
args: ["-c", 'pg_dump -Fc -U [Your Postgres Username] -W [Your Postgres Password] -h [Your Postgres Host] [Your Postgres Database] | ./mc pipe gcs/backups/$(date -Iseconds).dump']
restartPolicy: Never
A lot of tutorials use kubectl cp or transfer the file inside the pod, but you can also pipe the pg_dump container output directly to another process.
kubectl run --env=PGPASSWORD=$PASSWORD --image=bitnami/postgresql postgresql -it --rm -- \
bash -c "pg_dump -U $USER -h $HOST -d $DATABASE" |\
gzip > backup.sql.gz
The easiest way to dump without storing any additional copies on your pod:
kubectl -n [namespace] exec -it [pod name] -- bash -c "export PGPASSWORD='[db password]'; pg_dump -U [db user] [db name]" > [database].sql