Is it possible to use a bash script to do the liveness test in pod? - kubernetes

I'm currently setting up a kubernetes cluster with 3 nodes on 3 differents vm and each node is composed of 1 pod witch run the following docker image: ethereum/client-go:stable
The problem is that I want to do a health check test using a bash script (because I have to test a lot of things) but I don't understand how I can export this file to each container that are deployed with my yaml deployment file.
I've tried to add wget command in the yaml file to download my health check script from my github repo but it wasn't very clean from my point of view, maybe there is an other way ?
My current deployment file:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: goerli
name: goerli-deploy
spec:
replicas: 3
selector:
matchLabels:
app: goerli
template:
metadata:
labels:
app: goerli
spec:
containers:
- image: ethereum/client-go:stable
name: goerli-geth
args: ["--goerli", "--datadir", "/test2"]
env:
- name: LASTBLOCK
value: "0"
- name: FAILCOUNTER
value: "0"
ports:
- containerPort: 30303
name: geth
livenessProbe:
exec:
command:
- /bin/sh
- /test/health.sh
initialDelaySeconds: 60
periodSeconds: 100
volumeMounts:
- name: test
mountPath: /test
restartPolicy: Always
volumes:
- name: test
hostPath:
path: /test
I expect to put health check script in /test/health.sh
Any ideas ?

This could be a perfect usecase for the init container, As there could be different images for the init container and the Application container thus they have different file system inside the pods, therefore we need to use Emptydir in order to share the state.
for further detail follow the link init-containers

Thanks to Suresh Vishnoi:
A way to resolve my problem is to use init container this way:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: goerli
name: goerli-deploy
spec:
replicas: 3
selector:
matchLabels:
app: goerli
template:
metadata:
labels:
app: goerli
spec:
containers:
- image: ethereum/client-go:stable
name: goerli-geth
args: ["--goerli", "--datadir", "/test2"]
env:
- name: LASTBLOCK
value: "0"
- name: FAILCOUNTER
value: "0"
ports:
- containerPort: 30303
name: geth
livenessProbe:
exec:
command:
- /bin/sh
- /test/health.sh
initialDelaySeconds: 60
periodSeconds: 100
volumeMounts:
- name: test
mountPath: /test
initContainers:
- name: healthcheck
image: ethereum/client-go:stable
command: ["wget", "-O", "/test2/health.sh", "https://My-script-bash"]
volumeMounts:
- name: test
mountPath: "/test"
restartPolicy: Always
volumes:
- name: test
emptyDir: {}
The downloaded file will be visible in /test/health.sh

If you're using helm look at chart tests: https://github.com/helm/helm/blob/master/docs/chart_tests.md. This covers readinessProbe tho, not liveness.
For advanced liveness probe, I'd run some kind of healthcheck sidecar which does all the advanced tests continiosly via localhost, and exposes a single /healthcheck endpoint. Then use the endpoint in a liveness probe.

Related

K3s no longer recognizes Liveness and Readiness Probes in Deployment

I am redeploying a K3s deployment from a few months ago. Then, it worked fine, with no problems. However, when I try to deploy it now: I get the following error:
Error from server (BadRequest): error when creating "deployment.yaml": Deployment in version "v1" cannot be handled as a Deployment: strict decoding error: unknown field "spec.template.spec.livenessProbe", unknown field "spec.template.spec.readinessProbe"
My .yaml for the deployment is unchanged, and looks like this:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: vei-deployment
spec:
replicas: 1
selector:
matchLabels:
app: server-pod
template:
metadata:
labels:
app: server-pod
spec:
containers:
- name: server-pod
image: myname/mydeployment:latest
env:
- name: AWS_ACCESS_KEY_ID
value: $AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
value: $AWS_SECRET_ACCESS_KEY
ports:
- name: grpc
containerPort: 50051
livenessProbe:
exec:
command:
- grpcurl
- -plaintext
- localhost:50051
- ping.Pinger/Ping
readinessProbe:
exec:
command:
- grpc_health_probe
- -addr=:50051
I have linted the .yaml file, and their doesn't seem to be any problem on that end. Has the syntax for Liveness and readiness changed drastically over the past few months?
The probes were missing an indent, and needed to be indented four spaces so that they were defined as part of the container. Instead of in the question, the .yaml needs to be like so:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: vei-deployment
spec:
replicas: 1
selector:
matchLabels:
app: server-pod
template:
metadata:
labels:
app: server-pod
spec:
containers:
- name: server-pod
image: myname/mydeployment:latest
env:
- name: AWS_ACCESS_KEY_ID
value: $AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
value: $AWS_SECRET_ACCESS_KEY
ports:
- name: grpc
containerPort: 50051
livenessProbe:
exec:
command:
- grpcurl
- -plaintext
- localhost:50051
- ping.Pinger/Ping
readinessProbe:
exec:
command:
- grpc_health_probe
- -addr=:50051

How to use git-sync image as a sidecar in kubernetes that git pulls periodically

I am trying to use git-sync image as a side car in kubernetes that runs git-pull periodically and mounts cloned data to shared volume.
Everything is working fine when I configure it for sync one time. I want to run it periodically like every 10 mins. Somehow when I configure it to run periodically pod initializing is failing.
I read documentation but couldn't find proper answer. Would be nice if you help me to figure out what I am missing in my configuration.
Here is my configuration that failing.
Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx-helloworld
image: nginx
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: www-data
initContainers:
- name: git-sync
image: k8s.gcr.io/git-sync:v3.1.3
volumeMounts:
- name: www-data
mountPath: /data
env:
- name: GIT_SYNC_REPO
value: "https://github.com/musaalp/design-patterns.git" ##repo-path-you-want-to-clone
- name: GIT_SYNC_BRANCH
value: "master" ##repo-branch
- name: GIT_SYNC_ROOT
value: /data
- name: GIT_SYNC_DEST
value: "hello" ##path-where-you-want-to-clone
- name: GIT_SYNC_PERIOD
value: "10"
- name: GIT_SYNC_ONE_TIME
value: "false"
securityContext:
runAsUser: 0
volumes:
- name: www-data
emptyDir: {}
Pod
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: nginx-helloworld
name: nginx-helloworld
spec:
containers:
- image: nginx
name: nginx-helloworld
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Never
status: {}
you are using the git-sync as an initContainers, which run only during init (once in lifecycle)
A Pod can have multiple containers running apps within it, but it can also have one or more init containers, which are run before the app containers are started.
Init containers are exactly like regular containers, except:
Init containers always run to completion.
Each init container must complete successfully before the next one starts.
init-containers
So use this as a regular container
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: git-sync
image: k8s.gcr.io/git-sync:v3.1.3
volumeMounts:
- name: www-data
mountPath: /data
env:
- name: GIT_SYNC_REPO
value: "https://github.com/musaalp/design-patterns.git" ##repo-path-you-want-to-clone
- name: GIT_SYNC_BRANCH
value: "master" ##repo-branch
- name: GIT_SYNC_ROOT
value: /data
- name: GIT_SYNC_DEST
value: "hello" ##path-where-you-want-to-clone
- name: GIT_SYNC_PERIOD
value: "20"
- name: GIT_SYNC_ONE_TIME
value: "false"
securityContext:
runAsUser: 0
- name: nginx-helloworld
image: nginx
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: www-data
volumes:
- name: www-data
emptyDir: {}

GCP Firestore: Server request fails with Missing or insufficient permissions from GKE

I am trying to connect to Firestore from code running on GKE Container. Simple REST GET api is working fine, but when I access the Firestore from read/write, I am getting Missing or insufficient permissions.
An unhandled exception was thrown by the application.
Info
2021-06-06 21:21:20.283 EDT
Grpc.Core.RpcException: Status(StatusCode="PermissionDenied", Detail="Missing or insufficient permissions.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"#1623028880.278990566","description":"Error received from peer ipv4:172.217.193.95:443","file":"/var/local/git/grpc/src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Missing or insufficient permissions.","grpc_status":7}")
at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
Update I am trying to provide secret to pod with service account credentails.
Here is the k8 file which deploys a pod to cluster with no issues when no secrets are provided and I can do Get Operations which don't hit Firestore, and they work fine.
kind: Deployment
apiVersion: apps/v1
metadata:
name: foo-worldmanagement-production
spec:
replicas: 1
selector:
matchLabels:
app: foo
role: worldmanagement
env: production
template:
metadata:
name: worldmanagement
labels:
app: foo
role: worldmanagement
env: production
spec:
containers:
- name: worldmanagement
image: gcr.io/foodev/foo/master/worldmanagement.21
resources:
limits:
memory: "500Mi"
cpu: "300m"
imagePullworld: Always
readinessProbe:
httpGet:
path: /api/worldManagement/policies
port: 80
ports:
- name: worldmgmt
containerPort: 80
Now, if I try to mount secret, the pod never gets created fully, and it eventually fails
kind: Deployment
apiVersion: apps/v1
metadata:
name: foo-worldmanagement-production
spec:
replicas: 1
selector:
matchLabels:
app: foo
role: worldmanagement
env: production
template:
metadata:
name: worldmanagement
labels:
app: foo
role: worldmanagement
env: production
spec:
volumes:
- name: google-cloud-key
secret:
secretName: firestore-key
containers:
- name: worldmanagement
image: gcr.io/foodev/foo/master/worldmanagement.21
volumeMounts:
- name: google-cloud-key
mountPath: /var/
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/key.json
resources:
limits:
memory: "500Mi"
cpu: "300m"
imagePullworld: Always
readinessProbe:
httpGet:
path: /api/worldManagement/earth
port: 80
ports:
- name: worldmgmt
containerPort: 80
I tried to deploy the sample application and it works fine.
If I keep only the following the yaml file, the container gets deployed properly
- name: google-cloud-key
secret:
secretName: firestore-key
But once I add the following to yaml, it fails
volumeMounts:
- name: google-cloud-key
mountPath: /var/
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/key.json
And I can see in GCP events that the container is not able to find the google-cloud-key. Any idea how to troubleshoot this issue, i.e why I am not able to mount the secrets, I can bash into the pod if needed.
I am using multi stage docker file made of
From mcr.microsoft.com/dotnet/sdk:5.0 AS build
FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS runtime
Thanks
Looks like they key itself might not be correctly visible to the pod. I would start by getting into the pod with kubectl exec --stdin --tty <podname> -- /bin/bash and ensuring that the /var/key.json (per your config) is accessible and has the correct credentials.
The following would be a good way to mount the secret:
volumeMounts:
- name: google-cloud-key
mountPath: /var/run/secret/cloud.google.com
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/run/secret/cloud.google.com/key.json
The above assumes your secret was created with a command like:
kubectl --namespace <namespace> create secret generic firestore-key --from-file key.json
Also it is important to check your Workload Identity setup. The Workload Identity | Kubernetes Engine Documentation has a good section on this.

How to run a script as command in Kubernetes yaml file

I have this script. A Pod will have two containers, one for the main application and the other for logging. I want the logging container to sleep to help me debug an issue.
apiVersion: apps/v1
kind: Deployment
metadata:
name: codingjediweb
spec:
replicas: 2
selector:
matchLabels:
app: codingjediweb
template:
metadata:
labels:
app: codingjediweb
spec:
volumes:
- name: shared-logs
emptyDir: {}
containers:
- name: codingjediweb
image: docker.io/manuchadha25/codingjediweb:03072020v2
volumeMounts:
- name: shared-logs
mountPath: /deploy/codingjediweb-1.0/logs/
env:
- name: db.cassandraUri
value: cassandra://xx.yy.xxx.yyy:9042
- name: db.password
value: 9__
- name: db.keyspaceName
value: somei
- name: db.username
value: supserawesome
ports:
- containerPort: 9000
- name: logging
image: busybox
volumeMounts:
- name: shared-logs
mountPath: /deploy/codingjediweb-1.0/logs/
command: ["tail -f /deploy/codingjediweb-1.0/logs/*.log"]
Before running tail -f ..., I want to add a sleep/delay to avoid a race condition (the application takes sometime before logging and tail -f fails in the meanwhile because the log file doesn't exist. Alternatively, I am ok to run a script like this - while true; do sleep 86400; done .
How can I do that?
got it - had to do command: ['sh', '-c', "while true; do sleep 86400; done"]

Kubernetes volumeMount folder and file permissions?

Trying to mount config files from a hostPath to a kubernetes container. This works using minikube and VirtualBox shared folder, but I am unable to make this work on Linux.
I making use of AWS EKS and the following architecture https://aws.amazon.com/quickstart/architecture/amazon-eks/. I think my problem is that the files need to live on each of the EKS Node instances.
Here is the architecture diagram:
Below is the Deployment file.
apiVersion: apps/v1
kind: Deployment
metadata:
name: openhim-core-deployment
spec:
replicas: 1
selector:
matchLabels:
component: openhim-core
template:
metadata:
labels:
component: openhim-core
spec:
volumes:
- name: core-config
hostPath:
path: /var/config/openhim-core
containers:
- name: openhim-core
image: jembi/openhim-core:5.rc
ports:
- containerPort: 8080
- containerPort: 5000
- containerPort: 5001
volumeMounts:
- name: core-config
mountPath: /usr/src/app/config
env:
- name: NODE_ENV
value: development
After much pain I found that I am trying to place the configuration on the Linux Bastion host where I have access to kubectl but in fact this configuration will have to be on each of the EC2 instances in every availability zone.
The solution for me was to make use of a initContainer.
apiVersion: apps/v1
kind: Deployment
metadata:
name: openhim-core-deployment
spec:
replicas: 1
selector:
matchLabels:
component: openhim-core
template:
metadata:
labels:
component: openhim-core
spec:
volumes:
- name: core-config
hostPath:
path: /var/config/openhim-core
containers:
- name: openhim-core
image: jembi/openhim-core:5
ports:
- containerPort: 8080
- containerPort: 5000
- containerPort: 5001
volumeMounts:
- name: core-config
mountPath: /usr/src/app/config
env:
- name: NODE_ENV
value: development
initContainers:
- name: install
image: busybox
command:
- wget
- "-O"
- "/usr/src/app/config/development.json"
- https://s3.eu-central-1.amazonaws.com/../development.json
volumeMounts:
- name: core-config
mountPath: "/usr/src/app/config"
volumes:
- name: core-config
emptyDir: {}