Deploying multiple container in kubernetes through a deployed app - kubernetes

I am using minikube for deployment in my local machine.
I am deploying an app with help of helm charts. My deployment script looks like
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "app.fullname" . }}
spec:
replicas: {{ .Values.replicaCount }}
spec:
volumes:
- name: dockersock
hostPath:
path: "/var/run/docker.sock"
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
volumeMounts:
- name: dockersock
mountPath: "/var/run/docker.sock"
command: ["/bin/sh", "-c"]
args:
- python3 runMyApp.py;
resources:
limits:
nvidia.com/gpu: {{ .Values.numGpus }}
My script runMyApp.py when executed launch 4 other containers and I want that Kubernetes deploy them as well at minikube and this is expected behavior from my understanding.
But When I other 4 containers they were deployed on host machine, just like I executed some docker run command on my host machine.
To verify that I was not mistaken,
so I tried to access them through other application using which is in minikube cluster but I couldn't. Then I tried accessing the application from the local environment and I was able to do that.
Is there any flaw in my understanding. If the behaviour is expected then what can I do to deploy other application in k8s as well.

In order to create new pods from within your python script, you should use the Kubernetes Python client library: https://github.com/kubernetes-client/python .
When you running docker run inside your script, kubernetes isn't aware of those containers, and they are just orphand containers running on your node host machine.

Related

Command is executed before env mount

I'm very new to Kubernetes so sorry if i'm not explaining my problem right.
I'm trying to spin up 3 replicas of a pod that run a php command. After a while the command should crash and restart.
The problem is that it starts with the local .env the first few times, after a few restarts the mounted .env is used. When it fails and restarts it launches with the wrong local env again.
I suspect the the command is run before the mount, what should I try to mount before my entrypoint command starts?
apiVersion: apps/v1
kind: Deployment
spec:
template:
metadata:
labels:
app.kubernetes.io/name: project
app.kubernetes.io/instance: project-release
spec:
imagePullSecrets: {{ toYaml .Values.gitlab.secrets | nindent 8 }}
containers:
- name: project
image: {{ .Values.gitlab.image }}
imagePullPolicy: IfNotPresent
command: [ "/bin/sh","-c" ]
args: [ "bin/console php:command:name" ]
volumeMounts:
- name: env
mountPath: /var/www/deploy/env
volumes:
- name: env
secret:
secretName: project-env

K8S - livenessProbe - Restart pod if another pod is not ready / working ( MySql)

Happy New Year , I have a 2 deployments , MySQL and Application , My Application is depends on the MySQL pod , I have initContainers that make sure that the Application runs after the MySQL pod fully app and ready , but I'm trying to make the next scenario working.
I want the Application pod to check the MySQL pod and if the port 3306 is not available then the Application pod himself will restart , and this will keep happens until the MySQL pod will be fully ready.
I'm using this in the Application deployment / pod
livenessProbe:
httpGet:
host: ???
path: /
port: 3306
but instead of " ??? " I don't know what I need to write , because , i know that I can not write their DNS name , I was told that livenessProbe does not work with DNS , so I tried to enter this IP address for ENV , but still its not working.
how can I do this ?
SQL Deployment yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.deployment.mysql.name }}
namespace: {{ .Values.namespace }}
spec:
selector:
matchLabels:
app: {{ .Values.deployment.mysql.name }}
strategy:
type: Recreate
template:
metadata:
labels:
app: {{ .Values.deployment.mysql.name }}
spec:
containers:
- image: {{ .Values.deployment.mysql.image }}
name: {{ .Values.deployment.mysql.name }}
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: mysql-root-password
ports:
- containerPort: {{ .Values.deployment.mysql.port }}
name: {{ .Values.deployment.mysql.name }}
volumeMounts:
- name: sqlvol
mountPath: /var/lib/mysql/
readOnly: false
# - name: db
# mountPath: /etc/notebook-db/
# command:
# - mysql < /etc/notebook-db/crud.sql
livenessProbe:
tcpSocket:
port: 3306
initialDelaySeconds: 15
periodSeconds: 20
initContainers:
- name: init-myservice
image: busybox:1.28
command: ['sh', '-c', "sleep 10"]
volumes:
- name: sqlvol
persistentVolumeClaim:
claimName: mysqlvolume
readOnly: false
Application Deployment yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.deployment.nodejs.name }}
namespace: {{ .Values.namespace }}
labels:
app: {{ .Values.deployment.nodejs.name }}
name: {{ .Values.deployment.nodejs.name }}
spec:
replicas: 1
selector:
matchLabels:
app: {{ .Values.deployment.nodejs.name }}
template:
metadata:
labels:
app: {{ .Values.deployment.nodejs.name }}
spec:
containers:
- name: {{ .Values.deployment.nodejs.name }}
image: {{ .Values.deployment.nodejs.image }}:{{ .Values.deployment.nodejs.tag }}
ports:
- containerPort: {{ .Values.deployment.nodejs.targetPort }}
livenessProbe:
httpGet:
host: $MYSQL_CLUSTERIP_SERVICE_HOST
path: /
port: 3306
initContainers:
- name: init-myservice
image: busybox:1.28
command: ['sh', '-c', "sleep 60"]
$MYSQL_CLUSTERIP_SERVICE_HOST - this is ENV ( it did not worked for me this way ) .
so how can i restart pod application if the pod mysql is not ready ?
Create a service for MySQL Deployment , So that it can solve two problems
Service IP does not change
dns lookups & reverse lookups works fine with services
Example:
kubectl get svc -n default
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx ClusterIP 10.104.97.252 <none> 8080/TCP 33m
FQDN of the above service (in the fomat : <nameoftheservice>.<namespace>.cluster.local ):
nginx.default.svc.cluster.local
Then use the FQDN of the service as host in your livenessProbe in your application deployment.
liveness probe for with respect above service:
livenessProbe:
httpGet:
host: nginx.default.svc.cluster.local
path: /
port: 8080
Is your application supposed to crash if MySQL is not ready ? If it's the case as your application is a Deployment the self-healing will be applied by Kubernetes so your Pod will restart until restart is oki (in fact with a maximum number of retries that you can configure) .
TL;DR
DNS doesn't work for liveness probes, the kubelet network space cannot basically resolve any in-cluster DNS.
You can consider putting both of your services in a single pod as sidecars. This way they would share the same address space if one container fails then the whole pod is restarted.
Another option is to create an operator 🔧 for your pods/application and basically have it check the liveness through the in-cluster DNS for both pods separately and restart the pods through the Kubernetes API.
You can also just create your own script in a pod that just calls curl to check for a 200 OK and kubectl to restart your pod if you get something else.
Note that for the 2 options above you need to make sure that Coredns is stable and solid otherwise your health checks might fail to make your services have potential downtime.
from : Liveness-Probe of one pod via another

Kubernetes copying jars into a pod and restart

I have a Kubernetes problem where I need to copy 2 jars (each jar > 1Mb) into a pod after it is deployed. So ideally the solution is we cannot use configMap (> 1Mb) but we need to use "wget" in "initcontainer" and download the jars.
so below is my kubernetes-template configuration which i have modified. The original one is available at https://github.com/dremio/dremio-cloud-tools/blob/master/charts/dremio/templates/dremio-executor.yaml
{{ if not .Values.DremioAdmin }}
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: dremio-executor
spec:
serviceName: "dremio-cluster-pod"
replicas: {{.Values.executor.count}}
podManagementPolicy: "Parallel"
revisionHistoryLimit: 1
selector:
matchLabels:
app: dremio-executor
template:
metadata:
labels:
app: dremio-executor
role: dremio-cluster-pod
annotations:
dremio-configmap/checksum: {{ (.Files.Glob "config/*").AsConfig | sha256sum }}
spec:
terminationGracePeriodSeconds: 5
{{- if .Values.nodeSelector }}
nodeSelector:
{{- range $key, $value := .Values.nodeSelector }}
{{ $key }}: {{ $value }}
{{- end }}
{{- end }}
containers:
- name: dremio-executor
image: {{.Values.image}}:{{.Values.imageTag}}
imagePullPolicy: IfNotPresent
securityContext:
runAsUser: 0
resources:
requests:
memory: {{.Values.executor.memory}}M
cpu: {{.Values.executor.cpu}}
volumeMounts:
- name: dremio-executor-volume
mountPath: /opt/dremio/data
##################### START added this section #####################
- name: dremio-connector
mountPath: /opt/dremio/jars
#################### END added this section ##########################
- name: dremio-config
mountPath: /opt/dremio/conf
env:
- name: DREMIO_MAX_HEAP_MEMORY_SIZE_MB
value: "{{ template "HeapMemory" .Values.executor.memory }}"
- name: DREMIO_MAX_DIRECT_MEMORY_SIZE_MB
value: "{{ template "DirectMemory" .Values.executor.memory }}"
- name: DREMIO_JAVA_EXTRA_OPTS
value: >-
-Dzookeeper=zk-hs:2181
-Dservices.coordinator.enabled=false
{{- if .Values.extraStartParams }}
{{ .Values.extraStartParams }}
{{- end }}
command: ["/opt/dremio/bin/dremio"]
args:
- "start-fg"
ports:
- containerPort: 45678
name: server
initContainers:
################ START added this section ######################
- name: installjars
image: {{.Values.image}}:{{.Values.imageTag}}
imagePullPolicy: IfNotPresent
securityContext:
runAsUser: 0
volumeMounts:
- name: dremio-connector
mountPath: /opt/dremio/jars
command: ["/bin/sh","-c"]
args: ["wget --no-check-certificate -O /dir/connector.jar https://<some nexus repo URL>/connector.jar; sleep 10;"]
################ END added this section ###############
- name: wait-for-zk
image: busybox
command: ["sh", "-c", "until ping -c 1 -W 1 zk-hs > /dev/null; do echo waiting for zookeeper host; sleep 2; done;"]
# since we're mounting a separate volume, reset permission to
# dremio uid/gid
- name: chown-data-directory
image: {{.Values.image}}:{{.Values.imageTag}}
imagePullPolicy: IfNotPresent
securityContext:
runAsUser: 0
volumeMounts:
- name: dremio-executor-volume
mountPath: /opt/dremio/data
command: ["chown"]
args:
- "dremio:dremio"
- "/opt/dremio/data"
volumes:
- name: dremio-config
configMap:
name: dremio-config
{{- if .Values.imagePullSecrets }}
imagePullSecrets:
- name: {{ .Values.imagePullSecrets }}
{{- end}}
#################### START added this section ########################
- name: dremio-connector
emptyDir: {}
#################### END added this section ########################
volumeClaimTemplates:
- metadata:
name: dremio-executor-volume
spec:
accessModes: [ "ReadWriteOnce" ]
{{- if .Values.storageClass }}
storageClassName: {{ .Values.storageClass }}
{{- end }}
resources:
requests:
storage: {{.Values.executor.volumeSize}}
{{ end }}
So the above is NOT working and I don't see any jars being downloaded once I "exec" into the pod. I don't understand what is wrong with the above. however do note that inside the pod if i run the same wget command it does download the jar which baffles me. So the URL is working, readwrite of directory is no problem but still jar is not downloaded ???
If you can remove the need for Wget altogether it would make life easier...
Option 1
Using your own docker image will save some pain if thats an option
Dockerfile
# docker build -f Dockerfile -t ghcr.io/yourOrg/projectId/dockerImageName:0.0.1 .
# docker push ghcr.io/yourOrg/projectId/dockerImageName:0.0.1
FROM nginx:1.19.10-alpine
# Use local copies of config
COPY files/some1.jar /dir/
COPY files/some2.jar /dir/
Files will be ready in the container, no need for cryptic commands in your pod definition that will make little sense. Alternatively if you need to download the files you could copy a script to do that work into the Docker image instead and run that on startup via the docker directive CMD.
Option 2
Alternatively, you could do a two stage deployment...
Create a persistent volume
mount the volume to a pod (use busybox as a base?) that will run for enough time for the files to copy across from your local machine (or for them to be downloaded if you continue to use Wget)
kubectl cp the files you need to the (Retained) PersistentVolume
Now mount the PV to your pod's container(s) so the files are readily available when the pod fires up.
Your approch seems right.
Another solution could be to include the jar on the Docker image but I think it's not possible right ?
You could just use an emptyDir instead of a VolumeClaim.
Last one, I would have download the jar before waiting for ZooKeeper to gain some time.

Running scripts via Helm hooks

I have written Pre- and Post-upgrade hooks for my Helm chart, which will get invoked when I do a helm upgrade. My Pre-upgrade hook is supposed to write some information to a file in the shared persistent storage volume. Somehow, I dont see this file getting created though I am able to see the hook getting invoked.
This is what my pre-upgrade hook looks like:
apiVersion: batch/v1
kind: Job
metadata:
name: "{{.Release.Name}}-preupgrade"
labels:
heritage: {{.Release.Service | quote }}
release: {{.Release.Name | quote }}
chart: "{{.Chart.Name}}-{{.Chart.Version}}"
annotations:
"helm.sh/hook": pre-upgrade
"helm.sh/hook-weight": "0"
"helm.sh/hook-delete-policy": hook-succeeded
spec:
template:
metadata:
name: "{{.Release.Name}}"
labels:
heritage: {{.Release.Service | quote }}
release: {{.Release.Name | quote }}
chart: "{{.Chart.Name}}-{{.Chart.Version}}"
spec:
restartPolicy: Never
containers:
- name: pre-upgrade-job
image: {{ .Values.registry }}/{{ .Values.imageRepo }}:{{ .Values.imageTag }}
imagePullPolicy: {{ .Values.imagePullPolicy }}
volumeMounts:
- mountPath: {{ .Values.pvc.shared_storage_path }}/{{ template "fullname" . }}
name: shared-pvc
command: ['/bin/sh -c scripts/preUpgradeScript.sh {{ .Values.pvc.shared_storage_path }}/{{ template "fullname" . }}']
volumes:
- name: shared-pvc
persistentVolumeClaim:
claimName: {{ template "fullname" . }}-shared-pv-claim
My expectation is that the hook should be able to write information to the PVC volume which was already created prior to the upgrade. When I did a describe on the upgrade pods, I could see the following error:
Error: failed to start container "pre-upgrade-job": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"/bin/sh -c scripts/preUpgradeScript.sh /opt/flink/share/myfl-flink\": stat /bin/sh -c scripts/preUpgradeScript.sh /opt/flink/share/myfl-flink: no such file or directory"
Doesn't the hook first mount the volume before running the command? Also, I'm packaging the script with the docker image, so I believe it should be there.
I am unable to exec into the hook pod as it goes into the Failed state.
Can anyone help me with this?
[Update] I added a sleep command to enter the pod and check if the script is available and if the mount path exists. All looks fine. I don't understand why this error would come up.
Looks like I needed to give the command differently:
command: ["/bin/sh", "-c", "scripts/preUpgradeScript.sh","{{ .Values.pvc.shared_storage_path }}/{{ template "fullname" . }}"]

K8s TLS Secret for Postgres | GKE & Google Cloud SQL Postgres

I'm having troubles establishing a SSL connection between a web service and a remotely hosted Postgres database. With the same cert and key files being used for the web service, I can connect to the database with tools such as pgAdmin and DataGrip. These files were downloaded from Postgres instance in the Google Cloud Console.
Issue:
At the time of Spring Boot service start up, the following error occurs:
org.postgresql.util.PSQLException: Could not read SSL key file /tls/tls.key
Where I look at the Postgres server logs, the error is recorded as
LOG: could not accept SSL connection: UNEXPECTED_RECORD
Setup:
Spring Boot service running on Minikube (local) and GKE connecting to a Google Cloud SQL Postgres instance.
Actions Taken:
I have downloaded the client cert & key. I created a K8s TLS Secret using the downloaded client cert & key. I also have made sure the files can be read from the volume mount by running the following command on the k8s deployment config:
command: ["bin/sh", "-c", "cat /tls/tls.key"]
Here is the datasource url which is fed in via an environment variable (DATASOURCE).
"jdbc:postgresql://[Database-Address]:5432/[database]?ssl=true&sslmode=require&sslcert=/tls/tls.crt&sslkey=/tls/tls.key"
Here is the k8s deployment yaml, any idea where i'm going wrong?
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ template "service.name" . }}
labels:
release: {{ template "release.name" . }}
chart: {{ template "chart.name" . }}
chart-version: {{ template "chart.version" . }}
release: {{ template "service.fullname" . }}
spec:
replicas: {{ $.Values.image.replicaCount }}
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
template:
metadata:
labels:
app: {{ template "service.name" . }}
release: {{ template "release.name" . }}
env: {{ $.Values.environment }}
spec:
imagePullSecrets:
- name: {{ $.Values.image.pullSecretsName }}
containers:
- name: {{ template "service.name" . }}
image: {{ $.Values.image.repo }}:{{ $.Values.image.tag }}
# command: ["bin/sh", "-c", "cat /tls/tls.key"]
imagePullPolicy: {{ $.Values.image.pullPolicy }}
volumeMounts:
- name: tls-cert
mountPath: "/tls"
readOnly: true
ports:
- containerPort: 80
env:
- name: DATASOURCE_URL
valueFrom:
secretKeyRef:
name: service
key: DATASOURCE_URL
- name: DATASOURCE_USER
valueFrom:
secretKeyRef:
name: service
key: DATASOURCE_USER
- name: DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
name: service
key: DATASOURCE_PASSWORD
volumes:
- name: tls-cert
projected:
sources:
- secret:
name: postgres-tls
items:
- key: tls.crt
path: tls.crt
- key: tls.key
path: tls.key
So I figured it out, I was asking the wrong question!
Google Cloud SQL has a proxy component for the Postgres database. Therefore, trying to connect the traditional way (the problem I was trying to solve) has been resolved by implementing proxy. Instead of dealing with whitelisting IPs, SSL certs, and such, you just spin up the proxy, point it at a GCP credential file, then updated your database uri to access via localhost.
To set up the proxy, you can find directions here. There is a good example of a k8s deployment file here.
One situation I did come across was the GCP service account. Make sure to add Cloud SQL Client AND Cloud SQL Editor roles. I only added the Cloud SQL Client to start with and kept getting the 403 error.