I am trying to add a file to a pod's disk during initialization of the pod but without luck. Below is my deployment file which I use to deploy the pod. The file gets downloaded to the persistent volume, but the pod doesn't get into ready state. After a few seconds, the pods fail and get rebuilt. Which kicks off the whole process again.
Any help would be appreciated.
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: mapserver
spec:
selector:
matchLabels:
app: mapserver
template:
metadata:
labels:
app: mapserver
spec:
volumes:
- name: storage
persistentVolumeClaim:
claimName: mapserver-pv-claim
containers:
- name: maptiles
image: klokantech/tileserver-gl
command: ["/bin/sh"]
args:
- -c
- |
echo "[INFO] Startingcontainer"; if [ $(DOWNLOAD_MBTILES) = "true" ]; then
echo "[INFO] Download MBTILES_PLANET_URL";
rm /data/*
cd /data/
curl -k -sSL -X GET -u user:ww $(MBTILES_PLANET_URL) -O
echo "[INFO] Download finished";
fi;
env:
- name: MBTILES_PLANET_URL
value: 'https://abc-dev/nexus/repository/xyz-raw/2017-07-03_europe_netherlands.mbtiles'
- name: DOWNLOAD_MBTILES
value: 'true'
livenessProbe:
failureThreshold: 120
httpGet:
path: /health
port: 80
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 5
ports:
- containerPort: 80
name: http
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: 80
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: 300m
memory: 3Gi
requests:
cpu: 100m
memory: 1Gi
volumeMounts:
- mountPath: "/data"
name: storage
am trying to add a file to a pod's disk during initialization of the pod but without luck.
In that case you might want to use InitContainers instead.
Judging from your manifest, your main command gets executed (copies the file and then exits) terminating the container (and accompanying pod) in the process. Deployment then restarts the exited pod and cycle repeats. If you use InitContainers instead (with the same definition and same PV as you are doing now for main container) you should then prepopulate data using InitContaienrs that runs to completion and then continue to use it in your normal container (that should have non-exiting main process as its command/entry point).
Note: if you don't want to use InitContainers or just as a quick test, you could append a regular non-exiting command after your copy statement, and also, check if you need to start container with tty, depending on your use case and ways to keep container up and running.
Related
I've set up Kubernetes DaemonSet Manifest for handling metrics in my project, and having a little problem of pinging this DaemonSet, So my Eventual Question is, (If I have a daemonset in every pod of my project, how can I ping Specific One or Just the One In order to make sure that everything okay with it?) (Most Likely using curl, but would also consider some other ways too if it's possible:)
Example of DaemonSet I have
apiVersion: v1
kind: DaemonSet
metadata:
name: metrics-api-service
spec:
selector:
matchLabels:
app: api-metrics-app
template:
metadata:
labels:
app: api-metrics-app
spec:
terminationGracePeriodSeconds: 60
containers:
- name: api-metrics-service
image: image
livenessProbe:
exec:
< what I need to put here In order to ping the DaemonSet ? >
initialDelaySeconds: 60
resources:
limits:
cpu: "0.5"
memory: "500Mi"
requests:
cpu: "0.4"
memory: "200Mi"
Thanks
Healthcheck should be enough to making sure if its working or not, but if you still want to confirm from the external side then make sure your daemon set exposes the port and the security group allow internal traffic, you can ping/curl same node where the daemon set is running, every daemonset will get the node IP as an environment variable.
Pass the HOST_IP in the daemonset environment variable
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
and then update the liveness probe accordingly
livenessProbe:
exec:
command:
- sh
- -c
- curl -s $HOST_IP:PORT/healthcheck
I will recommend HTTP check over bash
healthcheck:
initialDelaySeconds: 100
periodSeconds: 10
timeoutSeconds: 5
httpGet:
path: /metrics
port: 3000
if /metrics seems working, just return 200 status code.
One way to do this would be to lookup for your Pods IP addresses, then query each of them.
for i in $(kubectl get pods -n logging get pods \
-l app=api-metrics-app \
-o jsonpath='{.items[*].status.podIP}')
do
curl http://$i:4242/metrics
done
I have pod that includes one init container and one app container,
between them there is a volume with shared folder.
My issue is that once a day or even more, the init container run itself and therefore it delete the node modules from the volume, then the main app crash because of missing modules.
The app container is not making any restarts, only the init container.
is anyone familiar with this issue in k8s? why those restarts happens only in the init container?
Thanks :)
edit:
the deployment yaml file -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "25"
creationTimestamp: "2020-05-19T06:48:18Z"
generation: 25
labels:
apps: ******
commit: ******
name: *******
namespace: fleet
resourceVersion: "24059934"
selfLink: *******
uid: *******
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: *******
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: *******
commit: *******
revision: *******
spec:
containers:
image: XXXXXXXXXXXX
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: http
scheme: HTTP
initialDelaySeconds: 120
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 30
name: *******
ports:
- containerPort: 1880
name: http
protocol: TCP
readinessProbe:
failureThreshold: 20
httpGet:
path: /ping
port: http
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 30
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/breeze
name: workdir
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: flowregistrykey
initContainers:
image: XXXXXXXXXXXX
imagePullPolicy: IfNotPresent
name: get-flow-json
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /work-dir
name: workdir
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- emptyDir: {}
name: workdir
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2020-06-01T12:30:10Z"
lastUpdateTime: "2020-06-01T12:30:10Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2020-05-19T06:48:18Z"
lastUpdateTime: "2020-06-01T12:45:05Z"
message: ReplicaSet "collection-associator.sandbox.services.collection-8784dcb9d"
has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 25
readyReplicas: 1
replicas: 1
updatedReplicas: 1
and this image explain the problem-
the pod was created 7 days ago, but the files inside were created today, and there is no node_modules folder - because only the init container run again and not the app container so there was no mpm install
the init container run itself and therefore it delete the node modules from the volume
initContainer is only run when a pod restart. Don't treat it as service or application. It should be a script, a job only for setup before your application.
then the main app crash because of missing modules.
node_modules is not dynamic loading. It's loaded when you npm start
You might want to try livenessProbe.
Many applications running for long periods of time eventually transition to broken states, and cannot recover except by being restarted. Kubernetes provides liveness probes to detect and remedy such situations.
initContainers:
- name: set-delete-time
command: ["/bin/sh","-c"]
args:
- |
# Let's set 3600
echo 3600 > deleteTime
...
containers:
- name: node-app
livenessProbe:
exec:
command:
# If deleteTime > 0 exit 0
# Else /tmp/deleteTime - $periodSeconds > /tmp/deleteTime
- cat
- /tmp/deleteTime
- ...
initialDelaySeconds: 5
periodSeconds: 5
I am trying to setup mongodb on kubernetes with istio. My statefulset is as follows:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: treeservice
namespace: staging
spec:
serviceName: tree-service-service
replicas: 1
selector:
matchLabels:
app: treeservice
template:
metadata:
labels:
app: treeservice
spec:
containers:
- name: mongodb-cache
image: mongo:latest
imagePullPolicy: Always
ports:
- containerPort: 30010
volumeMounts:
- name: mongodb-cache-data
mountPath: /data/db
resources:
requests:
memory: "4Gi" # 4 GB
cpu: "1000m" # 1 CPUs
limits:
memory: "4Gi" # 4 GB
cpu: "1000" # 1 CPUs
readinessProbe:
exec:
command:
- mongo
- --eval "db.stats()" --port 30010
initialDelaySeconds: 60 #wait this period after staring fist time
periodSeconds: 30 # polling interval every 5 minutes
timeoutSeconds: 60
livenessProbe:
exec:
command:
- mongo
- --eval "db.stats()" --port 30010
initialDelaySeconds: 60 #wait this period after staring fist time
periodSeconds: 30 # polling interval every 5 minutes
timeoutSeconds: 60
command: ["/bin/bash"]
args: ["-c","mongod --port 30010 --replSet test"] #bind to localhost
volumeClaimTemplates:
- metadata:
name: mongodb-cache-data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: fast
resources:
requests:
storage: 300Gi
however, the pod is not created and I see the following error:
kubectl describe statefulset treeservice -n staging
Warning FailedCreate 1m (x159 over 1h) statefulset-controller create Pod treeservice-0 in StatefulSet treeservice failed error: Pod "treeservice-0" is invalid: spec.containers[1].env[7].name: Invalid value: "ISTIO_META_statefulset.kubernetes.io/pod-name": a valid environment variable name must consist of alphabetic characters, digits, '_', '-', or '.', and must not start with a digit (e.g. 'my.env-name', or 'MY_ENV.NAME', or 'MyEnvName1', regex used for validation is '[-._a-zA-Z][-._a-zA-Z0-9]*')
I assum treeservice is a valid pod name. Am I missing something?
I guess it's due to this issue https://github.com/istio/istio/issues/9571 which is still open
I made it work temporarily using the following:
annotations:
sidecar.istio.io/inject: "false"
I would like to run specific command after initialization of deployment is successful.
This is my yaml file:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: auth
spec:
replicas: 1
template:
metadata:
labels:
app: auth
spec:
containers:
- name: auth
image: {{my-service-image}}
env:
- name: NODE_ENV
value: "docker-dev"
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 3000
However, I would like to run command for db migration after (not before) deployment is successfully initialized and pods are running.
I can do it manually for every pod (with kubectl exec), but this is not very scalable.
I resolved it using lifecycles:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: auth
spec:
replicas: 1
template:
metadata:
labels:
app: auth
spec:
containers:
- name: auth
image: {{my-service-image}}
env:
- name: NODE_ENV
value: "docker-dev"
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 3000
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", {{cmd}}]
You can use Helm to deploy a set of Kubernetes resources. And then, use a Helm hook, e.g. post-install or post-upgrade, to run a Job in a separate docker container. Set your Job to invoke db migration. A Job will run >=1 Pods to completion, so it fits here quite well.
I chose to use a readinessProbe
My application requires configuration after the process has completely started.
The postStart command was running before the app was ready.
readinessProbe:
exec:
command: [healthcheck]
initialDelaySeconds: 30
periodSeconds: 2
timeoutSeconds: 1
successThreshold: 3
failureThreshold: 10
When running a deployment I get downtime. Requests failing after a variable amount of time (20-40 seconds).
The readiness check for the entry container fails when the preStop sends SIGUSR1, waits for 31 seconds, then sends SIGTERM. In that timeframe the pod should be removed from the service as the readiness check is set to fail after 2 failed attempts with 5 second intervals.
How can I see the events for pods being added and removed from the service to find out what's causing this?
And events around the readiness checks themselves?
I use Google Container Engine version 1.2.2 and use GCE's network load balancer.
service:
apiVersion: v1
kind: Service
metadata:
name: myapp
labels:
app: myapp
spec:
type: LoadBalancer
ports:
- name: http
port: 80
targetPort: http
protocol: TCP
- name: https
port: 443
targetPort: https
protocol: TCP
selector:
app: myapp
deployment:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 3
strategy:
type: RollingUpdate
revisionHistoryLimit: 10
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
version: 1.0.0-61--66-6
spec:
containers:
- name: myapp
image: ****
resources:
limits:
cpu: 100m
memory: 250Mi
requests:
cpu: 10m
memory: 125Mi
ports:
- name: http-direct
containerPort: 5000
livenessProbe:
httpGet:
path: /status
port: 5000
initialDelaySeconds: 30
timeoutSeconds: 1
lifecycle:
preStop:
exec:
# SIGTERM triggers a quick exit; gracefully terminate instead
command: ["sleep 31;"]
- name: haproxy
image: travix/haproxy:1.6.2-r0
imagePullPolicy: Always
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 25Mi
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
env:
- name: "SSL_CERTIFICATE_NAME"
value: "ssl.pem"
- name: "OFFLOAD_TO_PORT"
value: "5000"
- name: "HEALT_CHECK_PATH"
value: "/status"
volumeMounts:
- name: ssl-certificate
mountPath: /etc/ssl/private
livenessProbe:
httpGet:
path: /status
port: 443
scheme: HTTPS
initialDelaySeconds: 30
timeoutSeconds: 1
readinessProbe:
httpGet:
path: /readiness
port: 81
initialDelaySeconds: 0
timeoutSeconds: 1
periodSeconds: 5
successThreshold: 1
failureThreshold: 2
lifecycle:
preStop:
exec:
# SIGTERM triggers a quick exit; gracefully terminate instead
command: ["kill -USR1 1; sleep 31; kill 1"]
volumes:
- name: ssl-certificate
secret:
secretName: ssl-c324c2a587ee-20160331
When the probe fails, the prober will emit a warning event with reason as Unhealthy and message as xx probe errored: xxx.
You should be able to find those events using either kubectl get events or kubectl describe pods -l app=myapp,version=1.0.0-61--66-6 (filter pods by its label).