Node Red on my k3 cluster using YAML not working - kubernetes

I am trying to deploy a simple node-red application to my k3s container. Background I have two pi 4's as my server and 3B+ as my workers.
For errors i get
NAME READY STATUS RESTARTS AGE
nodered-559875dbbd-mn474 0/1 CrashLoopBackOff 11 36m
Here are my yaml files
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.22.0 (955b78124)
creationTimestamp: null
labels:
io.kompose.service: nodered
name: nodered
namespace: nodered
spec:
replicas: 1
selector:
matchLabels:
io.kompose.service: nodered
strategy:
type: Recreate
template:
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.22.0 (955b78124)
creationTimestamp: null
labels:
io.kompose.service: nodered
spec:
containers:
- image: nodered/node-red
name: nodered
ports:
- containerPort: 1880
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 2
periodSeconds: 10
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 2
periodSeconds: 10
resources: {}
volumeMounts:
- mountPath: /data
name: nodered-claim0
restartPolicy: Always
securityContext:
runAsUser: 0
volumes:
- name: nodered-claim0
persistentVolumeClaim:
claimName: nodered-claim0
status: {}
---
apiVersion: v1
kind: Service
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.22.0 (955b78124)
creationTimestamp: null
labels:
io.kompose.service: nodered
name: nodered
namespace: nodered
spec:
ports:
- name: "1880"
port: 80
targetPort: 1880
selector:
io.kompose.service: nodered
status:
loadBalancer: {}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
labels:
io.kompose.service: nodered-claim0
name: nodered-claim0
namespace: nodered
spec:
storageClassName: longhorn
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
status: {}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nodered-ingress
namespace: nodered
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
rules:
- host: nodered.local
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nodered
port:
number: 80
Here is the error that I am getting when i type describe the node
Name: nodered-559875dbbd-mn474
Namespace: nodered
Priority: 0
Node: masternode/192.168.1.120
Start Time: Tue, 27 Apr 2021 10:38:28 -0400
Labels: io.kompose.service=nodered
pod-template-hash=559875dbbd
Annotations: kompose.cmd: kompose convert
kompose.version: 1.22.0 (955b78124)
Status: Running
IP: 10.42.1.197
IPs:
IP: 10.42.1.197
Controlled By: ReplicaSet/nodered-559875dbbd
Containers:
nodered:
Container ID: containerd://af3290fa34138a3361d5d73b7c1872eb1c523cefb0b8c17195db5d99c609f9fb
Image: nodered/node-red
Image ID: docker.io/nodered/node-red#sha256:f16e1ec7265829bcc381009dec175d9fbbc050ab6a1c42c4c906e689ff3bcf6b
Port: 1880/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 27 Apr 2021 10:38:52 -0400
Finished: Tue, 27 Apr 2021 10:38:52 -0400
Ready: False
Restart Count: 2
Liveness: http-get http://:80/ delay=2s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:80/ delay=2s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/data from nodered-claim0 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-j2f7l (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
nodered-claim0:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: nodered-claim0
ReadOnly: false
default-token-j2f7l:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-j2f7l
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 37s default-scheduler Successfully assigned nodered/nodered-559875dbbd-mn474 to masternode
Normal Pulled 33s kubelet Successfully pulled image "nodered/node-red" in 495.427449ms
Normal Pulled 32s kubelet Successfully pulled image "nodered/node-red" in 504.720641ms
Normal Pulling 14s (x3 over 34s) kubelet Pulling image "nodered/node-red"
Normal Pulled 14s kubelet Successfully pulled image "nodered/node-red" in 506.632077ms
Normal Created 13s (x3 over 33s) kubelet Created container nodered
Normal Started 13s (x3 over 33s) kubelet Started container nodered
Warning BackOff 6s (x5 over 30s) kubelet Back-off restarting failed container
Finally logs show the following:
internal/modules/cjs/loader.js:800
throw err;
^
SyntaxError: /usr/local/lib/node_modules/npm/package.json: Unexpected end of JSON input
at JSON.parse (<anonymous>)
at Object.Module._extensions..json (internal/modules/cjs/loader.js:797:27)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object.<anonymous> (/usr/local/lib/node_modules/npm/lib/utils/unsupported.js:3:17)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
I cant seem to traceback my log issue or atleast not sure where to look. Any help is apprecaited

For those wondering i got this to work. For some reason it was colliding with the install for node js...No idea why. Uninstalled nodejs and reinstalled and all is working. So nothing wrong with the config file above for those want to use it. Once you bind your config to a volume you can add authentication as well.

Related

Rabbitmq pod stuck in pending state

I tries running rabbitmq following the book Kubernetes for developers (page 180):
rabbitmq.yml
---
# EXPORT SERVICE INTERFACE
kind: Service
apiVersion: v1
metadata:
name: message-queue
labels:
app: rabbitmq
role: master
tier: queue
spec:
ports:
- port: 5672
targetPort: 5672
selector:
app: rabbitmq
role: master
tier: queue
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rabbitmq-pv-claim
labels:
app: rabbitmq
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rabbitmq
spec:
replicas: 1
selector:
matchLabels:
app: rabbitmq
role: master
tier: queue
template:
metadata:
labels:
app: rabbitmq
role: master
tier: queue
spec:
containers:
- name: rabbitmq
image: bitnami/rabbitmq:3.7
envFrom:
- configMapRef:
name: bitnami-rabbitmq-config
ports:
- name: queue
containerPort: 5672
- name: queue-mgmt
containerPort: 15672
livenessProbe:
exec:
command:
- rabbitmqctl
- status
initialDelaySeconds: 120
timeoutSeconds: 5
failureThreshold: 6
readinessProbe:
exec:
command:
- rabbitmqctl
- status
initialDelaySeconds: 10
timeoutSeconds: 3
periodSeconds: 5
volumeMounts:
- name: rabbitmq-storage
mountPath: /bitnami
volumes:
- name: rabbitmq-storage
persistentVolumeClaim:
claimName: rabbitmq-pv-claim
# kubectl describe pod rabbitmq-5499d4b67d-cdlb8
Name: rabbitmq-5499d4b67d-cdlb8
Namespace: default
Priority: 0
Node: <none>
Labels: app=rabbitmq
pod-template-hash=5499d4b67d
role=master
tier=queue
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/rabbitmq-5499d4b67d
Containers:
rabbitmq:
Image: bitnami/rabbitmq:3.7
Ports: 5672/TCP, 15672/TCP
Host Ports: 0/TCP, 0/TCP
Liveness: exec [rabbitmqctl status] delay=120s timeout=5s period=10s #success=1 #failure=6
Readiness: exec [rabbitmqctl status] delay=10s timeout=3s period=5s #success=1 #failure=3
Environment Variables from:
bitnami-rabbitmq-config ConfigMap Optional: false
Environment: <none>
Mounts:
/bitnami from rabbitmq-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xh899 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
rabbitmq-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: rabbitmq-pv-claim
ReadOnly: false
default-token-xh899:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xh899
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 28s (x13 over 12m) default-scheduler 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
The pod cannot be scheduled because the PVC that it is using is not "bound". You need to investigate why the PVC is not bound, something related to the storage system that you use.

Getting error mkdir: cannot create directory ‘/bitnami/rabbitmq’: Permission denied when creating Kubernetes pod of Rabbitmq

While learning Kubernetes going by the book Kubernetes for developer, I am stuck at this point now.
I am trying to start Rabbitmq pod but but after lot of troubleshooting I have managed to get to this point but do not get clue where do I fix to get rid of the permission denied error.
# kubectl get pods
NAME READY STATUS RESTARTS AGE
rabbitmq-56c67d8d7d-s8vp5 0/1 CrashLoopBackOff 5 5m40s
if I look at the logs of this contianer thats where I found:
# kubectl logs rabbitmq-56c67d8d7d-s8vp5
21:22:58.49
21:22:58.50 Welcome to the Bitnami rabbitmq container
21:22:58.51 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq
21:22:58.51 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues
21:22:58.52 Send us your feedback at containers#bitnami.com
21:22:58.52
21:22:58.52 INFO ==> ** Starting RabbitMQ setup **
21:22:58.54 INFO ==> Validating settings in RABBITMQ_* env vars..
21:22:58.56 INFO ==> Initializing RabbitMQ...
21:22:58.57 INFO ==> Generating random cookie
mkdir: cannot create directory ‘/bitnami/rabbitmq’: Permission denied
Here is my rabbitmq-deployment.yml
---
# EXPORT SERVICE INTERFACE
kind: Service
apiVersion: v1
metadata:
name: message-queue
labels:
app: rabbitmq
role: master
tier: queue
spec:
ports:
- port: 5672
targetPort: 5672
selector:
app: rabbitmq
role: master
tier: queue
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rabbitmq-pv-claim
labels:
app: rabbitmq
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rabbitmq
spec:
replicas: 1
selector:
matchLabels:
app: rabbitmq
role: master
tier: queue
template:
metadata:
labels:
app: rabbitmq
role: master
tier: queue
spec:
nodeSelector:
boardType: x86vm
containers:
- name: rabbitmq
image: bitnami/rabbitmq:3.7
envFrom:
- configMapRef:
name: bitnami-rabbitmq-config
ports:
- name: queue
containerPort: 5672
- name: queue-mgmt
containerPort: 15672
livenessProbe:
exec:
command:
- rabbitmqctl
- status
initialDelaySeconds: 120
timeoutSeconds: 5
failureThreshold: 6
readinessProbe:
exec:
command:
- rabbitmqctl
- status
initialDelaySeconds: 10
timeoutSeconds: 3
periodSeconds: 5
volumeMounts:
- name: rabbitmq-storage
mountPath: /bitnami
volumes:
- name: rabbitmq-storage
persistentVolumeClaim:
claimName: rabbitmq-pv-claim
This is the rabbitmq-storage-class.yml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: rabbitmq-storage-class
labels:
app: rabbitmq
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
and persistant-volume.yml
apiVersion: v1
kind: PersistentVolume
metadata:
name: rabbitmq-pv-claim
labels:
app: rabbitmq
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: /bitnami
Logs:
# kubectl describe pods rabbitmq-5f7f787479-fpg6g
Name: rabbitmq-5f7f787479-fpg6g
Namespace: default
Priority: 0
Node: kube-worker-vm2/192.168.1.36
Start Time: Mon, 03 May 2021 12:29:17 +0100
Labels: app=rabbitmq
pod-template-hash=5f7f787479
role=master
tier=queue
Annotations: cni.projectcalico.org/podIP: 192.168.222.4/32
cni.projectcalico.org/podIPs: 192.168.222.4/32
Status: Running
IP: 192.168.222.4
IPs:
IP: 192.168.222.4
Controlled By: ReplicaSet/rabbitmq-5f7f787479
Containers:
rabbitmq:
Container ID: docker://bbdbb9c5d4b6737519d3dcf4bdda242a7fe904f2336334afe686e9b204fd6d5c
Image: bitnami/rabbitmq:3.7
Image ID: docker-pullable://bitnami/rabbitmq#sha256:8b6057997b74ebc81e934dd6c94e9da745635faa2d79b382cfda27b9176e0e6d
Ports: 5672/TCP, 15672/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 03 May 2021 12:30:48 +0100
Finished: Mon, 03 May 2021 12:30:48 +0100
Ready: False
Restart Count: 4
Liveness: exec [rabbitmqctl status] delay=120s timeout=5s period=10s #success=1 #failure=6
Readiness: exec [rabbitmqctl status] delay=10s timeout=3s period=5s #success=1 #failure=3
Environment Variables from:
bitnami-rabbitmq-config ConfigMap Optional: false
Environment: <none>
Mounts:
/bitnami from rabbitmq-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-4qmxr (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
rabbitmq-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: rabbitmq-pv-claim
ReadOnly: false
default-token-4qmxr:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-4qmxr
Optional: false
QoS Class: BestEffort
Node-Selectors: boardType=x86vm
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m20s default-scheduler Successfully assigned default/rabbitmq-5f7f787479-fpg6g to kube-worker-vm2
Normal Created 96s (x4 over 2m18s) kubelet Created container rabbitmq
Normal Started 95s (x4 over 2m17s) kubelet Started container rabbitmq
Warning
BackOff 65s (x12 over 2m16s) kubelet Back-off restarting failed container
Normal Pulled 50s (x5 over 2m18s) kubelet Container image "bitnami/rabbitmq:3.7" already present on machine
When creating an image, the image creator often chooses to use a user other than root to run the process. This is the case for your image, and the user does not have write permissions on the /bitnami directory. You can verify this by commenting out the volume.
To fix the issue, you need to set a security contect for your pod: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
Not sure about the exact syntax, but something like this should do the trick:
spec:
securityContext:
fsGroup: 1001 # the userid that is used in the image
nodeSelector:
boardType: x86vm
containers:
- name: rabbitmq
image: bitnami/rabbitmq:3.7
envFrom:
- configMapRef:
name: bitnami-rabbitmq-config
This makes the directory writeable by the user in the image.
Another thing: A deployment is for stateless services by design. If you have state to keep, always use a statefulset. It's very similiar to a deployment from a configuration point of view, but Kubernetes treats it very differently. See https://www.youtube.com/watch?v=Vrxr-7rjkvM for good explanation.
As per bitnami documentation, it depends on the kubernetes distribution
Quote from documentation
Adjust permissions of persistent volume mountpoint
As the image run as non-root by default, it is necessary to adjust the ownership of the persistent volume so that the container can write data into it.
By default, the chart is configured to use Kubernetes Security Context to automatically change the ownership of the volume. However, this feature does not work in all Kubernetes distributions. As an alternative, this chart supports using an initContainer to change the ownership of the volume before mounting it in the final destination.
You can enable this initContainer by setting volumePermissions.enabled to true.

How to use fdb-document-layer with kubernetes?

I have a fdb cluster using https://github.com/FoundationDB/fdb-kubernetes-operator and now trying to raise a pod with https://github.com/FoundationDB/fdb-document-layer
The result is CrashLoopBackOff of the pod
The description of the pod:
Name: fdb-doc-layer-84c4b84595-9rv8c
Namespace: default
Priority: 0
Node: faraday/5.188.158.233
Start Time: Sat, 21 Nov 2020 03:10:06 +0300
Labels: app=fdb-doc-layer
pod-template-hash=84c4b84595
Annotations: cni.projectcalico.org/podIP: 10.1.80.235/32
cni.projectcalico.org/podIPs: 10.1.80.235/32
Status: Running
IP: 10.1.80.235
IPs:
IP: 10.1.80.235
Controlled By: ReplicaSet/fdb-doc-layer-84c4b84595
Containers:
fdb-doc-layer:
Container ID: containerd://86f599ef8bd0684023a093f0e725fde02ac60f3899681053857e411b7c8c4b3b
Image: foundationdb/fdb-document-layer-build:latest
Image ID: docker.io/foundationdb/fdb-document-layer-build#sha256:5d1e84c5954141ce67be3fa28a428f572c3d8bbff1541ec8588fe82da600cb97
Port: 27017/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 21 Nov 2020 03:36:23 +0300
Finished: Sat, 21 Nov 2020 03:36:23 +0300
Ready: False
Restart Count: 10
Limits:
cpu: 200m
memory: 128Mi
Requests:
cpu: 200m
memory: 128Mi
Environment:
FDB_CLUSTER_FILE: /etc/foundationdb/fdb.cluster
Mounts:
/etc/foundationdb from config-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mf8pp (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sample-cluster-config
Optional: false
default-token-mf8pp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mf8pp
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 28m default-scheduler Successfully assigned default/fdb-doc-layer-84c4b84595-9rv8c to faraday
Normal Pulled 28m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.421941982s
Normal Pulled 28m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.223503462s
Normal Pulled 27m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.253710381s
Normal Pulled 27m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.672481437s
Normal Created 27m (x4 over 28m) kubelet Created container fdb-doc-layer
Normal Started 27m (x4 over 28m) kubelet Started container fdb-doc-layer
Normal Pulling 26m (x5 over 28m) kubelet Pulling image "foundationdb/fdb-document-layer-build:latest"
Normal Pulled 26m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.270867366s
Warning BackOff 3m2s (x116 over 28m) kubelet Back-off restarting failed container
My deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: fdb-doc-layer
spec:
replicas: 1
selector:
matchLabels:
app: fdb-doc-layer
template:
metadata:
labels:
app: fdb-doc-layer
spec:
containers:
- name: fdb-doc-layer
image: foundationdb/fdb-document-layer-build:latest
env:
- name: FDB_CLUSTER_FILE
value: /etc/foundationdb/fdb.cluster
volumeMounts:
- name: config-volume
mountPath: /etc/foundationdb
resources:
limits:
memory: "128Mi"
cpu: "200m"
ports:
- containerPort: 27017
volumes:
- name: config-volume
configMap:
name: sample-cluster-config
How to make fdb-document-layer work with kubernetes?
There are two issues:
You are using the wrong image instead of foundationdb/fdb-document-layer-build:latest you should use foundationdb/fdb-document-layer:latest. The first image is only used to build the document layer.
The ConfigMap contains the fdb.cluster file under the key cluster-file so you need to remap this or adjust the env variable.
The following config works:
apiVersion: apps/v1
kind: Deployment
metadata:
name: fdb-doc-layer
spec:
replicas: 1
selector:
matchLabels:
app: fdb-doc-layer
template:
metadata:
labels:
app: fdb-doc-layer
spec:
containers:
- name: fdb-doc-layer
image: foundationdb/fdb-document-layer:latest
env:
- name: FDB_CLUSTER_FILE
value: /etc/foundationdb/fdb.cluster
volumeMounts:
- name: config-volume
mountPath: /etc/foundationdb
resources:
limits:
memory: "128Mi"
cpu: "200m"
ports:
- containerPort: 27017
volumes:
- name: config-volume
configMap:
name: sample-cluster-config
items:
- key: cluster-file
path: fdb.cluster

How to add an username when create a k8s pod

I have a yaml file for creating k8s pod with just one container. Is it possible to pre-add an username and its password from yaml file during k8s pod creation?
I searched many sites and found the env variable. However, I could not make the pod as my wish. The pod's status is always showing Crashoff after pod creation.
Is it possible to pre-add an username and its password from yaml file during k8s pod creation?
Following are my yaml file:
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: centos610-sp-v1
spec:
replicas: 1
template:
metadata:
labels:
app: centos610-sp-v1
spec:
containers:
- name: centos610-pod-v1
image: centos-done:6.10
env:
- name: SSH_USER
value: "user1"
- name: SSH_SUDO
value: "ALL=(ALL) NOPASSWD:ALL"
- name: PASSWORD
value: "password"
command: ["/usr/sbin/useradd"]
args: ["$(SSH_USER)"]
ports:
- containerPort: 22
resources:
limits:
cpu: "500m"
memory: "1G"
---
apiVersion: v1
kind: Service
metadata:
name: centos610-sp-v1
labels:
app: centos610-sp-v1
spec:
selector:
app: centos610-sp-v1
ports:
- port: 22
protocol: TCP
nodePort: 31022
type: NodePort
---
Should I use specific command as
env:
- name: MESSAGE
value: "hello world"
command: ["/bin/echo"]
args: ["$(MESSAGE)"]
or
command: ["/bin/sh"]
args: ["-c", "while true; do echo hello; sleep 10;done"]
pod's status after get
root#zero:~/k8s-temp# kubectl get pod
NAME READY STATUS RESTARTS AGE
centos610-sp-v1-6689c494b8-nb9kv 0/1 CrashLoopBackOff 5 3m
pod's status after describe
root#zero:~/k8s-temp# kubectl describe pod centos610-sp-v1-6689c494b8-nb9kv
Name: centos610-sp-v1-6689c494b8-nb9kv
Namespace: default
Node: zero/10.111.33.15
Start Time: Sat, 16 Mar 2019 01:16:59 +0800
Labels: app=centos610-sp-v1
pod-template-hash=2245705064
Annotations: <none>
Status: Running
IP: 10.233.127.104
Controlled By: ReplicaSet/centos610-sp-v1-6689c494b8
Containers:
centos610-pod-v1:
Container ID: docker://5fa076c5d245dd532ef7ce724b94033d93642dc31965ab3fbde61dd59bf7d314
Image: centos-done:6.10
Image ID: docker://sha256:26362e9cefe4e140933bf947e3beab29da905ea5d65f27fc54513849a06d5dd5
Port: 22/TCP
Host Port: 0/TCP
Command:
/usr/sbin/useradd
Args:
$(SSH_USER)
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 16 Mar 2019 01:17:17 +0800
Finished: Sat, 16 Mar 2019 01:17:17 +0800
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 16 Mar 2019 01:17:01 +0800
Finished: Sat, 16 Mar 2019 01:17:01 +0800
Ready: False
Restart Count: 2
Limits:
cpu: 500m
memory: 1G
Requests:
cpu: 500m
memory: 1G
Environment:
SSH_USER: user1
SSH_SUDO: ALL=(ALL) NOPASSWD:ALL
PASSWORD: password
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-qbd8x (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-qbd8x:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-qbd8x
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 22s default-scheduler Successfully assigned centos610-sp-v1-6689c494b8-nb9kv to zero
Normal SuccessfulMountVolume 22s kubelet, zero MountVolume.SetUp succeeded for volume "default-token-qbd8x"
Normal Pulled 5s (x3 over 21s) kubelet, zero Container image "centos-done:6.10" already present on machine
Normal Created 5s (x3 over 21s) kubelet, zero Created container
Normal Started 4s (x3 over 21s) kubelet, zero Started container
Warning BackOff 4s (x3 over 19s) kubelet, zero Back-off restarting failed container
2019/03/18 UPDATE
Although pre-add username and password from pod's yaml is not suggested but I just want to clarify how to use command & args from yaml file. Finally, I use following yaml file to create a username "user1" and its password "1234" successfully. Thank you all of you guys' great answer to make me more familiar with k8s about configMap, RBAC, container's behavior.
Actually, this link gave me a reference on how to use command & args
How to set multiple commands in one yaml file with Kubernetes?
Here are my final yaml file content:
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: centos610-sp-v1
spec:
replicas: 1
template:
metadata:
labels:
app: centos610-sp-v1
spec:
containers:
- name: centos610-pod-v1
image: centos-done:6.10
env:
- name: SSH_USER
value: "user1"
- name: SSH_SUDO
value: "ALL=(ALL) NOPASSWD:ALL"
- name: PASSWORD
value: "password"
command: ["/bin/bash", "-c"]
args: ["useradd $(SSH_USER); service sshd restart; echo $(SSH_USER):1234 | chpasswd; tail -f /dev/null"]
ports:
- containerPort: 22
resources:
limits:
cpu: "500m"
memory: "1G"
---
apiVersion: v1
kind: Service
metadata:
name: centos610-sp-v1
labels:
app: centos610-sp-v1
spec:
selector:
app: centos610-sp-v1
ports:
- port: 22
protocol: TCP
nodePort: 31022
type: NodePort
---
Keep username and password in a configMap or in a secret object. Load those values into container as environment variables
Follow the reference
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/
If you want to add the user anyway , regardless of the fact that you can achive the same thing using A kubernetes native way , then
Please setup your user in the Docker image ( Dockerfile and then build it) instead.
Hope this helps.
2019/03/18 UPDATE
Although pre-add username and password from pod's yaml is not suggested but I just want to clarify how to use command & args from yaml file. Finally, I use following yaml file to create a username "user1" and its password "1234" successfully. Thank you all of you guys' great answer to make me more familiar with k8s about configMap, RBAC, container's behavior.
Actually, this link gave me a reference on how to use command & args
How to set multiple commands in one yaml file with Kubernetes?
Here are my final yaml file content:
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: centos610-sp-v1
spec:
replicas: 1
template:
metadata:
labels:
app: centos610-sp-v1
spec:
containers:
- name: centos610-pod-v1
image: centos-done:6.10
env:
- name: SSH_USER
value: "user1"
- name: SSH_SUDO
value: "ALL=(ALL) NOPASSWD:ALL"
- name: PASSWORD
value: "password"
command: ["/bin/bash", "-c"]
args: ["useradd $(SSH_USER); service sshd restart; echo $(SSH_USER):1234 | chpasswd; tail -f /dev/null"]
ports:
- containerPort: 22
resources:
limits:
cpu: "500m"
memory: "1G"
---
apiVersion: v1
kind: Service
metadata:
name: centos610-sp-v1
labels:
app: centos610-sp-v1
spec:
selector:
app: centos610-sp-v1
ports:
- port: 22
protocol: TCP
nodePort: 31022
type: NodePort
---

kubernetes Pod's readinessProbe errored but endpoint not removed from Service

I'm running Spinnaker on Kubernetes 1.10.111. One of the Spinnaker services is a Pod running a service called Clouddriver. This Pod was running fine, but then the readinessProbe started erroring continuously. Kubernetes docs say
readinessProbe: Indicates whether the Container is ready to service requests. If the readiness probe fails, the endpoints controller removes the Pod’s IP address from the endpoints of all Services that match the Pod.
— https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
But this Pod's IP is still in the Service's endpoints. Why?
Clouddriver Pod YAML
kubectl -n spinnaker-test get pods spin-clouddriver-5559d44484-mp8q9 -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/psp: spotify.backend-service
creationTimestamp: 2019-02-15T20:46:38Z
generateName: spin-clouddriver-5559d44484-
labels:
app: spin
app.kubernetes.io/managed-by: halyard
app.kubernetes.io/name: clouddriver
app.kubernetes.io/part-of: spinnaker
app.kubernetes.io/version: 1.12.1
cluster: spin-clouddriver
pod-template-hash: "1115800040"
name: spin-clouddriver-5559d44484-mp8q9
namespace: spinnaker-test
ownerReferences:
- apiVersion: extensions/v1beta1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: spin-clouddriver-5559d44484
uid: ce79561c-3161-11e9-acdf-42010a800082
resourceVersion: "53541277"
selfLink: /api/v1/namespaces/spinnaker-test/pods/spin-clouddriver-5559d44484-mp8q9
uid: caa66d7c-3162-11e9-acdf-42010a800082
spec:
containers:
- env:
- name: JAVA_OPTS
value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2
- name: SPRING_PROFILES_ACTIVE
value: local
image: gcr.io/spinnaker-marketplace/clouddriver:4.3.1-20190130095322
imagePullPolicy: IfNotPresent
lifecycle: {}
name: clouddriver
ports:
- containerPort: 7002
protocol: TCP
readinessProbe:
exec:
command:
- wget
- --no-check-certificate
- --spider
- -q
- http://localhost:7002/health
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "20"
memory: 5000Mi
requests:
cpu: "20"
memory: 5000Mi
securityContext:
allowPrivilegeEscalation: false
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/spinnaker/config
name: spin-clouddriver-files-1952526246
- mountPath: /home/halyard/.hal/k8s-spinnaker/staging/dependencies
name: spin-clouddriver-files-1757773194
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-w2lt5
readOnly: true
dnsPolicy: ClusterFirst
nodeName: gke-production-us-ce-terraform-201812-d63606d6-9vq9
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 720
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: spin-clouddriver-files-1952526246
secret:
defaultMode: 420
secretName: spin-clouddriver-files-1952526246
- name: spin-clouddriver-files-1757773194
secret:
defaultMode: 420
secretName: spin-clouddriver-files-1757773194
- name: default-token-w2lt5
secret:
defaultMode: 420
secretName: default-token-w2lt5
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2019-02-15T20:46:38Z
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: 2019-02-15T20:53:40Z
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: 2019-02-15T20:46:38Z
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://3509b48511b1ea7bc97812cb82831c559d9410cb9eaaa26b4f492d881603fb31
image: gcr.io/spinnaker-marketplace/clouddriver:4.3.1-20190130095322
imageID: docker-pullable://gcr.io/spinnaker-marketplace/clouddriver#sha256:466228b97b8c4a61a0270c53ae4c397eb04bc3661bc4f1ee9ef4d5fce70d187d
lastState: {}
name: clouddriver
ready: true
restartCount: 0
state:
running:
startedAt: 2019-02-15T20:47:26Z
hostIP: 10.178.32.98
phase: Running
podIP: 10.179.34.24
qosClass: Guaranteed
startTime: 2019-02-15T20:46:38Z
Describing the Pod shows the readinessProbe has been continuously erroring for over a day.
kubectl -n spinnaker-test describe pods spin-clouddriver-5559d44484-mp8q9
Name: spin-clouddriver-5559d44484-mp8q9
Namespace: spinnaker-test
Node: gke-production-us-ce-terraform-201812-d63606d6-9vq9/10.178.32.98
Start Time: Fri, 15 Feb 2019 15:46:38 -0500
Labels: app=spin
app.kubernetes.io/managed-by=halyard
app.kubernetes.io/name=clouddriver
app.kubernetes.io/part-of=spinnaker
app.kubernetes.io/version=1.12.1
cluster=spin-clouddriver
pod-template-hash=1115800040
Annotations: kubernetes.io/psp=spotify.backend-service
Status: Running
IP: 10.179.34.24
Controlled By: ReplicaSet/spin-clouddriver-5559d44484
Containers:
clouddriver:
Container ID: docker://3509b48511b1ea7bc97812cb82831c559d9410cb9eaaa26b4f492d881603fb31
Image: gcr.io/spinnaker-marketplace/clouddriver:4.3.1-20190130095322
Image ID: docker-pullable://gcr.io/spinnaker-marketplace/clouddriver#sha256:466228b97b8c4a61a0270c53ae4c397eb04bc3661bc4f1ee9ef4d5fce70d187d
Port: 7002/TCP
Host Port: 0/TCP
State: Running
Started: Fri, 15 Feb 2019 15:47:26 -0500
Ready: True
Restart Count: 0
Limits:
cpu: 20
memory: 5000Mi
Requests:
cpu: 20
memory: 5000Mi
Readiness: exec [wget --no-check-certificate --spider -q http://localhost:7002/health] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
JAVA_OPTS: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2
SPRING_PROFILES_ACTIVE: local
Mounts:
/home/halyard/.hal/k8s-spinnaker/staging/dependencies from spin-clouddriver-files-1757773194 (rw)
/opt/spinnaker/config from spin-clouddriver-files-1952526246 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-w2lt5 (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
spin-clouddriver-files-1952526246:
Type: Secret (a volume populated by a Secret)
SecretName: spin-clouddriver-files-1952526246
Optional: false
spin-clouddriver-files-1757773194:
Type: Secret (a volume populated by a Secret)
SecretName: spin-clouddriver-files-1757773194
Optional: false
default-token-w2lt5:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-w2lt5
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 3m (x321 over 1d) kubelet, gke-production-us-ce-terraform-201812-d63606d6-9vq9 Readiness probe errored: rpc error: code = DeadlineExceeded desc = context deadline exceeded
But Service still has the Pod's IP of 10.179.34.24 in its Endpoints.
kubectl -n spinnaker-test describe services spin-clouddriver
Name: spin-clouddriver
Namespace: spinnaker-test
Labels: app=spin
cluster=spin-clouddriver
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"spin","cluster":"spin-clouddriver"},"name":"spin-clouddriver","namesp...
Selector: app=spin,cluster=spin-clouddriver
Type: ClusterIP
IP: 10.178.65.100
Port: <unset> 7002/TCP
TargetPort: 7002/TCP
Endpoints: 10.179.34.24:7002
Session Affinity: None
Events: <none>
kubectl -n spinnaker-test describe endpoints spin-clouddriver
Name: spin-clouddriver
Namespace: spinnaker-test
Labels: app=spin
cluster=spin-clouddriver
Annotations: <none>
Subsets:
Addresses: 10.179.34.24
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 7002 TCP
Events: <none>
footnotes
GKE 1.10.11-gke.1 to be exact, but the fact that it's GKE shouldn't matter.
A probe by the kubelet can end in one of three states:
successful
failed (command returned a non-0 exit code)
errored (command did not return before the timeout elapsed, the command does not exist inside the container, etc)
Here is the code (in 1.10.11) where the event probe errored is recorded. Note that err != nil.
Here is the code that calls the above function - when err != nil (the probe returned an error), the result is discarded.
Only probes that fail will actually cause the pod's ready state to be changed.