Kubernetes Private Repo Imagepullbackoff error - kubernetes

I followed this: https://www.linode.com/docs/guides/how-to-setup-a-private-docker-registry-with-lke-and-object-storage/
repo works I can push or pull from my local with user/pass. secret docker-registry regcred created and verified again and again.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-deployment
namespace: apps
spec:
selector:
matchLabels:
app: hello
replicas: 1
template:
metadata:
labels:
app: hello
spec:
containers:
- name: hello
image: privatrepo.ddns.net/hello
ports:
- containerPort: 3000
imagePullSecrets:
- name: regcred
any help?
here is the pod logs
Name:
Namespace: apps
Priority: 0
Node:
Start Time: Thu, 20 May 2021 14:13:27 +0200
Labels: app=hello
pod-template-hash=7d6674fdff
Annotations: cni.projectcalico.org/podIP: 10.2.0.88/32
Status: Pending
IP: 10.2.0.88
IPs:
IP: 10.2.0.88
Controlled By: ReplicaSet/
Containers:
hello:
Container ID:
Image: privatrepo.ddns.net/hello
Image ID:
Port: 3000/TCP
Host Port: 0/TCP
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-jlt9v (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-jlt9v:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-jlt9v
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal BackOff 24m (x680 over 179m) kubelet Back-off pulling image "privatrepo.ddns.net/hello"
Warning Failed 4m5s (x767 over 179m) kubelet Error: ImagePullBackOff

Related

How to use fdb-document-layer with kubernetes?

I have a fdb cluster using https://github.com/FoundationDB/fdb-kubernetes-operator and now trying to raise a pod with https://github.com/FoundationDB/fdb-document-layer
The result is CrashLoopBackOff of the pod
The description of the pod:
Name: fdb-doc-layer-84c4b84595-9rv8c
Namespace: default
Priority: 0
Node: faraday/5.188.158.233
Start Time: Sat, 21 Nov 2020 03:10:06 +0300
Labels: app=fdb-doc-layer
pod-template-hash=84c4b84595
Annotations: cni.projectcalico.org/podIP: 10.1.80.235/32
cni.projectcalico.org/podIPs: 10.1.80.235/32
Status: Running
IP: 10.1.80.235
IPs:
IP: 10.1.80.235
Controlled By: ReplicaSet/fdb-doc-layer-84c4b84595
Containers:
fdb-doc-layer:
Container ID: containerd://86f599ef8bd0684023a093f0e725fde02ac60f3899681053857e411b7c8c4b3b
Image: foundationdb/fdb-document-layer-build:latest
Image ID: docker.io/foundationdb/fdb-document-layer-build#sha256:5d1e84c5954141ce67be3fa28a428f572c3d8bbff1541ec8588fe82da600cb97
Port: 27017/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 21 Nov 2020 03:36:23 +0300
Finished: Sat, 21 Nov 2020 03:36:23 +0300
Ready: False
Restart Count: 10
Limits:
cpu: 200m
memory: 128Mi
Requests:
cpu: 200m
memory: 128Mi
Environment:
FDB_CLUSTER_FILE: /etc/foundationdb/fdb.cluster
Mounts:
/etc/foundationdb from config-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mf8pp (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sample-cluster-config
Optional: false
default-token-mf8pp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mf8pp
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 28m default-scheduler Successfully assigned default/fdb-doc-layer-84c4b84595-9rv8c to faraday
Normal Pulled 28m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.421941982s
Normal Pulled 28m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.223503462s
Normal Pulled 27m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.253710381s
Normal Pulled 27m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.672481437s
Normal Created 27m (x4 over 28m) kubelet Created container fdb-doc-layer
Normal Started 27m (x4 over 28m) kubelet Started container fdb-doc-layer
Normal Pulling 26m (x5 over 28m) kubelet Pulling image "foundationdb/fdb-document-layer-build:latest"
Normal Pulled 26m kubelet Successfully pulled image "foundationdb/fdb-document-layer-build:latest" in 1.270867366s
Warning BackOff 3m2s (x116 over 28m) kubelet Back-off restarting failed container
My deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: fdb-doc-layer
spec:
replicas: 1
selector:
matchLabels:
app: fdb-doc-layer
template:
metadata:
labels:
app: fdb-doc-layer
spec:
containers:
- name: fdb-doc-layer
image: foundationdb/fdb-document-layer-build:latest
env:
- name: FDB_CLUSTER_FILE
value: /etc/foundationdb/fdb.cluster
volumeMounts:
- name: config-volume
mountPath: /etc/foundationdb
resources:
limits:
memory: "128Mi"
cpu: "200m"
ports:
- containerPort: 27017
volumes:
- name: config-volume
configMap:
name: sample-cluster-config
How to make fdb-document-layer work with kubernetes?
There are two issues:
You are using the wrong image instead of foundationdb/fdb-document-layer-build:latest you should use foundationdb/fdb-document-layer:latest. The first image is only used to build the document layer.
The ConfigMap contains the fdb.cluster file under the key cluster-file so you need to remap this or adjust the env variable.
The following config works:
apiVersion: apps/v1
kind: Deployment
metadata:
name: fdb-doc-layer
spec:
replicas: 1
selector:
matchLabels:
app: fdb-doc-layer
template:
metadata:
labels:
app: fdb-doc-layer
spec:
containers:
- name: fdb-doc-layer
image: foundationdb/fdb-document-layer:latest
env:
- name: FDB_CLUSTER_FILE
value: /etc/foundationdb/fdb.cluster
volumeMounts:
- name: config-volume
mountPath: /etc/foundationdb
resources:
limits:
memory: "128Mi"
cpu: "200m"
ports:
- containerPort: 27017
volumes:
- name: config-volume
configMap:
name: sample-cluster-config
items:
- key: cluster-file
path: fdb.cluster

Some Kubernetes pods consistently not able to resolve internal DNS on only one node

I have just moved my first cluster from minikube up to AWS EKS. All went pretty smoothly so far, except I'm running into some DNS issues I think, but only on one of the cluster nodes.
I have two nodes in the cluster running v1.14, and 4 pods of one type, and 4 of another, 3 of each work, but 1 of each - both on the same node - start then error (CrashLoopBackOff) with the script inside the container erroring because it can't resolve the hostname for the database. Deleting the errored pod, or even all pods, results in one pod on the same node failing every time.
The database is in its own pod and has a service assigned, none of the other pods of the same type have problems resolving the name or connecting. The database pod is on the same node as the pods that can't resolve the hostname. I'm not sure how to migrate the pod to a different node, but that might be worth trying to see if the problem follows.
No errors in the coredns pods. I'm not sure where to start looking to discover the issue from here, and any help or suggestions would be appreciated.
Providing the configs below. As mentioned, they all work on Minikube, and also they work on one node.
kubectl get pods - note age, all pod1's were deleted at the same time and they recreated themselves, 3 worked fine, 4th does not.
NAME READY STATUS RESTARTS AGE
pod1-85f7968f7-2cjwt 1/1 Running 0 34h
pod1-85f7968f7-cbqn6 1/1 Running 0 34h
pod1-85f7968f7-k9xv2 0/1 CrashLoopBackOff 399 34h
pod1-85f7968f7-qwcrz 1/1 Running 0 34h
postgresql-865db94687-cpptb 1/1 Running 0 3d14h
rabbitmq-667cfc4cc-t92pl 1/1 Running 0 34h
pod2-94b9bc6b6-6bzf7 1/1 Running 0 34h
pod2-94b9bc6b6-6nvkr 1/1 Running 0 34h
pod2-94b9bc6b6-jcjtb 0/1 CrashLoopBackOff 140 11h
pod2-94b9bc6b6-t4gfq 1/1 Running 0 34h
postgresql service
apiVersion: v1
kind: Service
metadata:
name: postgresql
spec:
ports:
- port: 5432
selector:
app: postgresql
pod1 deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: pod1
spec:
replicas: 4
selector:
matchLabels:
app: pod1
template:
metadata:
labels:
app: pod1
spec:
containers:
- name: pod1
image: us.gcr.io/gcp-project-8888888/pod1:latest
env:
- name: rabbitmquser
valueFrom:
secretKeyRef:
name: rabbitmq-secrets
key: rmquser
volumeMounts:
- mountPath: /data/files
name: datafiles
volumes:
- name: datafiles
persistentVolumeClaim:
claimName: datafiles-pv-claim
imagePullSecrets:
- name: container-readonly
pod2 depoloyment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: pod2
spec:
replicas: 4
selector:
matchLabels:
app: pod2
template:
metadata:
labels:
app: pod2
spec:
containers:
- name: pod2
image: us.gcr.io/gcp-project-8888888/pod2:latest
env:
- name: rabbitmquser
valueFrom:
secretKeyRef:
name: rabbitmq-secrets
key: rmquser
volumeMounts:
- mountPath: /data/files
name: datafiles
volumes:
- name: datafiles
persistentVolumeClaim:
claimName: datafiles-pv-claim
imagePullSecrets:
- name: container-readonly
CoreDNS config map to forward DNS to external service if it doesn't resolve internally. This is the only place I can think that would be causing the issue - but as said it works for one node.
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
proxy . 8.8.8.8
cache 30
loop
reload
loadbalance
}
Errored Pod output. Same for both pods, as it occurs in library code common to both. As mentioned, this does not occur for all pods so the issue likely doesn't lie with the code.
Error connecting to database (psycopg2.OperationalError) could not translate host name "postgresql" to address: Try again
Errored Pod1 description:
Name: xyz-94b9bc6b6-jcjtb
Namespace: default
Priority: 0
Node: ip-192-168-87-230.us-east-2.compute.internal/192.168.87.230
Start Time: Tue, 15 Oct 2019 19:43:11 +1030
Labels: app=pod1
pod-template-hash=94b9bc6b6
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 192.168.70.63
Controlled By: ReplicaSet/xyz-94b9bc6b6
Containers:
pod1:
Container ID: docker://f7dc735111bd94b7c7b698e69ad302ca19ece6c72b654057627626620b67d6de
Image: us.gcr.io/xyz/xyz:latest
Image ID: docker-pullable://us.gcr.io/xyz/xyz#sha256:20110cf126b35773ef3a8656512c023b1e8fe5c81dd88f19a64c5bfbde89f07e
Port: <none>
Host Port: <none>
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 16 Oct 2019 07:21:40 +1030
Finished: Wed, 16 Oct 2019 07:21:46 +1030
Ready: False
Restart Count: 139
Environment:
xyz: <set to the key 'xyz' in secret 'xyz-secrets'> Optional: false
Mounts:
/data/xyz from xyz (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-m72kz (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
xyz:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: xyz-pv-claim
ReadOnly: false
default-token-m72kz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-m72kz
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 2m22s (x3143 over 11h) kubelet, ip-192-168-87-230.us-east-2.compute.internal Back-off restarting failed container
Errored Pod 2 description:
Name: xyz-85f7968f7-k9xv2
Namespace: default
Priority: 0
Node: ip-192-168-87-230.us-east-2.compute.internal/192.168.87.230
Start Time: Mon, 14 Oct 2019 21:19:42 +1030
Labels: app=pod2
pod-template-hash=85f7968f7
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 192.168.84.69
Controlled By: ReplicaSet/pod2-85f7968f7
Containers:
pod2:
Container ID: docker://f7c7379f92f57ea7d381ae189b964527e02218dc64337177d6d7cd6b70990143
Image: us.gcr.io/xyz-217300/xyz:latest
Image ID: docker-pullable://us.gcr.io/xyz-217300/xyz#sha256:b9cecdbc90c5c5f7ff6170ee1eccac83163ac670d9df5febd573c2d84a4d628d
Port: <none>
Host Port: <none>
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 16 Oct 2019 07:23:35 +1030
Finished: Wed, 16 Oct 2019 07:23:41 +1030
Ready: False
Restart Count: 398
Environment:
xyz: <set to the key 'xyz' in secret 'xyz-secrets'> Optional: false
Mounts:
/data/xyz from xyz (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-m72kz (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
xyz:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: xyz-pv-claim
ReadOnly: false
default-token-m72kz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-m72kz
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 3m28s (x9208 over 34h) kubelet, ip-192-168-87-230.us-east-2.compute.internal Back-off restarting failed container
At the suggestion of a k8s community member, I applied the following change to my coredns configuration to be more in line with the best practice:
Line: proxy . 8.8.8.8 changed to forward . /etc/resolv.conf 8.8.8.8
I then deleted the pods, and after they were recreated by k8s, the issue did not appear again.
EDIT:
Turns out, that was not the issue at all as shortly afterwards the issue re-occurred and persisted. In the end, it was this: https://github.com/aws/amazon-vpc-cni-k8s/issues/641
Rolled back to 1.5.3 as recommended by Amazon, restarted the cluster, and the issue was resolved.

what is Environment <none> in a kubectl describe pod command

When I do a kubectl describe pod, I can see
Environment: <none>
just after secrets. I wonder what it is. Is it possible to assign secrets to an environment? ( local, dev, staging, prod for instance ? )
➜ espace-client git:(master) ✗ kubectl describe pod -n espace-client espace-client-client-6b7b994b4c-gx58t
Name: espace-client-client-6b7b994b4c-gx58t
Namespace: espace-client
Priority: 0
Node: minikube/192.168.0.85
Start Time: Fri, 27 Sep 2019 11:37:06 +0200
Labels: app=espace-client-client
pod-template-hash=6b7b994b4c
Annotations: kubectl.kubernetes.io/restartedAt: 2019-09-27T11:37:06+02:00
Status: Running
IP: 172.17.0.21
IPs: <none>
Controlled By: ReplicaSet/espace-client-client-6b7b994b4c
Containers:
espace-client-client:
Container ID: docker://b3ee1efe45bb8ed9f27aca60e3bfecc1d7e29bc12600787d8d674ffb62ffc3f4
Image: espace_client_client:local
Image ID: docker://sha256:4cf73af7615ebfd30e7a8b0126154fa12b605dd34ead7cb0eefc43cd3ccc869b
Port: 3000/TCP
Host Port: 0/TCP
State: Running
Started: Fri, 27 Sep 2019 11:37:09 +0200
Ready: True
Restart Count: 0
Environment Variables from:
espace-client-client-env Secret Optional: false
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-lzb8h (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-lzb8h:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-lzb8h
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
The environment section contains any environment variables defined as part of the PodSpec:
apiVersion: v1
kind: Pod
metadata:
name: envar-demo
labels:
purpose: demonstrate-envars
spec:
containers:
- name: envar-demo-container
image: gcr.io/google-samples/node-hello:1.0
env:
- name: DEMO_GREETING
value: "Hello from the environment"
- name: DEMO_FAREWELL
value: "Such a sweet sorrow"
It is because most likely no Env vars where defined for the Pod. You can also assign Secrets to environment. They would show up in the Environment section like this:
SECURITY_JWT_PRIVATEKEY: <set to the key 'privateKey' in secret 'tokens'> Optional: false
For example:
apiVersion: v1
kind: Pod
metadata:
name: secrets-demo
labels:
purpose: demonstrate-secrets-in-env
spec:
containers:
- name: secret-demo-container
image: gcr.io/google-samples/node-hello:1.0
env:
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: password

Minikube MountVolume failed for volume "config-volume"

I'm facing an issue with my ubuntu:16.04 desktop, using minikube version: v0.35.0, kubectl both client/server: 1.13.4
When I try running a configmap example and get error kubelet, minikube MountVolume.SetUp failed every time I try. I tried to debug but no result on my end. It's about last 2 days and failed to make it work.
$ kubectl apply -f ./kube-pod.yml
apiVersion: v1
kind: Pod
metadata:
name: kuard-config
spec:
containers:
- name: test-container
image: gcr.io/kuar-demo/kuard-amd64:1
imagePullPolicy: Always
command:
- "/kuard"
- "$(EXTRA_PARAM)"
env:
- name: ANOTHER_PARAM
valueFrom:
configMapKeyRef:
name: my-config
key: another-param
- name: EXTRA_PARAM
valueFrom:
configMapKeyRef:
name: my-config
key: extra-param
volumeMounts:
- name: config-volume
mountPath: "/config"
volumes:
- name: config-volume
configMap:
name: my-config
restartPolicy: Never
Giving the following error when I run the following command
$ kubectl describe pods/kuard-config
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: minikube/10.0.2.15
Start Time: Thu, 20 Jun 2019 10:44:37 +0600
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Pod","metadata":{"ann
otations":{},"name":"kuard-config","namespace":"default"},"spec":{"containers":[{"command"...
Status: Pending
IP:
Containers:
test-container:
Container ID:
Image: gcr.io/kuar-demo/kuard-amd64:1
Image ID:
Port: <none>
Host Port: <none>
Command:
/kuard
$(EXTRA_PARAM)
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
ANOTHER_PARAM: <set to the key 'another-param' of config map 'my-config'> Optional: false
EXTRA_PARAM: <set to the key 'extra-param' of config map 'my-config'> Optional: false
Mounts:
/config from config-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-q42jl (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: my-config
Optional: false
default-token-q42jl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-q42jl
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12s default-scheduler Successfully assigned default/kuard-config to minikube
Warning FailedMount 4s (x5 over 11s) kubelet, minikube MountVolume.SetUp failed for volume "config-volume" : configmap "my-config" not found```
I'd expect if someone help me to figure out why `kubelet, minikube MountVolume.SetUp failed` is occur.

GKE with gcloud sql postgres: the sidecar proxy setup does not work

I am trying to setup a node.js app on GKE with a gcloud SQL Postgres database with a sidecar proxy. I am following along the docs but do not get it working. The proxy does not seem to be able to start (the app container does start). I have no idea why the proxy container can not start and also have no idea how to debug this (e.g. how do i get an error message!?).
mysecret.yaml:
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
username: [base64_username]
password: [base64_password]
Output of kubectl get secrets:
NAME TYPE DATA AGE
default-token-tbgsv kubernetes.io/service-account-token 3 5d
mysecret Opaque 2 7h
app-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
labels:
app: myapp
spec:
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: app
image: gcr.io/myproject/firstapp:v2
ports:
- containerPort: 8080
env:
- name: POSTGRES_DB_HOST
value: 127.0.0.1:5432
- name: POSTGRES_DB_USER
valueFrom:
secretKeyRef:
name: mysecret
key: username
- name: POSTGRES_DB_PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: password
- name: cloudsql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.11
command: ["/cloud_sql_proxy",
"-instances=myproject:europe-west4:databasename=tcp:5432",
"-credential_file=/secrets/cloudsql/mysecret.json"]
securityContext:
runAsUser: 2
allowPrivilegeEscalation: false
volumeMounts:
- name: cloudsql-instance-credentials
mountPath: /secrets/cloudsql
readOnly: true
volumes:
- name: cloudsql-instance-credentials
secret:
secretName: mysecret
output of kubectl create -f ./kubernetes/app-deployment.json:
deployment.apps/myapp created
output of kubectl get deployments:
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
myapp 1 1 1 0 5s
output of kubectl get pods:
NAME READY STATUS RESTARTS AGE
myapp-5bc965f688-5rxwp 1/2 CrashLoopBackOff 1 10s
output of kubectl describe pod/myapp-5bc955f688-5rxwp -n default:
Name: myapp-5bc955f688-5rxwp
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: gke-standard-cluster-1-default-pool-1ec52705-186n/10.164.0.4
Start Time: Sat, 15 Dec 2018 21:46:03 +0100
Labels: app=myapp
pod-template-hash=1675219244
Annotations: kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container app; cpu request for container cloudsql-proxy
Status: Running
IP: 10.44.1.9
Controlled By: ReplicaSet/myapp-5bc965f688
Containers:
app:
Container ID: docker://d3ba7ff9c581534a4d55a5baef2d020413643e0c2361555eac6beba91b38b120
Image: gcr.io/myproject/firstapp:v2
Image ID: docker-pullable://gcr.io/myproject/firstapp#sha256:80168b43e3d0cce6d3beda6c3d1c679cdc42e88b0b918e225e7679252a59a73b
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Sat, 15 Dec 2018 21:46:04 +0100
Ready: True
Restart Count: 0
Requests:
cpu: 100m
Environment:
POSTGRES_DB_HOST: 127.0.0.1:5432
POSTGRES_DB_USER: <set to the key 'username' in secret 'mysecret'> Optional: false
POSTGRES_DB_PASSWORD: <set to the key 'password' in secret 'mysecret'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-tbgsv (ro)
cloudsql-proxy:
Container ID: docker://96e2ed0de8fca21ecd51462993b7083bec2a31f6000bc2136c85842daf17435d
Image: gcr.io/cloudsql-docker/gce-proxy:1.11
Image ID: docker-pullable://gcr.io/cloudsql-docker/gce-proxy#sha256:5c690349ad8041e8b21eaa63cb078cf13188568e0bfac3b5a914da3483079e2b
Port: <none>
Host Port: <none>
Command:
/cloud_sql_proxy
-instances=myproject:europe-west4:databasename=tcp:5432
-credential_file=/secrets/cloudsql/mysecret.json
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 15 Dec 2018 22:43:37 +0100
Finished: Sat, 15 Dec 2018 22:43:37 +0100
Ready: False
Restart Count: 16
Requests:
cpu: 100m
Environment: <none>
Mounts:
/secrets/cloudsql from cloudsql-instance-credentials (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-tbgsv (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
cloudsql-instance-credentials:
Type: Secret (a volume populated by a Secret)
SecretName: mysecret
Optional: false
default-token-tbgsv:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-tbgsv
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 59m default-scheduler Successfully assigned default/myapp-5bc955f688-5rxwp to gke-standard-cluster-1-default-pool-1ec52705-186n
Normal Pulled 59m kubelet, gke-standard-cluster-1-default-pool-1ec52705-186n Container image "gcr.io/myproject/firstapp:v2" already present on machine
Normal Created 59m kubelet, gke-standard-cluster-1-default-pool-1ec52705-186n Created container
Normal Started 59m kubelet, gke-standard-cluster-1-default-pool-1ec52705-186n Started container
Normal Started 59m (x4 over 59m) kubelet, gke-standard-cluster-1-default-pool-1ec52705-186n Started container
Normal Pulled 58m (x5 over 59m) kubelet, gke-standard-cluster-1-default-pool-1ec52705-186n Container image "gcr.io/cloudsql-docker/gce-proxy:1.11" already present on machine
Normal Created 58m (x5 over 59m) kubelet, gke-standard-cluster-1-default-pool-1ec52705-186n Created container
Warning BackOff 4m46s (x252 over 59m) kubelet, gke-standard-cluster-1-default-pool-1ec52705-186n Back-off restarting failed container
EDIT: something seems wrong with my secret since when I do kubectl logs 5bc955f688-5rxwp cloudsql-proxy I get:
2018/12/16 22:26:28 invalid json file "/secrets/cloudsql/mysecret.json": open /secrets/cloudsql/mysecret.json: no such file or directory
I created the secret by doing:
kubectl create -f ./kubernetes/mysecret.yaml
I presume the secret is turned into JSON... When I change in app-deployment.yaml the mysecret.json into mysecret.yaml I still get similar error...
I was missing the correct key (credentials.json). It needs to be a key you generate from a service account; then you turn it into a secret. See also this issue.