Kubernetes pull from local registry not working, how to fix that? - kubernetes

Question

I followed this link and created a secret as below.
kubectl create secret docker-registry regsec --docker-server=192.168.56.106:5000 --docker-username=osboxes --docker-password=osboxes.org --insecure-skip-tls-verify=true
And the deployment as below.
kubectl create deploy nginx1 --image 192.168.56.106:5000/todoapp:1.0
And edited it using
kubectl edit deploy nginx1
And added imagePullSecrets to it
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "2"
creationTimestamp: "2021-07-21T10:23:23Z"
generation: 2
labels:
app: nginx1
name: nginx1
namespace: default
resourceVersion: "6872"
uid: 0b6917f0-10ac-4206-82a8-c49ae8ffa2b3
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: nginx1
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: nginx1
spec:
containers:
- image: 192.168.56.106:5000/todoapp:1.0
imagePullPolicy: IfNotPresent
name: todoapp
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: regsec
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
conditions:
- lastTransitionTime: "2021-07-21T10:23:23Z"
lastUpdateTime: "2021-07-21T10:23:23Z"
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: "False"
type: Available
- lastTransitionTime: "2021-07-21T10:23:23Z"
lastUpdateTime: "2021-07-21T10:28:36Z"
message: ReplicaSet "nginx1-75df7fd466" is progressing.
reason: ReplicaSetUpdated
status: "True"
type: Progressing
observedGeneration: 2
replicas: 2
unavailableReplicas: 2
updatedReplicas: 1
But still getting below error.
osboxes#osboxes:~/Desktop$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-6799fc88d8-vvzjf 1/1 Running 0 19m
nginx1-65d848d94f-dd4ck 0/1 ImagePullBackOff 0 12m
nginx1-75df7fd466-kn5mf 0/1 ImagePullBackOff 0 6m50s
osboxes#osboxes:~/Desktop$ kubectl describe pod nginx1-75df7fd466-kn5mf
Name: nginx1-75df7fd466-kn5mf
Namespace: default
Priority: 0
Node: samples-control-plane/172.19.0.3
Start Time: Wed, 21 Jul 2021 06:28:36 -0400
Labels: app=nginx1
pod-template-hash=75df7fd466
Annotations: <none>
Status: Pending
IP: 10.244.0.8
IPs:
IP: 10.244.0.8
Controlled By: ReplicaSet/nginx1-75df7fd466
Containers:
todoapp:
Container ID:
Image: 192.168.56.106:5000/todoapp:1.0
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-88clq (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-88clq:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 73s default-scheduler Successfully assigned default/nginx1-75df7fd466-kn5mf to samples-control-plane
Normal Pulling 33s (x3 over 73s) kubelet Pulling image "192.168.56.106:5000/todoapp:1.0"
Warning Failed 33s (x3 over 73s) kubelet Failed to pull image "192.168.56.106:5000/todoapp:1.0": rpc error: code = Unknown desc = failed to pull and unpack image "192.168.56.106:5000/todoapp:1.0": failed to resolve reference "192.168.56.106:5000/todoapp:1.0": failed to do request: Head "https://192.168.56.106:5000/v2/todoapp/manifests/1.0": http: server gave HTTP response to HTTPS client
Warning Failed 33s (x3 over 73s) kubelet Error: ErrImagePull
Normal BackOff 6s (x4 over 72s) kubelet Back-off pulling image "192.168.56.106:5000/todoapp:1.0"
Warning Failed 6s (x4 over 72s) kubelet Error: ImagePullBackOff
kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-13T02:40:46Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-21T23:01:33Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
Adding docker insecure registry is not useful as the container runtime is not docker now.It is containerd.
/etc/default/docker.json
And for reference, I have setup the local container on a virtualbox VM and connecting to kubernetes on the same network using Host-only network ip address.
The setup of local registry is created using the steps mentioned in link.
And the kubernetes was setup using kind.

Answer 1

cri-o
If you are using cri-o as Container Runtime Provider, docker settings are of no use. You would require to configure cri-o instead: https://github.com/cri-o/cri-o#configuration
Assuming you have cri-o installed on the host node, the official documentation recommends to have a $HOME/.config/containers/registries.conf file, or a global, /etc/containers/registries.conf file to configure registries in a runtime engine agnostic way.
The file structure spec is also documented: https://github.com/containers/image/blob/main/docs/containers-registries.conf.5.md
Example:
[registries.search]
registries = ['registry1.com', 'registry2.com']
[registries.insecure]
registries = ['registry3.com']
[registries.block]
registries = ['registry.untrusted.com', 'registry.unsafe.com']
containerd
Containerd does not seem to acknowledge the /etc/containers/registries.conf settings.
As per the documentation, The main config file /etc/containerd/config.toml can specify a registry config path as:
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
And inside the path specified above (/etc/containerd/certs.d) create a directory for host (docker.io) and in it, create a hosts.toml file. That is /etc/containerd/certs.d/docker.io/hosts.toml:
server = "https://registry-1.docker.io"
[host."http://my-custom-registry:5000"]
capabilities = ["pull", "resolve", "push"]
skip_verify = true
plain-http = true
For my-custom-registry:5000 section you can also provide credentials as well as certificates.
And restart containerd daemon/service on the host.
Configuration file spec is here: https://github.com/containerd/containerd/blob/main/docs/hosts.md