Can not pull any image in minikube - kubernetes

Question

Im on macOS and im using minikube with hyperkit driver: minikube start --driver=hyperkit
and everything seems ok...
with minikube status:
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
with minikube version:
minikube version: v1.24.0
with kubectl version:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:35:25Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
and with kubectl get no:
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane,master 13m v1.22.3
my problem is when i deploy anything, it wont pull any image...
for instance:
kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.4
then kubectl get pods:
NAME READY STATUS RESTARTS AGE
hello-minikube-6ddfcc9757-nfc64 0/1 ImagePullBackOff 0 13m
then i tried to figure out what is the problem?
k describe pod/hello-minikube-6ddfcc9757-nfc64
here is the result:
Name: hello-minikube-6ddfcc9757-nfc64
Namespace: default
Priority: 0
Node: minikube/192.168.64.8
Start Time: Sun, 16 Jan 2022 10:49:27 +0330
Labels: app=hello-minikube
pod-template-hash=6ddfcc9757
Annotations: <none>
Status: Pending
IP: 172.17.0.5
IPs:
IP: 172.17.0.5
Controlled By: ReplicaSet/hello-minikube-6ddfcc9757
Containers:
echoserver:
Container ID:
Image: k8s.gcr.io/echoserver:1.4
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k5qql (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-k5qql:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 18m default-scheduler Successfully assigned default/hello-minikube-6ddfcc9757-nfc64 to minikube
Normal Pulling 16m (x4 over 18m) kubelet Pulling image "k8s.gcr.io/echoserver:1.4"
Warning Failed 16m (x4 over 18m) kubelet Failed to pull image "k8s.gcr.io/echoserver:1.4": rpc error: code = Unknown desc = Error response from daemon: Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 16m (x4 over 18m) kubelet Error: ErrImagePull
Warning Failed 15m (x6 over 18m) kubelet Error: ImagePullBackOff
Normal BackOff 3m34s (x59 over 18m) kubelet Back-off pulling image "k8s.gcr.io/echoserver:1.4"
then tried to get some logs!:
k logs pod/hello-minikube-6ddfcc9757-nfc64 and k logs deploy/hello-minikube
both returns the same result:
Error from server (BadRequest): container "echoserver" in pod "hello-minikube-6ddfcc9757-nfc64" is waiting to start: trying and failing to pull image
this deployment was an example from minikube documentation
but i have no idea why it doesnt pull any image...

Answer 1

I had exactly same problem.
I found out that my internet connection was slow,
the timout to pull an image is 120 seconds, so kubectl could not pull the image in under 120 seconds.
first use minikube to pull the image you need
for example:
minikube image load k8s.gcr.io/echoserver:1.4
and then everything will work because now kubectl will use the image that is stored locally.

Answer 2

According to this article:
The status ImagePullBackOff means that a Pod couldn’t start, because Kubernetes couldn’t pull a container image. The ‘BackOff’ part means that Kubernetes will keep trying to pull the image, with an increasing delay (‘back-off’).
Here is also a handbook about pushing images into a minikube cluster.
This handbook describes your issue:
Unable to pull images..Client.Timeout exceeded while awaiting headers
Unable to pull images, which may be OK:
failed to pull image "k8s.gcr.io/kube-apiserver:v1.13.3": output: Error response from daemon:
Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection
(Client.Timeout exceeded while awaiting headers)
This error indicates that the container runtime running within the VM does not have access to the internet.
See possible workarounds.

Answer 3

I encountered some similar issue, it is fixed by using echoserver:1.10 instead of echoserver:1.4