I try to configure Kubernetes to pull images from our private Artifactory Docker repo.
First I configured a secret with kubectl:
kubectl create secret docker-registry artifactorysecret --docker-server=ourcompany.jfrog.io/path/list/docker-repo/ --docker-username=artifactory-user --docker-password=artipwd --docker-email=myemail
After creating a pod using kubectl with
apiVersion: v1
kind: Pod
metadata:
name: base-infra
spec:
containers:
- name: api-gateway
image: api-gateway
imagePullSecrets:
- name: artifactorysecret
I get a "ImagePullBackOff" error in Kubernetes:
3m 3m 1 default-scheduler Normal
Scheduled Successfully assigned consort-base-infra to k8s-agent-ab2f29b2-2
3m 0s 5 kubelet, k8s-agent-ab2f29b2-2 spec.containers{api-gateway} Normal
Pulling pulling image "api-gateway"
2m <invalid> 5 kubelet, k8s-agent-ab2f29b2-2 spec.containers{api-gateway} Warning
Failed Failed to pull image "api-gateway": rpc error: code = 2 desc = Error: image library/api-gateway:latest not found
2m <invalid> 5 kubelet, k8s-agent-ab2f29b2-2 Warning
FailedSync Error syncing pod, skipping: failed to "StartContainer" for "api-gateway" with ErrImagePull: "rpc error: code = 2 desc = Error: image library/api-gateway:latest not found"
2m <invalid> 17 kubelet, k8s-agent-ab2f29b2-2 spec.containers{api-gateway} Normal BackOff
Back-off pulling image "api-gateway"
2m <invalid> 17 kubelet, k8s-agent-ab2f29b2-2 Warning FailedSync
Error syncing pod, skipping: failed to "StartContainer" for "api-gateway" with ImagePullBackOff: "Back-off pulling image \"api-gateway\""
There is of course a latest version in the repo. I don't know what I'm missing here. It seems Kubernetes is able to log in to the repo...
Ok - I found out to connect Artifactory thanks to Pull image Azure Container Registry - Kubernetes
There are two things to pay attention to:
1) in the secret definition don't forget https:// in the server-attribute:
kubectl create secret docker-registry regsecret --docker-server=https://our-repo.jfrog.io --docker-username=myuser --docker-password=<your-pword> --docker-email=<your-email>
2) in the deployment descriptor use the full image path and specify the secret (or append it to the default ServiceAccount):
apiVersion: v1
kind: Pod
metadata:
name: consort-base-infra-art
spec:
containers:
- name: api-gateway
image: our-repo.jfrog.io/api-gateway
imagePullSecrets:
- name: regsecret
Related
This is my Skaffold config.
apiVersion: skaffold/v2beta19
kind: Config
metadata:
name: my-api
build:
local:
useDockerCLI: true
artifacts:
- image: registry.gitlab.com/lodiee/my-api
docker:
buildArgs:
DATABASE_HOST: '{{.DATABASE_HOST}}'
deploy:
helm:
releases:
- name: my-api
artifactOverrides:
image: registry.gitlab.com/lodiee/my-api
imageStrategy:
helm: {}
remoteChart: bitnami/node
setValueTemplates:
image.tag: '{{.DIGEST_HEX}}'
setValues:
image.repository: registry.gitlab.com/lodiee/my-api
When I run Skaffold dev, everything works fine until I see an ImagePullBackoff error in pod logs. Here is a partial output of the log:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 102s default-scheduler Successfully assigned default/my-api-node-55cf68d989-7vltw to kind-control-plane
Normal BackOff 21s (x5 over 99s) kubelet Back-off pulling image "docker.io/registry.gitlab.com/lodiee/my-api:60ef7d4a463e269de9e176afbdd2362fd50870e494ff7a8abf25ece16c0d100c"
Warning Failed 21s (x5 over 99s) kubelet Error: ImagePullBackOff
Normal Pulling 6s (x4 over 101s) kubelet Pulling image "docker.io/registry.gitlab.com/lodiee/my-api:60ef7d4a463e269de9e176afbdd2362fd50870e494ff7a8abf25ece16c0d100c"
Warning Failed 4s (x4 over 100s) kubelet Failed to pull image "docker.io/registry.gitlab.com/lodiee/my-api:60ef7d4a463e269de9e176afbdd2362fd50870e494ff7a8abf25ece16c0d100c": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/registry.gitlab.com/lodiee/my-api:60ef7d4a463e269de9e176afbdd2362fd50870e494ff7a8abf25ece16c0d100c": failed to resolve reference "docker.io/registry.gitlab.com/lodiee/my-api:60ef7d4a463e269de9e176afbdd2362fd50870e494ff7a8abf25ece16c0d100c": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
As you can see from the output, kubernetes adds docker.io prefix in front of my image repo address and so it can't find the image at that address.
Is this the normal behavior? Also this is a local development setup and I use skaffold dev to launch it, why does k8s try to go to that address?
Thank you.
Apply the following YAML file into a Kubernetes cluster:
apiVersion: v1
kind: Pod
metadata:
name: freebox
spec:
containers:
- name: busybox
image: busybox:latest
imagePullPolicy: IfNotPresent
Could the status be "Running" if I run kubectl get pod freebox? Why?
If formatting errors are ignored , no pod wont be in running status :
controlplane $ kubectl get pods freebox
NAME READY STATUS RESTARTS AGE
freebox 0/1 CrashLoopBackOff 3 81s
Becuase if you look at Dockerfile of busy box , The CMD argument "sh" which will complete immediately so pod gets restarted ( becuase default restart policy is always')
https://hub.docker.com/layers/busybox/library/busybox/latest/images/sha256-bc02457f8f5a4a3cd931028ec76c7468cfa8b44d7d89c4a91df1fd82285da681?context=explore
ADD file ... in /708.51 KB
CMD ["sh"]
see the describe of the pod as following :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8s default-scheduler Successfully assigned default/freebox to node01
Normal Pulled 7s (x2 over 8s) kubelet, node01 Container image "busybox:latest" already present on machine
Normal Created 6s (x2 over 7s) kubelet, node01 Created container busybox
Normal Started 6s (x2 over 7s) kubelet, node01 Started container busybox
Warning BackOff 5s (x2 over 6s) kubelet, node01 Back-off restarting failed container
the busybox image need to run a command for running.
add the command in the .spec.containers section under the busybox container
apiVersion: v1
kind: Pod
metadata:
name: freebox
spec:
containers:
- name: busybox
command:
- sleep
- 4800
image: busybox:latest
imagePullPolicy: IfNotPresent
On our K8S Worker node with below command have created "secret" to pull images from our private (Nexus) registry.
kubectl create secret docker-registry regcred --docker-server=https://nexus-server/nexus/ --docker-username=admin --docker-password=password --docker-email=user#company.com
Created my-private-reg-pod.yaml in K8S Worker node, It has below.
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
containers:
- name: private-reg-container
image: nexus-server:4546/ubuntu-16:version-1
imagePullSecrets:
- name: regcred
Created pod with below command
kubectl create -f my-private-reg-pod.yaml
kubectl get pods
NAME READY STATUS RESTARTS AGE
test-pod 0/1 ImagePullBackOff 0 27m
kubectl describe pod test-pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/test-pod to k8s-worker01
Warning Failed 26m (x6 over 28m) kubelet, k8s-worker01 Error: ImagePullBackOff
Normal Pulling 26m (x4 over 28m) kubelet, k8s-worker01 Pulling image "sonatype:4546/ubuntu-16:version-1"
Warning Failed 26m (x4 over 28m) kubelet, k8s-worker01 Failed to pull image "nexus-server:4546/ubuntu-16:version-1": rpc error: code = Unknown desc = Error response from daemon: Get https://nexus-server.domain.com/nexus/v2/ubuntu-16/manifests/ver-1: no basic auth credentials
Warning Failed 26m (x4 over 28m) kubelet, k8s-worker01 Error: ErrImagePull
Normal BackOff 3m9s (x111 over 28m) kubelet, k8s-worker01 Back-off pulling image "nexus-server:4546/ubuntu-16:version-1"
On terminal nexus login works
docker login nexus-server:4546
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
Something i am missing with this section?
Since my docker login to nexus succeeded on terminal, So i have deleted my secret and created with kubectl create secret generic regcred \ --from-file=.dockerconfigjson=<path/to/.docker/config.json> \ --type=kubernetes.io/dockerconfigjson it worked.
I'm new to k8s, so some of my terminology might be off. But basically, I'm trying to deploy a simple web api: one load balancer in front of n pods (where right now, n=1).
However, when I try to visit the load balancer's IP address it doesn't show my web application. When I run kubectl get deployments, I get this:
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
tl-api 1 1 1 0 4m
Here's my YAML file. Let me know if anything looks off--I'm very new to this!
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: tl-api
spec:
replicas: 1
template:
metadata:
labels:
app: tl-api
spec:
containers:
- name: tl-api
image: tlk8s.azurecr.io/devicecloudwebapi:v1
ports:
- containerPort: 80
imagePullSecrets:
- name: acr-auth
nodeSelector:
beta.kubernetes.io/os: windows
---
apiVersion: v1
kind: Service
metadata:
name: tl-api
spec:
type: LoadBalancer
ports:
- port: 80
selector:
app: tl-api
Edit 2: When I try using ACS (which supports Windows), I get this:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned tl-api-3466491809-vd5kg to dc9ebacs9000
Normal SuccessfulMountVolume 11m kubelet, dc9ebacs9000 MountVolume.SetUp succeeded for volume "default-token-v3wz9"
Normal Pulling 4m (x6 over 10m) kubelet, dc9ebacs9000 pulling image "tlk8s.azurecr.io/devicecloudwebapi:v1"
Warning FailedSync 1s (x50 over 10m) kubelet, dc9ebacs9000 Error syncing pod
Normal BackOff 1s (x44 over 10m) kubelet, dc9ebacs9000 Back-off pulling image "tlk8s.azurecr.io/devicecloudwebapi:v1"
I then try examining the failed pod:
PS C:\users\<me>\source\repos\DeviceCloud\DeviceCloud\1- Presentation\DeviceCloud.Web.API> kubectl logs tl-api-3466491809-vd5kg
Error from server (BadRequest): container "tl-api" in pod "tl-api-3466491809-vd5kg" is waiting to start: trying and failing to pull image
When I run docker images I see the following:
REPOSITORY TAG IMAGE ID CREATED SIZE
devicecloudwebapi latest ee3d9c3e231d 24 hours ago 7.85GB
tlk8s.azurecr.io/devicecloudwebapi v1 ee3d9c3e231d 24 hours ago 7.85GB
devicecloudwebapi dev bb33ab221910 25 hours ago 7.76GB
Your problem is that the container image tlk8s.azurecr.io/devicecloudwebapi:v1 is in a private container registry. See the events at the bottom of the following command:
$ kubectl describe po -l=app=tl-api
The official Kubernetes docs describe how to resolve this issue, see Pull an Image from a Private Registry, essentially:
Create a secret kubectl create secret docker-registry
Use it in your deployment, under the spec.imagePullSecrets key
I'm using minikube on macOS 10.12 and trying to use a private image hosted at docker hub. I know that minikube launches a VM that as far as I know will be the unique node of my local kubernetes cluster and that will host all my pods.
I read that I could use the VM's docker runtime by running eval $(minikube docker-env). So I used those variables to change from my local docker runtime to the other. Running docker images I could see that the change was done effectively.
My next step was to log in at docker hub using docker login and then pulling my image manually, which ended without error. After that I thought that the image will by ready to by be used by any pod in the cluster but I'm always getting ImagePullBackOff. I also tried to ssh into the VM via minikube ssh and the result is the same, the image is there to be used but for some reason I don't know it's refusing to use it.
In case it helps, this is my deployment description file:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: web-deployment
spec:
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: godraude/nginx
imagePullPolicy: Always
ports:
- containerPort: 80
- containerPort: 443
And this is the output of kubectl describe pod <podname>:
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1m 1m 1 {default-scheduler } Normal Scheduled Successfully assigned web-deployment-2451628605-vtbl8 to minikube
1m 23s 4 {kubelet minikube} spec.containers{nginx} Normal Pulling pulling image "godraude/nginx"
1m 20s 4 {kubelet minikube} spec.containers{nginx} Warning Failed Failed to pull image "godraude/nginx": Error: image godraude/nginx not found
1m 20s 4 {kubelet minikube} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "nginx" with ErrImagePull: "Error: image godraude/nginx not found"
1m 4s 5 {kubelet minikube} spec.containers{nginx} Normal BackOff Back-off pulling image "godraude/nginx"
1m 4s 5 {kubelet minikube} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "nginx" with ImagePullBackOff: "Back-off pulling image \"godraude/nginx\""
i think what u need is to create a secrete which will tell kube from where it can pull your private image and its credentials
kubectl create secret docker-registry my-secret --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL
below command to list your secretes
kubectl get secret
NAME TYPE DATA AGE
my-secret kubernetes.io/dockercfg 1 100d
now in deployment defination u need to define whcih secret to use
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: web-deployment
spec:
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: godraude/nginx
imagePullPolicy: Always
ports:
- containerPort: 80
- containerPort: 443
imagePullSecrets:
- name: my-secret
The problem was the image pull policy. It was set to Always so docker was trying to pull the imagen even if it was present. Setting imagePullPolicy: Never solved the issue.