We have created a nexus3 docker host private registry on CentOS machine and same ip details updated on daemon.json under docker folder.
Docker pull and push is working fine.
Same image while trying to kubernetes deploy is failing with image pull state.
$ Kubectl run deployname --image=nexus3provaterepo:port/image
Before we create secret entries via command $ Kubectl create secret with same inform of user ID and password, like docker login -u userid -p passwd
Here my problem is image pull is failing from nexus3 docker host.
Please suggest me how to verify login via kubernetes command and resolve this pull image issue.
Looking yours suggestions, Thanks in advance
So when pulling from private repos you need to specify an imagePullSecret like such:
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: <your-private-image>
# Specify the secret with your users credentials
imagePullSecrets:
- name: regcred
You would then use the kubectl apply -f functionality, I am not actually sure you can use this in the imperative cli version of running a deployment but all the doucmentation on this can be found at here
Related
I setup kubernetes V1.20.1 with containerd instead of Docker. Now I failed to pull Docker images from my private registry (Harbor).
I already changed the /etc/containerd/config.toml like this:
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.foo.com"]
endpoint = ["https://registry.foo.com"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.foo.com"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.foo.com".auth]
username = "admin"
password = "Harbor12345"
But this did not work. The pull failed with the message:
Failed to pull image "registry.foo.com/library/myimage:latest": rpc error: code = Unknown
desc = failed to pull and unpack image "registry.foo.com/library/myimage:latest": failed to
resolve reference "registry.foo.com/library/myimage:latest": unexpected status code
[manifests latest]: 401 Unauthorized
My Harbor registry is available via HTTPS with a Let's Encrypt certificate. So https should not be the problem here.
Even if I try to create a docker-secret this did not work:
kubectl create secret docker-registry registry.foo.com --docker-server=https://registry.foo.com --docker-username=admin --docker-password=Harbor12345 --docker-email=info#foo.com
Can anybody give me an example how to configure a private registry in Kubernetes with containerd?
Set imagePullSecrets in the pod/deployment specification:
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: <your-private-image>
imagePullSecrets:
- name: registry.foo.com
More info: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
Create file, put username:password in it and get the base64 code of it:
touch pass.txt
nano pass.txt
# write like that => username:password
base64 pass.txt
# get the base64 code: cmxxxxxxxxyyyyyyCg==
nano /etc/containerd/config.toml (use auth="", instead of using username/password):
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.foo.com"]
endpoint = ["https://registry.foo.com"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.foo.com".tls]
insecure_skip_verify=true
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.foo.com".auth]
auth ="cmxxxxxxxxyyyyyyCg=="
Restart containerd service:
sudo systemctl restart containerd.service
Pull image directly from node:
sudo crictl pull registry.foo.com/imageName:Tag
I've deployed an docker registry inside my kubernetes:
$ kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
registry-docker-registry ClusterIP 10.43.39.81 <none> 443/TCP 162m
I'm able to pull images from my machine (service is exposed via an ingress rule):
$ docker pull registry-docker-registry.registry/skaffold-covid-backend:c5dfd81-dirty#sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd
...
Status: Downloaded newer image for registry-do...
When I'm trying to test it in order to deploy my image into the same kubernetes:
apiVersion: apps/v1
kind: Deployment
metadata:
name: covid-backend
namespace: skaffold
spec:
replicas: 3
selector:
matchLabels:
app: covid-backend
template:
metadata:
labels:
app: covid-backend
spec:
containers:
- image: registry-docker-registry.registry/skaffold-covid-backend:c5dfd81-dirty#sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd
name: covid-backend
ports:
- containerPort: 8080
Then, I've tried to deploy it:
$ cat pod.yaml | kubectl apply -f -
However, kubernetes isn't able to reach registry:
Extract of kubectl get events:
6s Normal Pulling pod/covid-backend-774bd78db5-89vt9 Pulling image "registry-docker-registry.registry/skaffold-covid-backend:c5dfd81-dirty#sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd"
1s Warning Failed pod/covid-backend-774bd78db5-89vt9 Failed to pull image "registry-docker-registry.registry/skaffold-covid-backend:c5dfd81-dirty#sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd": rpc error: code = Unknown desc = failed to pull and unpack image "registry-docker-registry.registry/skaffold-covid-backend#sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd": failed to resolve reference "registry-docker-registry.registry/skaffold-covid-backend#sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd": failed to do request: Head https://registry-docker-registry.registry/v2/skaffold-covid-backend/manifests/sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd: dial tcp: lookup registry-docker-registry.registry: Try again
1s Warning Failed pod/covid-backend-774bd78db5-89vt9 Error: ErrImagePull
As you can see, kubernetes is not able to get access to the internal deployed registry...
Any ideas?
I would recommend to follow docs from k3d, they are here.
More precisely this one
Using your own local registry
If you don't want k3d to manage your registry, you can start it with some docker commands, like:
docker volume create local_registry
docker container run -d --name registry.local -v local_registry:/var/lib/registry --restart always -p 5000:5000 registry:2
These commands will start you registry in registry.local:5000. In order to push to this registry, you will need to add the line at /etc/hosts as we described in the previous section . Once your registry is up and running, we will need to add it to your registries.yaml configuration file. Finally, you must connect the registry network to the k3d cluster network: docker network connect k3d-k3s-default registry.local. And then you can check you local registry.
Pushing to your local registry address
The registry will be located, by default, at registry.local:5000 (customizable with the --registry-name and --registry-port parameters). All the nodes in your k3d cluster can resolve this hostname (thanks to the DNS server provided by the Docker daemon) but, in order to be able to push to this registry, this hostname but also be resolved from your host.
The easiest solution for this is to add an entry in your /etc/hosts file like this:
127.0.0.1 registry.local
Once again, this will only work with k3s >= v0.10.0 (see the section below when using k3s <= v0.9.1)
Local registry volume
The local k3d registry uses a volume for storying the images. This volume will be destroyed when the k3d registry is released. In order to persist this volume and make these images survive the removal of the registry, you can specify a volume with the --registry-volume and use the --keep-registry-volume flag when deleting the cluster. This will create a volume with the given name the first time the registry is used, while successive invocations will just mount this existing volume in the k3d registry container.
Docker Hub cache
The local k3d registry can also be used for caching images from the Docker Hub. You can start the registry as a pull-through cache when the cluster is created with --enable-registry-cache. Used in conjuction with --registry-volume/--keep-registry-volume can speed up all the downloads from the Hub by keeping a persistent cache of images in your local machine.
Testing your registry
You should test that you can
push to your registry from your local development machine.
use images from that registry in Deployments in your k3d cluster.
We will verify these two things for a local registry (located at registry.local:5000) running in your development machine. Things would be basically the same for checking an external registry, but some additional configuration could be necessary in your local machine when using an authenticated or secure registry (please refer to Docker's documentation for this).
Firstly, we can download some image (like nginx) and push it to our local registry with:
docker pull nginx:latest
docker tag nginx:latest registry.local:5000/nginx:latest
docker push registry.local:5000/nginx:latest
Then we can deploy a pod referencing this image to your cluster:
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-test-registry
labels:
app: nginx-test-registry
spec:
replicas: 1
selector:
matchLabels:
app: nginx-test-registry
template:
metadata:
labels:
app: nginx-test-registry
spec:
containers:
- name: nginx-test-registry
image: registry.local:5000/nginx:latest
ports:
- containerPort: 80
EOF
Then you should check that the pod is running with kubectl get pods -l "app=nginx-test-registry".
Additionaly there are 2 github links worth visting
K3d not able resolve dns
You could try to use an answer provided by #rjshrjndrn, might solve your issue with dns.
docker images are not pulled from docker repository behind corporate proxy
Open github issue on k3d with same problem as yours.
I am having some real trouble getting my pods to pull images from a private docker registry that I have setup and am able to authenticate to (I can do docker login https://my.website.com/ and I get Login Succeeded without having to put in my username:password) (I am able to run docker pull my.website.com:5000/human/forum and see all the layers being downloaded.) .
I use https://github.com/bazelbuild/rules_k8s#aliasing-eg-k8s_deploy where I specify the namespace to be "default".
I made sure to put "HTTPS://my.website.com:5000/V2/" (in lowercase) in the auth section in the docker config file before I generated the regcred secret.
Notice that I specify the imagePullSecrets below:
# deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: angular-bazel-example-prod
spec:
replicas: 1
template:
metadata:
labels:
app: angular-bazel-example-prod
spec:
containers:
- name: angular-bazel-example
image: human/forum:dev
imagePullPolicy: Always
ports:
- containerPort: 8080
imagePullSecrets:
- name: regcred # Notice
I made sure to update my certificate authority certificates:
cp /etc/docker/certs.d/my.website.com\:5000/ca.crt /usr/local/share/ca-certificates/my.website.registry.com/
sudo update-ca-certificates
I see sudo curl --user testuser:testpassword --cacert /usr/local/share/ca-certificates/my.website.registry.com/ca.crt -X GET https://mywebsite.com:5000/v2/_catalog
> {"repositories":["human/forum"]}
I see sudo curl --user testuser:testpassword --cacert /usr/local/share/ca-certificates/mywebsite.registry.com/ca.crt -X GET https://mywebsite.com:5000/v2/human/forum/tags/list
> {"name":"a/repository","tags":["dev"]}
There must be a way to troubleshoot this but I don't know how.
One thing I am curious about is
kubectl describe pod my-first-pod...
...
Volumes:
default-token-mtz9g:
Type: Secret
Where can I find this volume? I can't kubectl exec into a container because none is running.. because the pod can't pull the image.
Do you have any ideas on how I could troubleshoot this?
Thank you!
Slackware
Create a kubernetes secret to access the custom repository. One way is to provide the server, user and password manually. Documentation link.
kubectl create secret docker-registry $YOUR_REGISTRY_NAME --docker-server=https://$YOUR_SERVER_DNS/v2/ --docker-username=$YOUR_USER --docker-password=$YOUR_PASSWORD --docker-email=whatever#gmail.com --namespace=default
Then use it in your yaml
...
imagePullSecrets:
- name: $YOUR_REGISTRY_NAME
Regarding the default-token that you see mounted, it belongs to the service account that your pod uses to talk to the kubernetes api. You can find the service account by running kubectl get sa and kubectl describe sa $DEFAULT_SERVICE_ACCOUNT_ID. Find the token by running kubectl get secrets and kubectl describe secret $SECRET_ID. To clarify this service account and token have nothing to do with the docker registry unless you specify it. To include the registry in the service account follow this guide link
I am trying to deploy an application via kubectl using an image stored on Codefresh. I have it running perfectly when i place the image on a public registry.
The problem is when I apply the deployment.yaml I get a "ImagePullBackOff" error on the pods. I'm assuming, i think correctly, that this is because I need a secret to be able to access my Codefresh image.
This is the container part of my current deployment.yaml:
spec:
containers:
- name: dockapp
#States the image that will be put inside the pod. Secret to get access is declared below
#registry.hub.docker.com/jamiedovu/dockapp:latest
image: r.cfcr.io/jamiew87/my-app-image:master
ports:
- containerPort: 8080
name: http
imagePullSecrets:
- name: regcred
My question is, what is it i need to put into the secret "regcred" to be able to connect to this private registry. The Kubernetes documentation only demonstrates how to do one for docker.
I think it's explained in the docs.
export DOCKER_REGISTRY_SERVER=r.cfcr.io
export DOCKER_USER=YOUR_USERNAME
export DOCKER_PASSWORD=YOUR_REGISTRY_PASSWORD
export DOCKER_EMAIL=YOUR_EMAIL
kubectl create secret docker-registry cfcr\
--docker-server=$DOCKER_REGISTRY_SERVER\
--docker-username=$DOCKER_USER\
--docker-password=$DOCKER_PASSWORD\
--docker-email=$DOCKER_EMAIL
For people in the future with problems,
The codefresh repository is an actual docker repository. Not knowing this was giving me the problems.
So in the docker-username etc places you put your codefresh credentials, and instead of the password you put a secret that you generate within codefresh. This gives you access to the r.cfcr.io repository.
Today is my first day playing with GCR and GKE. So apologies if my question sounds childish.
So I have created a new registry in GCR. It is private. Using this documentation, I got hold of my Access Token using the command
gcloud auth print-access-token
#<MY-ACCESS_TOKEN>
I know that my username is oauth2accesstoken
On my local laptop when I try
docker login https://eu.gcr.io/v2
Username: oauth2accesstoken
Password: <MY-ACCESS_TOKEN>
I get:
Login Successful
So now its time to create a docker-registry secret in Kubernetes.
I ran the below command:
kubectl create secret docker-registry eu-gcr-io-registry --docker-server='https://eu.gcr.io/v2' --docker-username='oauth2accesstoken' --docker-password='<MY-ACCESS_TOKEN>' --docker-email='<MY_EMAIL>'
And then my Pod definition looks like:
apiVersion: v1
kind: Pod
metadata:
name: my-app
spec:
containers:
- name: my-app
image: eu.gcr.io/<my-gcp-project>/<repo>/<my-app>:latest
ports:
- containerPort: 8090
imagePullSecrets:
- name: eu-gcr-io-registry
But when I spin up the pod, I get the ERROR:
Warning Failed 4m (x4 over 6m) kubelet, node-3 Failed to pull image "eu.gcr.io/<my-gcp-project>/<repo>/<my-app>:latest": rpc error: code = Unknown desc = Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
I verified my secrets checking the YAML file and doing a base64 --decode on the .dockerconfigjson and it is correct.
So what have I missed here ?
If your GKE cluster & GCR registry are in the same project: You don't need to configure authentication. GKE clusters are authorized to pull from private GCR registries in the same project with no config. (Very likely you're this!)
If your GKE cluster & GCR registry are in different GCP projects: Follow these instructions to give "service account" of your GKE cluster access to read private images in your GCR cluster: https://cloud.google.com/container-registry/docs/access-control#granting_users_and_other_projects_access_to_a_registry
In a nutshell, this can be done by:
gsutil iam ch serviceAccount:[PROJECT_NUMBER]-compute#developer.gserviceaccount.com:objectViewer gs://[BUCKET_NAME]
where [BUCKET_NAME] is the GCS bucket storing your GCR images (like artifacts.[PROJECT-ID].appspot.com) and [PROJECT_NUMBER] is the numeric GCP project ID hosting your GKE cluster.