kubectl pod fails to pull down an AWS ECR image - kubernetes

step 1 sudo $(aws ecr get-login --no-include-email --region xx-xxxx-x)
step 2 curl -LSs https://github.com/fermayo/ecr-k8s-secret/raw/master/gen-secret.sh | bash -
step 3 kubectl describe secret aws-ecr-credentials
Name: aws-ecr-credentials
Namespace: default
Labels: <none>
Annotations: <none>
Type: kubernetes.io/dockerconfigjson
Data
.dockerconfigjson: 32 bytes
step 4 kubectl describe pod x
Warning Failed 5s kubelet, ip-10-46-250-151 Failed to pull image "my-account.dkr.ecr.us-east-1.amazonaws.com/my-image:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://my-account.dkr.ecr.us-east-1.amazonaws.com/my-image/latest: no basic auth credentials
Why can't the pod pull down the image?

Created a script that pulls the token from AWS-ECR
ACCOUNT=xxxxxxxxxxxx
REGION=xx-xxxx-x
SECRET_NAME=${REGION}-ecr-registry
EMAIL=email#email.com
#
#
TOKEN=`aws ecr --region=$REGION get-authorization-token --output text \
--query authorizationData[].authorizationToken | base64 -d | cut -d: -f2`
#
# Create or replace registry secret
#
kubectl delete secret --ignore-not-found $SECRET_NAME
kubectl create secret docker-registry $SECRET_NAME \
--docker-server=https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com \
--docker-username=AWS \
--docker-password="${TOKEN}" \
--docker-email="${EMAIL}"
and created a Linux cronjob to run this every 10 hours

Your Deployment manifest will need to specify that the container registry credentials are in a secret. This is as simple as adding imagePullSecrets:
apiVersion: v1
kind: Deployment
metadata:
name: deployment-name
spec:
containers:
- image: your-registry/image/name:tag
imagePullSecrets:
- name: secret-name

I too was banging my head on this and realized it was a region mismatch. I was getting my token from us-east-2 when the image is located in us-west-2.
Snippet from https://docs.aws.amazon.com/AmazonECR/latest/userguide/common-errors-docker.html#error-403
There are times when you may receive an HTTP 403 (Forbidden) error, or the error message no basic auth credentials from the docker push command, even if you have successfully authenticated to Docker using the aws ecr get-login command. The following are some known causes of this issue:
You have authenticated to a different region Authentication requests are tied to specific regions, and cannot be used across regions. For example, if you obtain an authorization token from US West (Oregon), you cannot use it to authenticate against your repositories in US East (N. Virginia). To resolve the issue, ensure that you are using the same region for both authentication and docker push command calls.

Related

Bitnami Redis on Kubernetes Authentication Failure with Existing Secret

I'm trying to install Redis on Kubernetes environment with Bitnami Redis HELM Chart. I want to use a defined password rather than randomly generated one. But i'm getting error below when i want to connect to redis master or replicas with redis-cli.
I have no name!#redis-client:/$ redis-cli -h redis-master -a $REDIS_PASSWORD
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Warning: AUTH failed
I created a Kubernetes secret like this.
---
apiVersion: v1
kind: Secret
metadata:
name: redis-secret
namespace: redis
type: Opaque
data:
redis-password: YWRtaW4xMjM0Cg==
And in values.yaml file i updated auth spec like below.
auth:
enabled: true
sentinel: false
existingSecret: "redis-secret"
existingSecretPasswordKey: "redis-password"
usePasswordFiles: false
If i don't define existingSecret field and use randomly generated password then i can connect without an issue. I also tried AUTH admin1234 after Warning: AUTH failed error but it didn't work either.
You can achieve it in much simpler way i.e. by running:
$ helm install my-release \
--set auth.password="admin1234" \
bitnami/redis
This will update your "my-release-redis" secret, so when you run:
$ kubectl get secrets my-release-redis -o yaml
you'll see it contains your password, already base64-encoded:
apiVersion: v1
data:
redis-password: YWRtaW4xMjM0Cg==
kind: Secret
...
In order to get your password, you need to run:
export REDIS_PASSWORD=$(kubectl get secret --namespace default my-release-redis -o jsonpath="{.data.redis-password}" | base64 --decode)
This will set and export REDIS_PASSWORD environment variable containing your redis password.
And then you may run your redis-client pod:
kubectl run --namespace default redis-client --restart='Never' --env REDIS_PASSWORD=$REDIS_PASSWORD --image docker.io/bitnami/redis:6.2.4-debian-10-r13 --command -- sleep infinity
which will set REDIS_PASSWORD environment variable within your redis-client pod by assigning to it the value of REDIS_PASSWORD set locally in the previous step.
The issue was about how i encoded password with echo command. There was a newline character at the end of my password. I tried with printf command rather than echo and it created a different result.
printf admin1234 | base64

How to pull ACR image from k3s pods

I have customized the coredns image and pushed it to my azure container registry (ACR). Now in default coredns pod that is coming after k3s installation, I want to use my_azure_acr_repo/proj/customize-coredns:latest image instead of rancher/coredns-coredns:1.8.3. So I edited the coredns deployment kubectl edit deploy coredns -n kube-system and replaced my acr image with rancher one. But now coredns pod is not able to pull my acr image and giving error in pod description:
Failed to pull image "my_azure_acr_repo/proj/customize-coredns:latest": rpc error:
code = Unknown desc = failed to pull and unpack image "my_azure_acr_repo/proj/customize-coredns:latest":
failed to resolve reference "my_azure_acr_repo/proj/customize-coredns:latest": failed to
authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized
How can I authenticate acr image, so that pod should pull it ?
That's because your container is not authorized to pull image from your private ACR.
First you've to create secret so that you can access your ACR, then pass that secret in your deployment using imagePullSecrets
you can create secret by this command, make sure to replace your credential variables
kubectl create secret docker-registry <name> --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL
For ACR it will be something like this
kubectl create secret docker-registry regkey --docker-server=https://myregistry.azurecr.io --docker-username=ACR_USERNAME --docker-password=ACR_PASSWORD --docker-email=ANY_EMAIL_ADDRESS
your deployment spec
spec:
containers:
- name: foo
image: janedoe/awesomeapp:v1
imagePullSecrets:
- name: regkey
More info related to this.
https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod

Kubernetes containerd - failed to pull image from private registry

I setup kubernetes V1.20.1 with containerd instead of Docker. Now I failed to pull Docker images from my private registry (Harbor).
I already changed the /etc/containerd/config.toml like this:
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.foo.com"]
endpoint = ["https://registry.foo.com"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.foo.com"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.foo.com".auth]
username = "admin"
password = "Harbor12345"
But this did not work. The pull failed with the message:
Failed to pull image "registry.foo.com/library/myimage:latest": rpc error: code = Unknown
desc = failed to pull and unpack image "registry.foo.com/library/myimage:latest": failed to
resolve reference "registry.foo.com/library/myimage:latest": unexpected status code
[manifests latest]: 401 Unauthorized
My Harbor registry is available via HTTPS with a Let's Encrypt certificate. So https should not be the problem here.
Even if I try to create a docker-secret this did not work:
kubectl create secret docker-registry registry.foo.com --docker-server=https://registry.foo.com --docker-username=admin --docker-password=Harbor12345 --docker-email=info#foo.com
Can anybody give me an example how to configure a private registry in Kubernetes with containerd?
Set imagePullSecrets in the pod/deployment specification:
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:
- name: private-reg-container
image: <your-private-image>
imagePullSecrets:
- name: registry.foo.com
More info: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
Create file, put username:password in it and get the base64 code of it:
touch pass.txt
nano pass.txt
# write like that => username:password
base64 pass.txt
# get the base64 code: cmxxxxxxxxyyyyyyCg==
nano /etc/containerd/config.toml (use auth="", instead of using username/password):
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.foo.com"]
endpoint = ["https://registry.foo.com"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.foo.com".tls]
insecure_skip_verify=true
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry.foo.com".auth]
auth ="cmxxxxxxxxyyyyyyCg=="
Restart containerd service:
sudo systemctl restart containerd.service
Pull image directly from node:
sudo crictl pull registry.foo.com/imageName:Tag

Pod cannot pull image from private docker registry

I am having some real trouble getting my pods to pull images from a private docker registry that I have setup and am able to authenticate to (I can do docker login https://my.website.com/ and I get Login Succeeded without having to put in my username:password) (I am able to run docker pull my.website.com:5000/human/forum and see all the layers being downloaded.) .
I use https://github.com/bazelbuild/rules_k8s#aliasing-eg-k8s_deploy where I specify the namespace to be "default".
I made sure to put "HTTPS://my.website.com:5000/V2/" (in lowercase) in the auth section in the docker config file before I generated the regcred secret.
Notice that I specify the imagePullSecrets below:
# deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: angular-bazel-example-prod
spec:
replicas: 1
template:
metadata:
labels:
app: angular-bazel-example-prod
spec:
containers:
- name: angular-bazel-example
image: human/forum:dev
imagePullPolicy: Always
ports:
- containerPort: 8080
imagePullSecrets:
- name: regcred # Notice
I made sure to update my certificate authority certificates:
cp /etc/docker/certs.d/my.website.com\:5000/ca.crt /usr/local/share/ca-certificates/my.website.registry.com/
sudo update-ca-certificates
I see sudo curl --user testuser:testpassword --cacert /usr/local/share/ca-certificates/my.website.registry.com/ca.crt -X GET https://mywebsite.com:5000/v2/_catalog
> {"repositories":["human/forum"]}
I see sudo curl --user testuser:testpassword --cacert /usr/local/share/ca-certificates/mywebsite.registry.com/ca.crt -X GET https://mywebsite.com:5000/v2/human/forum/tags/list
> {"name":"a/repository","tags":["dev"]}
There must be a way to troubleshoot this but I don't know how.
One thing I am curious about is
kubectl describe pod my-first-pod...
...
Volumes:
default-token-mtz9g:
Type: Secret
Where can I find this volume? I can't kubectl exec into a container because none is running.. because the pod can't pull the image.
Do you have any ideas on how I could troubleshoot this?
Thank you!
Slackware
Create a kubernetes secret to access the custom repository. One way is to provide the server, user and password manually. Documentation link.
kubectl create secret docker-registry $YOUR_REGISTRY_NAME --docker-server=https://$YOUR_SERVER_DNS/v2/ --docker-username=$YOUR_USER --docker-password=$YOUR_PASSWORD --docker-email=whatever#gmail.com --namespace=default
Then use it in your yaml
...
imagePullSecrets:
- name: $YOUR_REGISTRY_NAME
Regarding the default-token that you see mounted, it belongs to the service account that your pod uses to talk to the kubernetes api. You can find the service account by running kubectl get sa and kubectl describe sa $DEFAULT_SERVICE_ACCOUNT_ID. Find the token by running kubectl get secrets and kubectl describe secret $SECRET_ID. To clarify this service account and token have nothing to do with the docker registry unless you specify it. To include the registry in the service account follow this guide link

Pulling Images from GCR into GKE

Today is my first day playing with GCR and GKE. So apologies if my question sounds childish.
So I have created a new registry in GCR. It is private. Using this documentation, I got hold of my Access Token using the command
gcloud auth print-access-token
#<MY-ACCESS_TOKEN>
I know that my username is oauth2accesstoken
On my local laptop when I try
docker login https://eu.gcr.io/v2
Username: oauth2accesstoken
Password: <MY-ACCESS_TOKEN>
I get:
Login Successful
So now its time to create a docker-registry secret in Kubernetes.
I ran the below command:
kubectl create secret docker-registry eu-gcr-io-registry --docker-server='https://eu.gcr.io/v2' --docker-username='oauth2accesstoken' --docker-password='<MY-ACCESS_TOKEN>' --docker-email='<MY_EMAIL>'
And then my Pod definition looks like:
apiVersion: v1
kind: Pod
metadata:
name: my-app
spec:
containers:
- name: my-app
image: eu.gcr.io/<my-gcp-project>/<repo>/<my-app>:latest
ports:
- containerPort: 8090
imagePullSecrets:
- name: eu-gcr-io-registry
But when I spin up the pod, I get the ERROR:
Warning Failed 4m (x4 over 6m) kubelet, node-3 Failed to pull image "eu.gcr.io/<my-gcp-project>/<repo>/<my-app>:latest": rpc error: code = Unknown desc = Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
I verified my secrets checking the YAML file and doing a base64 --decode on the .dockerconfigjson and it is correct.
So what have I missed here ?
If your GKE cluster & GCR registry are in the same project: You don't need to configure authentication. GKE clusters are authorized to pull from private GCR registries in the same project with no config. (Very likely you're this!)
If your GKE cluster & GCR registry are in different GCP projects: Follow these instructions to give "service account" of your GKE cluster access to read private images in your GCR cluster: https://cloud.google.com/container-registry/docs/access-control#granting_users_and_other_projects_access_to_a_registry
In a nutshell, this can be done by:
gsutil iam ch serviceAccount:[PROJECT_NUMBER]-compute#developer.gserviceaccount.com:objectViewer gs://[BUCKET_NAME]
where [BUCKET_NAME] is the GCS bucket storing your GCR images (like artifacts.[PROJECT-ID].appspot.com) and [PROJECT_NUMBER] is the numeric GCP project ID hosting your GKE cluster.