I'm getting this weird error in azure pipelines. We use a ACR to supply our base images and we connect to this using a service connection which works well most of the time. However occasionally we get an error:
/usr/bin/docker pull /base-images/python:3.7-buster-vanilla
Error response from daemon: Get https:///v2/base-images/python/manifests/3.7-buster-vanilla: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.
There does not seem to be any clear explanation for this because when we perform a retry of the specific task this works fine.
We have no separate login to the acr and we only define the connection that needs to be used in the job itself:
container:
image: servers.azurecr.io/base-images/python:3.7-buster-vanilla
endpoint: server-Service-Connection
I have been looking at the build logs but was not able to get any usefull information from there. If any additional information is needed i would of course be happy to supply it.
I fixed this error using the Azure CLI az acr login --name *registry-name* Solution explained here at Azure Registry troubleshooting docs
I think you may be running into a Docker Hub issue. We're experiencing this intermittently in our CI (not Azure) during regular pulls for python images.
From https://status.docker.com/pages/history/533c6539221ae15e3f000031
February 3, 2021 5:21PM
UTC[Investigating] A number of users may experience problems logging into Docker Hub.
Related
I'm trying to push to GAR from my local machine, but I always get this error:
failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
First, I've confirmed that my account has the Artifact Registry Writer role through IAM.
I have done the following locally:
# Login with my Google account
gcloud auth login --update-adc --force
# Configure docker to use the gcloud CLI auth helper
gcloud auth configure-docker us-west1-docker.pkg.dev
# docker login for good measure
docker login
# Tag my image (already built)
docker tag myimage us-west1-docker.pkg.dev/myproject/myrepo/myimage
# Push it
docker push us-west1-docker.pkg.dev/myproject/myrepo/myimage
On this final command I get the error above.
I have read all the Google documentation I could find but they all suggest the above steps:
https://cloud.google.com/artifact-registry/docs/docker/pushing-and-pulling
https://cloud.google.com/artifact-registry/docs/docker/troubleshoot
Note: I can't pull either, using the command provided directly from the GCP web UI.
I'm on M1 Mac.
So I was able to solve this problem by completely nuking Docker, specifically with these steps: https://stackoverflow.com/a/69437543/3846032. I couldn't uninstall it by normal means, it would just hang, implying that the problems I was getting were a result of my Docker installation being very broken. Indeed, I managed to follow the above steps on another machine and it worked, which led me to conclude the steps above and my credentials were totally fine.
The 403 was a red herring, it must have come from my local Docker being broken in such a way that it doesn't send properly authenticated requests.
I am using openstack container to enable integration testing against swift
The container used is : https://hub.docker.com/r/jeantil/openstack-swift-keystone-docker/
And the steps followed are : https://github.com/jeantil/openstack-swift-keystone-docker
The configuration is working fine on local and open internet(concourse pipeline job)
But when I am using the same in concourse pipeline job on INTRANET, I am getting the below error:
Failed to discover available identity versions when contacting http://127.0.0.1:35357/v3. Attempting to parse version from URL.
Unauthorized (HTTP 401)
I am getting this error while creating a new service or even loading user lists:
Example:
openstack endpoint create --region RegionOne object-store internal http://127.0.0.1:8080/v1/KEY_%\(tenant_id\)s
openstack endpoint create --region RegionOne object-store admin http://127.0.0.1:8080/v1
openstack user list
Is it due to some proxy related configuration, because everything is working fine if I am running this concourse job on internet
I tried multiple approaches and at the end I was able to solve the issue.
Include ENV NO_PROXY=localhost in the dockerfile so that the proxy configurations are removed for this config
I created tekton pipeline on minikube as per this link (Basically I'm pulling the repo from github and generating image and pushing it to ECR)
But in my case, I'm pushing the image to AWS ECR.
I configured credentials of AWS ECR on my cluster as per this
When I'm running the pipeline I'm getting the following error.
Note: For testing if my AWS credentials were configured correctly or not, I created a simple deployment spec file and ran it. The image is pulled and the application is running. But with tekton I'm getting 401 issue. Can someone help me with this issue, please?
INFO[0000] GET KEYCHAIN
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "12345678910.dkr.ecr.us-east-1.amazonaws.com/test-api:latest": POST https://12345678910.dkr.ecr.us-east-1.amazonaws.com/v2/test-api/blobs/uploads/: unexpected status code 401 Unauthorized: Not Authorized
I'm trying to deploy the docker container on Azure App Service from Azure DevOps services. I've pushed the docker image to Azure Container Registry. When I try to create the release definition, I could not able to find the service connection for Azure Container Registry. I have created the service connection for ACR but it's not showing up in the list in Azure DevOps portal.
When I selected 'Azure Container Repository' as the source type, the service connection is not visible in the drop down box. I'm using DockerHub as another option. It's displaying the service connection in the list.
The steps I followed to create the service connection for ACR:
Selected Docker Registry from the list.
Selected Azure Container Registry as Registry Type. Provided the subscription ID and the registry from ACR.
Provided the service connection name and saved.
UPDATE
I have created service connection for Azure Resource Manager using managed identity authentication by providing both subscription id and tenant id. I'm trying to use this connection in Artifact settings. I got the below error.
Variable with name endpoint.serviceprincipalid could not be found for the given service connection.
It's failing to pull the docker image from ACR. The logs from App service shows the pull access denied for the repository.
Service Connection problem solved but facing docker permission issue from App service
2020-02-10 12:31:11.781 INFO - Pulling image from Docker hub:
kbdockerregis/kbdockerimage:15
2020-02-10 12:31:14.406 ERROR - DockerApiException: Docker API responded with
status code=NotFound, response={"message":"pull access denied for
kbdockerregis/kbdockerimage, repository does not exist or may require 'docker
login': denied: requested access to the resource is denied"}
2020-02-10 12:31:14.408 ERROR - Image pull failed: Verify docker image
configuration and credentials (if using private repository)
2020-02-10 12:31:14.412 INFO - Stoping site kbapp1 because it failed during
startup.
When I selected 'Azure Container Repository' as the source type, the
service connection is not visible in the drop down box.
For this first issue, this because the api our system used is shown as below while you choosing ACR as release source:
https://dev.azure.com/{org}/{project}/_apis/serviceendpoint/endpoints?type=azurerm
You can see the parameters this api attached is type=azurerm. It only fetched the service connection which type is Azure Resource Manager. But Container Registry does not belong to this.
So, you'd better to create and use a service connection which type is Azure Resource Manager type.
Variable with name endpoint.serviceprincipalid could not be found for
the given service connection.
For this second issue, haven't get too much info from you (like checking stake trace). So based on my known, I'd suggest you changed the type from Managed Identity Authentication to Service Principal Authentication. Then follow this doc to config it.
This is more secure and can authorized firstly.
Service Principal Client id, it is the application id after you create the app in Azure app registrations:
Service principal key:
Stack overflow is a open forum and not secure to share some key info(especially Fiddler trace) which I need and used to investigate from backend. You'd better go here because you could choose Microsoft Only there. If possible, I can go that community and let that community's engineer show it to me. So that I could continue dig into it.
Have you encountered the error mentioned in the subject line before?, I'm intrigued by it and unable to a solution. Here are the steps I executed to create a service principal
az login
-- Gives me a device code to enter in browser
Login to az non-interactively
az account set --subscription="xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
Check correct account by
az account show
Create a SP
az ad sp create-for-rbac -n "tf-authenticator" --role contributor
Run terraform plan as like below:
[jim#rhel75 msdn]$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
azurerm_resource_group.tf-rg-cluster-sql: Refreshing state... os_profile_windows_config.1060902566.enable_automatic_upgrades: "true"
Plan: 28 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
Now, I run terraform again and I get error that No authorization token found!
[jim#rhel75 msdn]$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
Error: Error refreshing state: 1 error(s) occurred:
* provider.azurerm: Error building AzureRM Client: Error populating Client ID from the Azure CLI: No Authorization Tokens were found - please re-authenticate using `az login`.
Note, I tested and same behavior exhibits on
1. CentOS 7.6
2. Ubuntu 16.04
3. Azure Pipelines Release Pipeline (Which utilizes Ubuntu 16.04 agent)
Update: I encounter a weird error when I try to create a SP. I have already my default subscription but still run the below command and receive error:
az ad sp create-for-rbac -n "tf-authenticator" --role="Contributor" --scopes="/subscriptions/xxxxxxxxxxxxxxxxxxx"
Error message:
Changing "tf-authenticator" to a valid URI of "http://tf-authenticator", which is the required format used for service principal names
Get Token request returned http error: 400 and server response: {"error":"unauthorized_client","error_description":"AADSTS700016: Application with identifier 'bd58677d-d0a4-46a5-9b5a-02c6662ae149' was not found in the directory '055b56f0-423b-4b1a-b6bc-6ae09ab61485'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.\r\nTrace ID: cc58c97b-2247-4bf7-ab6c-7d1b993f0b00\r\nCorrelation ID: 42540a48-7d44-494b-9dc2-a8dd9d6c10b7\r\nTimestamp: 2019-03-14 00:46:49Z","error_codes":[700016],"timestamp":"2019-03-14 00:46:49Z","trace_id":"cc58c97b-2247-4bf7-ab6c-7d1b993f0b00","correlation_id":"42540a48-7d44-494b-9dc2-a8dd9d6c10b7","error_uri":"https://login.microsoftonline.com/error?code=700016"}
I have further drilled down that this has to do something with azure tokens, as I can create a VM instances with Azure CLI itself, please see evidence below:
az vm create --resource-group "myResourceGroup" --name "myVM" --image "Win2016Datacenter" --admin-username "Demouser" --admin-password "Demouser#123" --use-unmanaged-disk --location "Australia Southeast"
Azure Error: InvalidTemplateDeployment
Message: The template deployment failed with error: 'The resource with id: '/subscriptions/11855b5b-484f-4104-b77b-a73eb8cd74dc/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachines/myVM' failed validation with message: 'The requested size for resource '/subscriptions/11855b5b-484f-4104-b77b-a73eb8cd74dc/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachines/myVM' is currently not available in location 'australiasoutheast' zones '' for subscription '11855b5b-484f-4104-b77b-a73eb8cd74dc'. Please try another size or deploy to a different location or zones. See https://aka.ms/azureskunotavailable for details.'.'.
It did not launched a VM but at least its clear that it was able to proceed should size would have been right.
Can you please try to follow this documentation - https://learn.microsoft.com/en-us/azure/developer/terraform/get-started-cloud-shell-bash?tabs=bash
You need to create environment variables in order to deploy azure portal
One more thing - you need to have environment variables in .tf file
I would suggest you to not store the secrets in plain text
If you are using azure devops pipeline to deploy you can store it in the keyvault or in variables
If you are using terraform cloud with workspaces you can store it in variables