I created a docker container using Dockerfile.
&& apk add --virtual=build gcc libffi-dev musl-dev openssl-dev make python3-dev \
&& pip3 --no-cache-dir install azure-cli==${AZURE_CLI_VERSION} \
but still the Azure DevOps container job fail with the error
## [error]Azure CLI 2.x is not installed on this machine.
Can you please let me know is there anything i can do with path or do I need to install Azure cli by some other means ??
I tried a sample command from my pipeline and az seems to work fine. This is the yaml that I tested with Azure CLI task version 1:
steps:
- task: AzureCLI#1
displayName: 'Azure CLI '
inputs:
azureSubscription: xxxxx
scriptLocation: inlineScript
inlineScript: 'az acr list'
As shown in the output of the task execution, the Azure CLI is intalled at: C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az.cmd
If you are still blocked, please post the details of the steps and tasks used in your build pipeline and we can troubleshoot further.
Related
We have a devops pipeline which has an azure cli task that triggers and adf pipeline. All was working untill last friday but now the same task started giving error: ModuleNotFound: azure.core.rest.
The azure cli task looks like this:
az login --service-principal --username $(test-service-principle-id) --password $(test-service-principle-password) --tenant $(TENANT_ID)
az config set extension.use_dynamic_install=yes_without_prompt
$runId=az datafactory pipeline create-run --factory-name test-adf --name "pl_load_training_data" --resource-group test-rg --query "runId" --output tsv
I have added azure-core to requirements.txt and running pip list | grep auzre-core also list that it is installed.
Here is the trace:
Our build agent is running Podman 3.4.2 and there is a global alias in place for each terminal session that simply replaces docker with podman, so the command docker --version yields podman version 3.4.2 as a result.
The goal is to use podman for the Docker#2 task in a Azure DevOps pipeline:
steps:
- task: Docker#2
displayName: Build and push an image to container registry
inputs:
command: buildAndPush
repository: aspnet-web-mhi
dockerfile: $(dockerfilePath)
containerRegistry: $(dockerRegistryServiceConnection)
tags: |
$(tag)
Turns out I was a bit naive in my assumptions, that this would work as the ado_agent is having none of it:
##[error]Unhandled: Unable to locate executable file: 'docker'. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
Is there a way to make that replacement work without too much fuss? I'd avoid scripting everything by myself to use podman instead of docker and push it to a registry, if I can avoid it.
Since I needed to make progress on this, I've decided to go the down the bash-route and built, pushed, pulled and run the images manually. This is the gist of it:
steps:
- task: Bash#3
displayName: Build Docker Image for DemoWeb
inputs:
targetType: inline
script: |
podman build -f $(dockerfilePath) -t demoweb:$(tag) .
- task: Bash#3
displayName: Login and Push to ACR
inputs:
targetType: inline
script: |
podman login -u $(acrServicePrincipal) -p $(acrPassword) $(acrName)
podman push demoweb-mhi:$(tag) $(acrName)/demoweb:$(tag)
- task: Bash#3
displayName: Pull image from ACR
inputs:
targetType: inline
script: |
podman pull $(acrName)/demoweb:$(tag) --creds=$(acrServicePrincipal):$(acrPassword)
- task: Bash#3
displayName: Run container
inputs:
targetType: inline
script: |
podman run -p 8080:80 --restart unless-stopped $(acrName)/demoweb:$(tag)
If you decide to go down that route, please make sure to not expose your service principal and password as variables in your yml file, but create them as secrets.
I'll keep this question open - maybe someone with more expertise in handling GNU/Linux finds a more elegant way.
You could install package podman-docker as well. It installs a wrapper in /usr/bin/docker that points to /usr/bin/podman. So tasks that originally use docker binary (or even docker socket) can be run transparently as podman like Docker#2 build and push.
cat /usr/bin/docker
#!/bin/sh
[ -e /etc/containers/nodocker ] || \
echo "Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg." >&2
exec /usr/bin/podman "$#"
In my Azure Devops project, under the tab "Artifacts", I have a package MyPackage.
In my build pipeline, I have this step:
- stage: Build
displayName: "Build"
jobs:
- job:
steps:
- task: Docker#2
inputs:
containerRegistry: 'TEST container registry'
repository: 'mycontainerregistry/backend'
command: 'buildAndPush'
buildContext: '$(System.DefaultWorkingDirectory)'
Dockerfile: '**/Dockerfile'
tags: |
$(Build.BuildId)
latest
The Dockerfile being built is the standard generated one by Visual Studio:
FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
WORKDIR /src
COPY ["MyProject.API.csproj", "MyProject.API/"]
RUN dotnet restore "MyProject.API/MyProject.API.csproj"
COPY . .
WORKDIR "/src/MyProject.API"
RUN dotnet build "MyProject.API.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "MyProject.API.csproj" -c Release -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "MyProject.API.dll"]
Now, the problem is with the dotnet restore command. This step fails because the restore command can't find the MyPackage nuget from the docker build context. How can I make dotnet restore find MyPackage when running through docker build?
If you have private feed you need to add a source using dotnet nuget add source
dotnet nuget sources add -name "SomeName" -source https://pkgs.dev.azure.com/YourFeed/nuget/v3/index.json -username anything -password $TOKEN
And to pass System.AccessToken you need to use ARG
FROM alpine
ARG TOKEN
RUN dotnet nuget sources add -name "SomeName" -source https://pkgs.dev.azure.com/YourFeed/nuget/v3/index.json -username anything -password $TOKEN
and then in YMAL
- task: Docker#2
inputs:
containerRegistry: 'devopsmanual-acr'
command: 'build'
Dockerfile: 'stackoverflow/85-docker/DOCKERFILE'
arguments: '--build-arg TOKEN=$(System.AccessToken)'
Please split you buildAndPush as it doesn't allow passing arguments into two separate task. For more details please check this question.
Please also make sure that you can Build Service has contributor role on feed settings.
The solution of Krystof Madey does not work for me. At the end I followed the guide How to use secrets inside your Docker build using Azure DevOps and Use your Azure DevOps System.AccessToken for Docker builds… safely to finish the job successfully.
At the end my result looks like the following:
Job:
- job: create_image_and_push_to_acr
displayName: "Create image and push to ACR"
variables:
DOCKER_BUILDKIT: 1
steps:
- script: echo $(System.AccessToken) >> azure_devops_pat
displayName: Get PAT
- task: Docker#2
displayName: "Build"
inputs:
command: build
containerRegistry: $(connection_name)
Dockerfile: $(Build.SourcesDirectory)/Dockerfile
repository: "my_repository"
tags: $(applicationComponentVersion)
arguments: '--secret id=AZURE_DEVOPS_PAT,src=./azure_devops_pat'
- task: Docker#2
displayName: "Push"
inputs:
command: push
containerRegistry: $(connection_name)
repository: "my_repository"
tags: $(applicationComponentVersion)
And inside the docker file:
RUN --mount=type=secret,id=AZURE_DEVOPS_PAT,dst=/azure_devops_pat \
dotnet nuget add source --username this_value_could_be_anything --password `cat /azure_devops_pat` --store-password-in-clear-text --name my_name "https://pkgs.dev.azure.com/.../nuget/v3/index.json" && \
dotnet restore "src/MyProject.csproj"
From an Azure DevOps pipeline, using a self hosted linux build agent running on docker, I get the error below in terraform plan. I tried many things, even running the steps from a bash shell on the build agent: this worked well.
Do you have any suggestion?
2019-09-12T13:55:21.8133489Z ##[debug]Evaluating condition for step: 'Terraform plan'
2019-09-12T13:55:21.8134075Z ##[debug]Evaluating: succeeded()
2019-09-12T13:55:21.8134246Z ##[debug]Evaluating succeeded:
2019-09-12T13:55:21.8134443Z ##[debug]=> True
2019-09-12T13:55:21.8134723Z ##[debug]Result: True
2019-09-12T13:55:21.8134976Z ##[section]Starting: Terraform plan
2019-09-12T13:55:21.8138406Z ==============================================================================
2019-09-12T13:55:21.8138526Z Task : Run Terraform
2019-09-12T13:55:21.8138605Z Description : Run a Terraform on the build agent
2019-09-12T13:55:21.8138647Z Version : 2.4.0
2019-09-12T13:55:21.8138688Z Author : Peter Groenewegen - Xpirit
2019-09-12T13:55:21.8138772Z Help : [More Information](https://pgroene.wordpress.com/2016/06/14/getting-started-with-terraform-on-windows-and-azure/)
2019-09-12T13:55:21.8138828Z ==============================================================================
2019-09-12T13:55:21.8347594Z ##[error]The current operating system is not capable of running this task. That typically means the task was written for Windows only. For example, written for Windows Desktop PowerShell.
2019-09-12T13:55:21.8361383Z ##[debug]System.Exception: The current operating system is not capable of running this task. That typically means the task was written for Windows only. For example, written for Windows Desktop PowerShell.
at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.RunAsync()
at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
2019-09-12T13:55:21.8365066Z ##[section]Finishing: Terraform plan
The pipelines fails on "terraform plan". Here's the yml:
variables:
env: 'environment'
steps:
- task: petergroenewegen.PeterGroenewegen-Xpirit-Vsts-Release-Terraform.Xpirit-Vsts-Release-Terraform.Terraform#2
displayName: 'Terraform plan'
inputs:
TemplatePath: '$(System.DefaultWorkingDirectory)/_repository/tf'
Arguments: 'plan -var-file=$(System.DefaultWorkingDirectory)/_repository/tf/$(env)/$(env).tfvars '
InstallTerraform: true
UseAzureSub: true
ConnectedServiceNameARM: 'deploy-sco'
ManageState: true
SpecifyStorageAccount: true
StorageAccountResourceGroup: 'rg-terraform'
StorageAccountRM: sta
StorageContainerName: terraform
InitArguments: '-backend-config=$(System.DefaultWorkingDirectory)/_repository/tf/$(env)/$(env).beconf'
The build agent is a docker container, built with the following dockerfile
FROM ubuntu:16.04
ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl jq git iputils-ping libcurl3 libunwind8 netcat libssl-dev unzip wget apt-utils apt-transport-https make binutils gcc lsb-release gnupg
RUN wget -P /tmp/download https://releases.hashicorp.com/terraform/0.12.7/terraform_0.12.7_linux_amd64.zip
RUN wget -P /tmp/download -q https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb
RUN unzip /tmp/download/terraform_0.12.7_linux_amd64.zip -d /tmp/download/
RUN mv /tmp/download/terraform /usr/local/bin
RUN chmod a+x /usr/local/bin/terraform
RUN apt-get install /tmp/download/packages-microsoft-prod.deb
RUN apt-get update
RUN apt-get -y install powershell
RUN wget -P /tmp/download -q https://curl.haxx.se/download/curl-7.65.3.tar.gz
RUN cd /tmp/download; tar xzf curl-7.65.3.tar.gz
RUN cd /tmp/download/curl-7.65.3; ./configure --prefix=/opt/curl-7.65.3 --disable-ipv6 --with-ssl; make; make install
RUN mv /usr/bin/curl /tmp; ln -s /opt/curl-7.65.3/bin/curl /usr/bin/curl
RUN curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | tee /etc/apt/trusted.gpg.d/microsoft.asc.gpg > /dev/null
RUN AZ_REPO=$(lsb_release -cs); echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" | tee /etc/apt/sources.list.d/azure-cli.list
RUN apt-get update
RUN apt-get install azure-cli
RUN pwsh -c "Install-Module -Name Az -Force"
RUN pwsh -c "Install-Module -Name Azure -Force"
WORKDIR /azp
COPY ./start.sh .
RUN chmod +x start.sh
CMD ["./start.sh"]
As what you said, yes, the Terraform plan can be compatible including linux, windows and MacOS. But now, the issue you are facing, is the extension and task you are using which named Run Terraform task can only be executed in the agent which installed on Windows. You can see its doc to know that: Getting started with Terraform on Windows and Azure.
There has another extensions which created by our Microsoft DevLabs and individual developer: Terraform and Terraform Build & Release Tasks. The tasks in these two extensions can all compiled in windows, linux and macOS.
You'd better change to use the tasks which in these two extensions.
I've successfully run pipeline with step below to push docker image to registry. When I try to re-run it gives an ERROR: Please run 'az login' to setup account
- bash: az acr helm push -n $(registryName) -u $(registryLogin) -p $(registryPassword) $(build.artifactStagingDirectory)/$(projectName)-$(build.buildId).tgz
displayName: 'az acr helm push'
condition: and(succeeded(), eq(variables['build.sourceBranch'], 'refs/heads/master'))
The step is seen in the logs as below so shouldn't require any additional information (it works when run from command line on my local machine)
az acr helm push -n acrname -u acruser -p password /home/vsts/work/1/a/chart.tgz
The azure cli genuinely does need you to be logged in, so az cli commands in a bash task will fail with this error. That is expected behaviour.
Instead of using the bash task, you should use the Azure CLI task; this includes authentication against an azure subscription as part of its setup, so you will be able to run a bash script including az cli commands.
For example:
- task: AzureCLI#2
displayName: az acr helm push
inputs:
azureSubscription: <Name of the Azure Resource Manager service connection>
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
az acr helm push -n $(registryName) -u $(registryLogin) -p $(registryPassword) $(build.artifactStagingDirectory)/$(projectName)-$(build.buildId).tgz