We are currently using CircleCI to run our automated tests and would like to migrate to Azure DevOps to run those tests on an Azure Pipeline. Our applications are fully dockerized and I am having trouble executing the tests in the container on Azure Pipeline.
My goal is simply to build the image, push it to our Docker Hub repo and then pull it to execute PHPUnit. The first part is OK, I managed to push the image.
Then I created a job to execute a simple script, and I would like for it to run inside the container. My pipeline conf file will follow. The step that fails currently is the container initialization of the second job. It fails with the error :
/usr/bin/docker pull [redacted]:azure-master
Error response from daemon: pull access denied for [redacted], repository does not exist or may require 'docker login': denied: requested access to the resource is denied
trigger:
- master
resources:
- repo: self
stages:
- stage: Build
displayName: Build image
jobs:
- job: Build_and_push
displayName: Build and push image
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Docker#2
displayName: Build and push image
inputs:
containerRegistry: 'Docker Hub'
repository: '[redacted]'
command: 'buildAndPush'
Dockerfile: '**/Dockerfile'
tags: 'azure-$(Build.SourceBranchName)'
- task: Docker#2
displayName: Login to docker repo
inputs:
containerRegistry: 'Docker Hub'
command: 'login'
- job: Install_composer_and_run_tests
dependsOn: ['Build_and_push']
pool:
vmImage: 'ubuntu-latest'
container: [redacted]:azure-$(Build.SourceBranchName)
steps:
- task: Docker#2
displayName: Login to docker repo
inputs:
containerRegistry: 'Docker Hub'
command: 'login'
- script: composer install -n --prefer-dist
- script: php vendor/bin/phpunit tests/ --group me
I don't really understand how or where I should login because I use the container param in the job, not a task to pull the image. Plus I have not problem pushing the image even though I did not explicitly login at that step. Last thing is that I have created a container registry in Azure DevOps (Docker Hub), with my credentials and it works correctly.
Thanks for your help :)
See Endpoints:
Containers can be hosted on registries other than Docker Hub. To host an image on Azure Container Registry or another private container registry, add a service connection to the private registry. Then you can reference it in a container spec:
container:
image: xxx/xxx:tag
endpoint: xxx
According to your error message, you may need to provide credentials for the Initialize Containers step. So we should use this format:
- job: Install_composer_and_run_tests
dependsOn: ['Build_and_push']
pool:
vmImage: 'ubuntu-latest'
container:
image: [redacted]:azure-$(Build.SourceBranchName)
endpoint: 'Docker Hub'
steps:
...
Related
I'm trying to use service containers within Azure DevOps pipelines
The agent is an ubuntu host
I would like to have the agent run a powershell container and a playwright container
The doc for this is not very verbose
So far I have this in my main 'azure-pipelines.yml'
trigger: none
pr: none
resources:
containers:
- container: playwright
image: mcr.microsoft.com/playwright:v1.29.0-focal
- container: pwsh
image: mcr.microsoft.com/powershell
pool:
vmImage: "ubuntu-latest"
services:
playwright: playwright
pwsh: pwsh
stages:
- stage: dev
displayName: dev
jobs:
- template: templates/test.yml
And this in my 'template/test.yml' file
- job: run_tests
displayName: Test
pool:
vmImage: ubuntu-latest
steps:
- powershell: |
Write-Host "This is powershell"
target:
container: pwsh
- script: yarn test:integration:ci
displayName: "Run tests"
env:
environment: dev
CI: true
target:
container: playwright
Azure pipelines does not like this. It is failing with:
/.azure/azure-pipelines.yml (Line: 18, Col: 1): Unexpected value 'stages'
when I try to run the pipeline. I thought stages: was the first part of a pipeline? (but I am very new to Azure pipelines so my understanding might be way off)
Could anyone help to clarify why/where I am screwing up at all please?
Thanks
Make the following changes to your yaml files.
azure-pipelines.yml
trigger: none
pr: none
pool:
vmImage: ubuntu-latest
resources:
containers:
- container: playwright
image: mcr.microsoft.com/playwright:v1.29.0-focal
- container: pwsh
image: mcr.microsoft.com/powershell
stages:
- stage: dev
displayName: dev
jobs:
- template: templates/test.yml
template.yml
jobs:
- job: run_tests
displayName: Test
pool:
vmImage: ubuntu-latest
services:
playwright: playwright
pwsh: pwsh
steps:
- powershell: |
Write-Host "This is powershell"
target:
container: pwsh
- script: yarn test:integration:ci
displayName: "Run tests"
env:
environment: dev
CI: true
target:
container: playwright
Reason why you were getting the error (/.azure/azure-pipelines.yml (Line: 18, Col: 1): Unexpected value 'stages') is because of the property services. According to the example in service containers documentation, the property services is defined in the root level of the yaml because the example did not use any stage or jobs.
Since you are using stages and jobs in your yaml pipeline, the services property should be nested within your job.
Hence, I have moved the services to the template.yml file. You can check which property is allowed under a stage or job using this YAML schema documentation
Reference: https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/jobs-job-container?view=azure-pipelines
I'm trying to run a container that runs a program until it finishes, as a step in Azure DevOps pipeline Job.
From documentation it looks that what's needed is a service container.
My pipeline yaml is:
trigger:
- main
resources:
containers:
- container: mycontainer
image: mycontainer:latest
endpoint: myregistry
pool:
vmImage: ubuntu-latest
services:
syncice: mycontainer
steps:
- script: |
ls
printenv
When the container is docker run locally the program shows output, but from DevOps Job no output is showing.
How to start the container and see output in Job?
My self-hosted docker based runner already activated at GitHub organization level. Separately to perform npm build through our private docker image, Trying with below workflow yml code.
name: CI with Docker
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
build:
runs-on: docker-runner # This is our Self-hosted docker runner
container:
image: ubuntu:npm-1 # This is our private docker image.
steps:
- uses: actions/checkout#v2
- name: Build
run: npm install
It fails with Error: docker: command not found this is expected since my docker-runner image doesn't loaded with docker engine package.
With Self-hosted docker runner, How do i call a private docker image and perform the build? Some pointers to achieve would be helpful. Thanks!
I have a Multistage YAML pipeline containing two stages 1) Build and 2) Deploy. Deploy stage is mentioned below and I want to add pre deploy approvals in that stage before deploy task. How can I add pre deployment and Post deployment approvals in multistage YAML pipeline?
stages:
- stage: 'Build'
# RESTORE
# Some task implementation
# BUILD
# Some task implementation
# PUBLISH
# Some task implementation
# DEPLOY STAGE
- stage: 'Dev'
displayName: 'Deploy to the dev environment'
dependsOn: Build
jobs:
- deployment: Deploy
pool:
vmImage: 'ubuntu-16.04'
environment: dev
variables:
- group: Release
strategy:
# HOW TO ADD PRE DEPLOYMENT AND POST DEPLOYMENT APPROVALS?
runOnce:
deploy:
steps:
- download: current
artifact: drop
- task: AzureWebApp#1
displayName: 'Azure App Service Deploy: website'
inputs:
azureSubscription: 'Resource Manager - Tailspin - Space Game'
appName: '$(WebAppNameDev)'
package: '$(Pipeline.Workspace)/drop/$(buildConfiguration)/*.zip'
For this issue ,currently, manual approval and evaluate artifact are the only available checks, and they can be configured on environments, service connections and agent pools only.
To define an approval on an environment:
In your Azure DevOps project, go to the environment that needs to be protected. (Learn more about creating an environment.)
Navigate to Approvals and Checks for the environment.
Select Create, provide the approvers and an optional message, and select Create again to to complete addition of the manual approval check.
Then use the environment: 'xxx' parameter in your yaml file. For example:
- stage: deploy
jobs:
- deployment: DeployWeb
displayName: deploy Web App
pool:
vmImage: 'Ubuntu-16.04'
# creates an environment if it doesn't exist
environment: 'multiStage'
The GUI and the yaml are interdependent in this case, it's not straight yaml.
For details ,please refer to this official document.
I am running a server with 10 azure devops agents. I have created a build that has 5 jobs and once in awhile we get an error initialize the container for a job. The image is already on the server so it doesn't need to download the image but as you can see the below log it fails to login
View raw log
'1.40'
Docker daemon API version: '1.40'
C:\Program Files\Docker\docker.EXE version --format '{{.Client.APIVersion}}'
'1.40'
Docker client API version: '1.40'
C:\Program Files\Docker\docker.EXE ps --all --quiet --no-trunc --filter "label=413526"
C:\Program Files\Docker\docker.EXE network prune --force --filter "label=413526"
C:\Program Files\Docker\docker.EXE login --username "***" --password *** https://***
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in C:\Users\devagent\.docker\config.json.
Configure a credential helper to remove this warning. See
Login Succeeded
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
C:\Program Files\Docker\docker.EXE pull ***/builds/vs2017:1.0.0.0
Error response from daemon: Get https://***/v2/builds/vs2017/manifests/1.0.0.0: unauthorized: authentication required
##[warning]Docker pull failed with exit code 1, back off 3.597 seconds before retry.
C:\Program Files\Docker\docker.EXE pull ***/builds/vs2017:1.0.0.0
Error response from daemon: Get https://***/v2/builds/vs2017/manifests/1.0.0.0: unauthorized: authentication required
##[warning]Docker pull failed with exit code 1, back off 3.068 seconds before retry.
C:\Program Files\Docker\docker.EXE pull ***/builds/vs2017:1.0.0.0
Error response from daemon: Get https://***/v2/builds/vs2017/manifests/1.0.0.0: unauthorized: authentication required
C:\Program Files\Docker\docker.EXE logout https://***
Not logged in to ***
##[error]Docker pull failed with exit code 1
Finishing
Yaml Build
variables:
application.Name: 'Product'
application.version: '5.12.0.0'
name: '$(application.name)_$(application.version)_$(Date:yyyMMdd)$(Rev:.rr)'
resources:
- repo: self
clean: true
schedules:
- cron: "0 8 * * Mon-Fri"
displayName: Nightly Build
branches:
include:
- master
- release/*/base
stages:
- stage: CI
displayName: 'Continuous Integration'
condition: |
or(
in(variables['Build.Reason'], 'PullRequest'),
and(
in(variables['Build.Reason'], 'Manual'),
eq(variables['ManualCI'], 'true')
)
)
jobs:
- job: ProductCI
displayName: Product
condition: succeeded()
pool:
name: DW
workspace:
clean: all
timeoutInMinutes: 120
steps:
- template: templates/product-buildsteps.yml
- template: templates/product-api-buildsteps.yml
- job: AutomationCI
displayName: Automation
condition: succeeded()
workspace:
clean: all
pool:
name: DW
container:
image: AzureContainerRepo/builds/vs2017:1.0.0.0
endpoint: Azure Container Builds
steps:
- template: templates/product-automation-buildsteps.yml
- job: FluentMigratorCI
displayName: FluentMigrator
condition: succeeded()
workspace:
clean: all
pool:
name: DW
container:
image: AzureContainerRepo/builds/vs2017:1.0.0.0
endpoint: Azure Container Builds
steps:
- template: templates/product-fm-buildsteps.yml
- job: DeploymentCheckerCI
displayName: Deployment Checker
condition: succeeded()
pool:
name: DW
container:
image: AzureContainerRepo/builds/vs2017:1.0.0.0
endpoint: Azure Container Builds
steps:
- template: templates/product-deploymentchecker-buildsteps.yml