I have a repo of which I'm looking at various folders at and building different things in each repo.
Since a lot of the steps are similar I was trying to streamline things a bit and use output mapping to "rename" the dir to a common name, but it doesn't seem to behave. All I can get is an error: "unknown artifact source: repo"
(A snippet of) My pipeline is:
resources:
# I have more of these, one for each path I'm interested in but not shown here.
- name: repo-folder--11.1--common
type: git
source:
uri: git#github.com:myorg/project
branch: concourse-pipeline
private_key: {{github_private_key}}
paths:
- 11.1/common
jobs:
- name: common-image-build
plan:
- get: repo-folder--11.1--common
output_mapping:
repo-folder--11.1--common: repo
trigger: true
- get: centos-docker-image
- task: generate-tag
file: repo/task-generate-tag.yml
params:
prefix: "1.11-"
I was hoping that the output_mapping on my get would let me refer to that git repo via a simpler name ("repo") in this build plan, but it doesn't seem to.
Am I missing some way of achieving this or is this a bug/design decision?
No need to use output_mapping, resource get has its own way of "renaming", by specifying the resource.
resources:
- name: repo-folder--11.1--common
type: git
source:
uri: git#github.com:myorg/project
branch: concourse-pipeline
private_key: {{github_private_key}}
paths:
- 11.1/common
jobs:
- name: common-image-build
plan:
- get: repo
resource: repo-folder--11.1--common
trigger: true
- get: centos-docker-image
- task: generate-tag
file: repo/task-generate-tag.yml
params:
prefix: "1.11-"
Related
I have a shared Azure pipeline yaml definition with the purpose to define one CodeAnalysis pipeline per repository.
How can I define the repository name dynamically?
I tried with name: '$(projectName)' which leads to the error:
The repository $(projectName) in project 8ab9d22b-6819-483b-829d-******* could not be retrieved. Verify the name and credentials being used.
azure-pipelines.yml
resources:
repositories:
- repository: codeAnalysisRepo
type: git
name: shared/codeanalysis
- repository: SourceRepo
type: git
name: '$(projectName)'
jobs:
- job: 'BackendCodeAnalysis'
pool:
name: '$(AgentPool)'
steps:
- checkout: SourceRepo
clean: true
- template: sonarqube_msbuild_prepare.yml#codeAnalysisRepo
parameters:
projectKey: '$(project)'
projectName: '$(project)'
- task: DotNetCoreCLI#2
displayName: "build DestRepo"
inputs:
command: 'build'
projects: '$(Build.Repository.LocalPath)/**/*.csproj'
configuration: Release
- template: sonarqube_execute.yml#codeAnalysisRepo
It works when I hardcode the name
Currently, set parameter and variable is not supported in resources -> repositories.
A work around for this, you could set this at the checkout step. Here is s sample: Check out multiple repositories in your pipeline - Azure Pipelines | Microsoft Docs. Please note that, the repos should be in the same organization.
resources:
repositories:
- repository: Repo1
type: git
name: Artifacts/Repo1
jobs:
- job: 'BackendCodeAnalysis'
pool:
vmimage: windows-latest
steps:
- checkout: git://$(projectName)
clean: true
For your demand, you could create a suggestion ticket via: https://developercommunity.visualstudio.com/report?space=21&entry=problem.
With Auzre pipeline, I would like to how to reuse the variables declared in another repository.
Just a background, if all the three files organization_vars.yml, project_vars.yml, and cicd.yml are in the same repo, without using the resources like shown in project_vars.yml, it works.
Now I need to put the file organization_vars.yml in another repository in order to share it to every projects in the same organization, I tried with resources as explained here: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops#use-other-repositories , but it doesn't work.
In the pipeline, I got the error message Unexpected value 'resources' from the very beginning.
Hereunder a simplified example for the 3 files:
# file organization_vars.yml in the calling repository
variables:
orgName: xxx
# file project_vars.yml in the caller repository
resources:
repositories:
- repository: calling_repository
type: git
name: another_project/calling_repository
variables:
- template: organization_vars.yml#caller_repository
- name: callerVar
value: $(orgName)_xxx
# file cicd.yml in the caller repository, this file will be runned by the pipeline
trigger: xxx
variables:
- template: project_vars.yml
- name: otherVar
value: xxx
pool:
vmImage: xxx
steps:
xxx
I found it by myself, just need to move the resources part from the file project_vars.yml to the file cicd.yml
I have run in to an odd problem after converting a bunch of my YAML pipelines to use templates for holding job logic as well as for defining my pipeline variables. The pipelines run perfectly fine, however I get a "Some recent issues detected related to pipeline trigger." warning at the top of the pipeline summary page and viewing details only states: "Configuring the trigger failed, edit and save the pipeline again."
The odd part here is that the pipeline works completely fine, including triggers. Nothing is broken and no further details are given about the supposed issue. I currently have YAML triggers overridden for the pipeline, but I did also define the same trigger in the YAML to see if that would help (it did not).
I'm looking for any ideas on what might be causing this or how I might be able to further troubleshoot it given the complete lack of detail that the error/warning provides. It's causing a lot of confusion among developers who think there might be a problem with their builds as a result of the warning.
Here is the main pipeline. the build repository is a shared repository for holding code that is used across multiple repos in the build system. dev.yaml contains dev environment specific variable values. Shared holds conditionally set variables based on the branch the pipeline is running on.
name: ProductName_$(BranchNameLower)_dev_$(MajorVersion)_$(MinorVersion)_$(BuildVersion)_$(Build.BuildId)
resources:
repositories:
- repository: self
- repository: build
type: git
name: Build
ref: master
# This trigger isn't used yet, but we want it defined for later.
trigger:
batch: true
branches:
include:
- 'dev'
variables:
- template: YAML/variables/shared.yaml#build
- template: YAML/variables/dev.yaml#build
jobs:
- template: ProductNameDevJob.yaml
parameters:
pipelinePool: ${{ variables.PipelinePool }}
validRef: ${{ variables.ValidRef }}
Then this is the start of the actual job yaml. It provides a reusable definition of the job that can be used in more than one over-arching pipeline:
parameters:
- name: dependsOn
type: object
default: {}
- name: pipelinePool
default: ''
- name: validRef
default: ''
- name: noCI
type: boolean
default: false
- name: updateBeforeRun
type: boolean
default: false
jobs:
- job: Build_ProductName
displayName: 'Build ProductName'
pool:
name: ${{ parameters.pipelinePool }}
demands:
- msbuild
- visualstudio
dependsOn:
- ${{ each dependsOnThis in parameters.dependsOn }}:
- ${{ dependsOnThis }}
condition: and(succeeded(), eq(variables['Build.SourceBranch'], variables['ValidRef']))
steps:
**step logic here
Finally, we have the variable YAML which conditionally sets pipeline variables based on what we are building:
variables:
- ${{ if or(eq(variables['Build.SourceBranch'], 'refs/heads/dev'), eq(variables['Build.SourceBranch'], 'refs/heads/users/ahenderson/azure_devops_build')) }}:
- name: BranchName
value: Dev
** Continue with rest of pipeline variables and settings of each value for each different context.
You can check my post here : Azure DevOps pipeline trigger issue message not going away
As I can see in your YAML file, you are using this branch : 'refs/heads/users/ahenderson/azure_devops_build'.
I think some YAML files you are refering are missing from the branch defined as default in your build there :
Switch to your branch
I think I may have figured out the problem. It appears that this is related to the use of conditionals in the variable setup. While the variables will be set in any valid trigger configuration, it appears that the proper values are not used during validation and that may have been causing the problem. Switching my conditional variables to first set a default value and then replace the value conditionally seems to have fixed the problem.
It would be nice if Microsoft would give a more useful error message here, something to the extent of the values not being found for a given variable, but adding defaults does seem to have fixed the problem.
In our case it was because the path to the YAML file started with a slash: /builds/build.yaml
Removing the slash fixed the error: builds/build.yaml
In my case I had replaced the trigger in the yaml-file. The pipeline then does not now where to start.
# ASP.NET
# Build and test ASP.NET projects.
# Add steps that publish symbols, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/apps/aspnet/build-aspnet-4
trigger:
- "main" # <-- That was name wrong
For me it was this causing the problem...
The following pipeline.yaml causes the error “Configuring the trigger failed, edit and save the pipeline again”. It has to do with the environment name name: FeatureVMs.${{ variables.resourceName }}, if i replace ${{ variables.resourceName }} with something else e.g. FeatureVMs.develop then the error does not occur. The strange thing is, if i once save the pipeline with all triggers I want and a valid environment FeatureVMs.develop then it saves the triggers, if i then change it to what i actually want a dynamic environment resource selection FeatureVMs.${{ variables.resourceName }} then the error occurs but Azure Dev Ops the pipeline works as i expect it. So the workaround is to save it once without a variable and the triggers you want and then with the variable and live with the error on top of the pipeline
This causes the error
trigger: none
variables:
- name: resourceName
value: $(Build.SourceBranchName)
- name: sourcePipeline
value: vetsxl-ci
resources:
pipelines:
- pipeline: vetsxl-ci
source: vetsxl-ci
trigger:
branches:
include:
- develop
- feature/F*
- release/*
- review/*
- demo/*
- hotfix/H*
- tests/*
- test/*
stages:
- stage: Deploy
displayName: 'Deploy'
condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
jobs:
- deployment: DeployVM
displayName: 'Deploy to develop VM'
environment:
name: FeatureVMs.${{ variables.resourceName }}
strategy:
rolling:
deploy:
steps:
- template: deploy.yml
parameters:
sourcePipeline: ${{ variables.sourcePipeline }}
This works without any errors.
trigger: none
variables:
- name: resourceName
value: $(Build.SourceBranchName)
- name: sourcePipeline
value: vetsxl-ci
resources:
pipelines:
- pipeline: vetsxl-ci
source: vetsxl-ci
trigger:
branches:
include:
- develop
- feature/F*
- release/*
- review/*
- demo/*
- hotfix/H*
- tests/*
- test/*
stages:
- stage: Deploy
displayName: 'Deploy'
condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
jobs:
- deployment: DeployVM
displayName: 'Deploy to develop VM'
environment:
name: FeatureVMs.develop
strategy:
rolling:
deploy:
steps:
- template: deploy.yml
parameters:
sourcePipeline: ${{ variables.sourcePipeline }}
For some reasons, I want my A.yml calling another B.yml from others repository.
Pipeline A is consider an 'empty' pipeline, its job is basically calling pipeline B, and B done the real job.
But B is not just steps, it contain all the details, it included 'Resources' also.
This is how I do:
Pipeline A:
trigger:
- main
resources:
repositories:
- repository: script
type: github
name: xxx/script
ref: refs/heads/main
endpoint: smartracks
steps: **<---- What should I put ?**
- template: B.yml#script
Pipeline B:
resources:
repositories:
- repository: rcu-service
type: github
name: abc/rcu-service
ref: refs/heads/main
endpoint: test
jobs:
- job: OpenEmbedded_Build_And_Export
steps:
- checkout: rcu-service
- script: |
......
If I excluded the "resources" in pipeline B, it will success (need add those resources into pipeline A).
But once I included resource in pipeline B, it fails with these message:
B.yml#script(Line: 1, Col: 1): Unexpected value 'resource'
B.yml#script(Line: 24, Col: 1): Unexpected value 'jobs'
In Pipeline A, this is how I call the pipeline B, I use steps, but it seems doesn't work.
steps: **<---- What should I put ?**
- template: B.yml#script
I try with stages, jobs, but fail too.
So, I am wonder what should I do ?
Please teach me, thank you.
Azure Pipelines supports four kinds of templates:
Stage
Job
Step
Variable
It doesn't support resources, you need to put the resources in your A.yml.
https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema#template-references
Imagine that you were to expand the template so that it showed up in the file where you placed - template:.... In this case, you would have
steps:
- jobs:
...
This doesn't work, because the schema requires steps to be a part of a job.
jobs:
- template: B.yml#script
or
stages:
- stage: stageName
displayName: "my stage"
jobs:
- template: B.yml#script
I know its not quite simple to do this and tried to explore many approaches but either I couldn't understand it properly or didn't work for me.
I have a concourse job which runs angular build (ng build) and creates /dist folder. This works well.
jobs:
- name: cache
plan:
- get: source
trigger: true
- get: npm-cache
- name: build
plan:
- get: source
trigger: true
passed: [cache]
- get: npm-cache
passed: [cache]
- task: run build
file: source/ci/build.yml
build.yml
---
platform: linux
image_resource:
type: docker-image
source: { repository: alexsuch/angular-cli, tag: '7.3' }
inputs:
- name: source
- name: npm-cache
path: /cache
outputs:
- name: artifact
run:
path: source/ci/build.sh
build.sh
#!/bin/sh
mv cache/node_modules source
cd source
npm rebuild node-saas # temporary fix
npm run build_prod
cp -R dist ../artifact/
I have mentioned output as artifact where I am storing the dist content.
But when I am trying to use this in next job, it doesn't work. Failed with missing input error.
Here is the next job that supposed to consume this dist folder:
jobs:
...
...
- name: list
plan:
- get: npm-cache
passed: [cache, test, build]
trigger: true
- task: list-files
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: artifact
run:
path: ls
args: ['-la', 'artifact/']
Can anyone please help me with this. How I can use the dist folder in above job.
I'm not quite sure why would you want to have different plan definitions for each task but here is the simplest way of doing what you want to do:
jobs:
- name: deploying-my-app
plan:
- get: source
trigger: true
passed: []
- get: npm-cache
passed: []
- task: run build
file: source/ci/build.yml
- task: list-files
file: source/ci/list-files.yml
build.yml
---
platform: linux
image_resource:
type: docker-image
source: { repository: alexsuch/angular-cli, tag: '7.3' }
inputs:
- name: source
- name: npm-cache
path: /cache
outputs:
- name: artifact
run:
path: source/ci/build.sh
list-files.yml
---
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: artifact
run:
path: ls
args: ['-la', 'artifact/']
build.sh
#!/bin/sh
mv cache/node_modules source
cd source
npm rebuild node-saas # temporary fix
npm run build_prod
cp -R dist ../artifact/
Typically you would pass folders as inputs and outputs between TASKS instead of JOBS (althought there's some alternatives)
Concourse is statelessness and that is the idea behind it. But if you want to pass something between jobs the only way to do that is to use a concourse resource and depending on the nature of the project that could be anything from a git repo to a s3 bucket, docker image etc. You can create your own custom resources too.
Using something like s3 concourse resource for example
This way you can push your artifact to an external storage and then use it again on the next jobs on the get step as a resource. But that just may create some unnecesary complexity understanding that what you want to do is pretty straightforward
In my experience I found that sometimes the visual aspect of a job plan in the concourse dashboard gives the impression that a job-plan should by task atomic, which is not always needed
Hope that helps.