How do I unlock a concourse pool resource? - concourse

We are trying to use a concourse pool to get locking. It locks just fine but when we try to release the lock, we are getting this error error releasing lock: open /tmp/build/put/maleficent-env/name: no such file or directory.
This is what the directory tree in our pool repo looks like
.
├── README.md
└── maleficent
├── claimed
│ └── maleficent-env
└── unclaimed
this is our resource:
- name: 1.12-env
type: pool
source:
uri: git#github.com:<repo>
branch: master
pool: maleficent
private_key: {{key}}
this acquires the lock:
- put: 1.12-env
params: {acquire: true}
and this is the job yml we think should release it:
- name: run-1.12-errand
plan:
- aggregate:
- get: 1.12-env
passed: [the-job-that-got-the-lock]
- get: ci
on_failure:
put: 1.12-env
params: {release: maleficent-env}
- task: run-errand
file: ci/run-errand/task.yml
params:
BOSH_DIRECTOR_URL: {{url}}
BOSH_CLIENT_SECRET: {{secret}}
ENV_NAME: maleficent
ensure:
put: 1.12-env
params: {release: maleficent-env}
One thing we noticed was when we did the get for the resource, it seems to be Cloning into '/tmp/build/get'… which is a different directory from where the unlocking step is looking for the file. What are we doing wrong?

The following job.yml will release your environment successfully
- name: run-1.12-errand
plan:
- aggregate:
- get: 1.12-env
passed: [the-job-that-got-the-lock]
- get: ci
on_failure:
put: 1.12-env
params: {release: 1.12-env}
- task: run-errand
file: ci/run-errand/task.yml
params:
BOSH_DIRECTOR_URL: {{url}}
BOSH_CLIENT_SECRET: {{secret}}
ENV_NAME: maleficent
ensure:
put: 1.12-env
params: {release: 1.12-env}
The relevant line
put: 1.12-env
params: {release: 1.12-env}
actually refers to two separate things, despite being both using 1.12-env. The put: 1.12-env means "Do a put of the resource named 1.12-env", while the release: 1.12-env means "Release the environment in the file that was the output of the get: 1.12-env"
This is hopefully clearer in the following example
- name: run-1.12-errand
plan:
- aggregate:
- get: my-environment
resource: 1.12-env
passed: [the-job-that-got-the-lock]
- get: ci
on_failure:
put: 1.12-env
params: {release: my-environment}
- task: run-errand
file: ci/run-errand/task.yml
params:
BOSH_DIRECTOR_URL: {{url}}
BOSH_CLIENT_SECRET: {{secret}}
ENV_NAME: maleficent
ensure:
put: 1.12-env
params: {release: my-environment}

I think the issue is with the release: maleficent-env part.
release needs you to provide the path where your lock file is located on the container.
In your case try updating the release block to:
ensure:
put: 1.12-env
params: {release: 1.12-env}
Let me know if that works. If it doesn't work, you can try fly hijack to the put container and see what path the lock is located in. You would then provide that path as the release value.

Related

Checkout another repository in azure pipelines yml

Well, I have the following code that uses a template file in azure Devops:
resources:
repositories:
- repository: templates
type: git
name: "Framework Back-end/templates-devops"
extends:
template: azure-pipelines-template.yml#templates
This works very well, downloading yml file from another project inside same organization. But inside my "azure-pipelines-template.yml" I'm trying to do the following:
- job: Deploy
pool: ${{parameters.agent}}
displayName: Deploy on Kubernetes
dependsOn: Push
condition: and(succeeded(), in(variables['Build.SourceBranchName'], 'master', 'main', 'qas', 'develop'))
steps:
- checkout: self
- checkout: templates
But I got the error:
remote: TF401019: The Git repository with name or identifier templates-devops does not exist or you do not have permissions for the operation you are attempting.
fatal: repository 'https://xx#xx/xxx/Framework%20Back-end/_git/templates-devops/' not found
I need to make a checkout because in other steps I will need to use the files that exist in "template-devops" repository. I can't understand why my pipeline can download the "azure-pipelines-template.yml" file but can't checkout the repository.
SOLVED
Was a permission problem in Settings , I disabled the flags:
Limit job authorization scope to referenced Azure DevOps repositories
Limit job authorization scope to current project for non-release pipelines
Limit job authorization scope to current project for release pipelines
I create a demo to reproduce your environment, but it works well on my side. The checkout step works well. Here is my yaml file and temp file:
Main.yaml
resources:
repositories:
- repository: templates
type: git
name: MyAgile TestPro/yaml
pool:
name: 'default'
extends:
template: azure-pipelines-template.yml#templates
Temp.yaml
stages:
- stage:
jobs:
- job: Deploy
steps:
- checkout: self
- checkout: templates
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "Hello World MyAgileTestPro temp"
My test result:
At present, we recommend you can check if your account has the project admin level for your project Framework Back-end. And please check if there is any name changed about your project.

Pass a file reference as a parameter to a inserted step from another repo in a Azure YAML deployment

I have a repository of yaml based jobs that I'd like to reuse in a number of yaml script. The script that will use the jobs however are in other repository.
The reusable job takes a file path as an input parameter. And for some reason the file can't be found when the imported job executes by the pipeline.
How do I reference the file in the parameter from the main job so it can be found when executing the imported job?
# MyMainScriptTemplate.yml that will be executed by the pipeline
trigger:
- master
resources:
repositories:
- repository: AzureTemplates
type: git
name: AzureTemplates
jobs:
- template: /FunctionApp/DeployFunctionApp.yml#AzureTemplates
parameters:
file: /Azure/Functions/template.json #This can be found when executing ...
# ReusableJobTemplate.yml defines a job that should be referenced from the main script
parameters:
- name: file
type: string
jobs:
- job: DeployFunctionApp
steps:
- task: AzureResourceManagerTemplateDeployment#3
inputs:
deploymentScope: "Resource Group"
azureResourceManagerConnection: "Dev"
subscriptionId: "XYZ"
action: "Create Or Update Resource Group"
resourceGroupName: "XYZ"
location: "West Europe"
templateLocation: "Linked artifact"
csmFile: ${{ parameters.file }}
deploymentMode: "Incremental"
displayName: "Run a one-line script"
Please check how mulirepo behaves.
I would recommend you two steps:
add - checkout: AzureTemplates step before calling template
and change path from /Azure/Functions/template.json to (Agent.BuildDirectory)/AzureTemplates/Azure/Functions/template.json

Pass build directory (/dist) from a job to next job in concourse

I know its not quite simple to do this and tried to explore many approaches but either I couldn't understand it properly or didn't work for me.
I have a concourse job which runs angular build (ng build) and creates /dist folder. This works well.
jobs:
- name: cache
plan:
- get: source
trigger: true
- get: npm-cache
- name: build
plan:
- get: source
trigger: true
passed: [cache]
- get: npm-cache
passed: [cache]
- task: run build
file: source/ci/build.yml
build.yml
---
platform: linux
image_resource:
type: docker-image
source: { repository: alexsuch/angular-cli, tag: '7.3' }
inputs:
- name: source
- name: npm-cache
path: /cache
outputs:
- name: artifact
run:
path: source/ci/build.sh
build.sh
#!/bin/sh
mv cache/node_modules source
cd source
npm rebuild node-saas # temporary fix
npm run build_prod
cp -R dist ../artifact/
I have mentioned output as artifact where I am storing the dist content.
But when I am trying to use this in next job, it doesn't work. Failed with missing input error.
Here is the next job that supposed to consume this dist folder:
jobs:
...
...
- name: list
plan:
- get: npm-cache
passed: [cache, test, build]
trigger: true
- task: list-files
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: artifact
run:
path: ls
args: ['-la', 'artifact/']
Can anyone please help me with this. How I can use the dist folder in above job.
I'm not quite sure why would you want to have different plan definitions for each task but here is the simplest way of doing what you want to do:
jobs:
- name: deploying-my-app
plan:
- get: source
trigger: true
passed: []
- get: npm-cache
passed: []
- task: run build
file: source/ci/build.yml
- task: list-files
file: source/ci/list-files.yml
build.yml
---
platform: linux
image_resource:
type: docker-image
source: { repository: alexsuch/angular-cli, tag: '7.3' }
inputs:
- name: source
- name: npm-cache
path: /cache
outputs:
- name: artifact
run:
path: source/ci/build.sh
list-files.yml
---
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: artifact
run:
path: ls
args: ['-la', 'artifact/']
build.sh
#!/bin/sh
mv cache/node_modules source
cd source
npm rebuild node-saas # temporary fix
npm run build_prod
cp -R dist ../artifact/
Typically you would pass folders as inputs and outputs between TASKS instead of JOBS (althought there's some alternatives)
Concourse is statelessness and that is the idea behind it. But if you want to pass something between jobs the only way to do that is to use a concourse resource and depending on the nature of the project that could be anything from a git repo to a s3 bucket, docker image etc. You can create your own custom resources too.
Using something like s3 concourse resource for example
This way you can push your artifact to an external storage and then use it again on the next jobs on the get step as a resource. But that just may create some unnecesary complexity understanding that what you want to do is pretty straightforward
In my experience I found that sometimes the visual aspect of a job plan in the concourse dashboard gives the impression that a job-plan should by task atomic, which is not always needed
Hope that helps.

trigger concourse job via CLI: "resource not found"

I am attempting to trigger a concourse job from the command line. My pipeline has one resource (a git repo) and one job, which uses that repo. I am seeing:
$ fly -t tutorial trigger-job -j my-pipeline/my-job -w
error: resource not found
However, when I go the web UI and manually trigger the job by pressing the "+" button in the top right, it works fine.
Here is the full pipeline:
resources:
- name: cruise-source
type: git
source:
uri: git#github.com:my-org/cruise.git
branch: develop
jobs:
- name: build-image
public: true
plan:
- get: cruise-source
- task: list-files
config:
platform: linux
image_resource:
type: docker-image
source: {repository: alpine}
inputs:
- name: cruise-source
run:
path: ls
args: [cruise-source]
How can I trigger this job from the CLI?
The "resource not found" you get has nothing to do with the git resource :-) it actually means that the pipeline or job name is wrong. Looking at your pipeline configuration, you should issue
fly -t tutorial trigger-job -j my-pipeline/build-image -w
or if your configuration is different from what you have posted, maybe you have a typo in the name of the pipeline or job.

ConcourseCI: Run task from mapped/renamed output of get resource

I have a repo of which I'm looking at various folders at and building different things in each repo.
Since a lot of the steps are similar I was trying to streamline things a bit and use output mapping to "rename" the dir to a common name, but it doesn't seem to behave. All I can get is an error: "unknown artifact source: repo"
(A snippet of) My pipeline is:
resources:
# I have more of these, one for each path I'm interested in but not shown here.
- name: repo-folder--11.1--common
type: git
source:
uri: git#github.com:myorg/project
branch: concourse-pipeline
private_key: {{github_private_key}}
paths:
- 11.1/common
jobs:
- name: common-image-build
plan:
- get: repo-folder--11.1--common
output_mapping:
repo-folder--11.1--common: repo
trigger: true
- get: centos-docker-image
- task: generate-tag
file: repo/task-generate-tag.yml
params:
prefix: "1.11-"
I was hoping that the output_mapping on my get would let me refer to that git repo via a simpler name ("repo") in this build plan, but it doesn't seem to.
Am I missing some way of achieving this or is this a bug/design decision?
No need to use output_mapping, resource get has its own way of "renaming", by specifying the resource.
resources:
- name: repo-folder--11.1--common
type: git
source:
uri: git#github.com:myorg/project
branch: concourse-pipeline
private_key: {{github_private_key}}
paths:
- 11.1/common
jobs:
- name: common-image-build
plan:
- get: repo
resource: repo-folder--11.1--common
trigger: true
- get: centos-docker-image
- task: generate-tag
file: repo/task-generate-tag.yml
params:
prefix: "1.11-"