Wanted to know if there is any flag/option for concourse tasks inside a single job so that all tasks gets executed regardless of any task failing.
Thanks!
Totally. By default, tasks run sequentially. If you want them to run independently of the sequence place them in the in_parallel key, like in the following pipeline:
jobs:
- name: parallel-tasks
plan:
- in_parallel:
- task: failing-task
config:
platform: linux
image_resource:
type: docker-image
source:
repository: alpine
run:
path: /bin/sh
args: [ "-c", "exit 1"]
- task: passing-task
config:
platform: linux
image_resource:
type: docker-image
source:
repository: alpine
run:
path: /bin/sh
args: [ "-c", "exit 0"]
Running it will produce the following output:
in_parallel works with tasks as well as resources (e.g. running get in parallel)
Related
I have a workflow that runs DAG with 2 tasks.
Each task clones a repo.
I want that the first repo in the 1st task, will be available for the second task which runs another container.
How can I share the repos between them?
Using volume persisent, artifacts, outputs?
kind: Workflow
metadata:
name: create-configmap-dag-workflow
spec:
entrypoint: create-configmap-dag
templates:
- name: create-configmap-dag
dag:
tasks:
- name: automation-npm-packages
template: run-shell-command
arguments:
parameters:
- name: cmd
value: git clone https://github.com/org/pulumi-automation.git && cd automation && npm install
- name: campaign-update-npm-packages
template: run-shell-command
depends: automation-npm-packages.Succeeded
arguments:
parameters:
- name: cmd
value: git clone https://github.com/org/campaign-update.git && cd campaign-update/infra && npm install
- name: run-shell-command
inputs:
parameters:
- name: cmd
container:
image: amazonaws.com/jenkins-slave:ecs-global-node_master-3
command: [ "sh", "-c" ]
args: [ "{{inputs.parameters.cmd}}" ]
I manage to use them fine as long as I don't need to pass custom arguments.
Lets say I want to use an official Docker image: somePublicImage:1.2.3; then the following works fine:
stages:
- stage: Build
jobs:
- job: BuildTestPack
displayName: 'Build, test & pack'
timeoutInMinutes: 5
cancelTimeoutInMinutes: 2
services:
someService:
image: somePublicImage:1.2.3
ports:
- 4223:4222
There's an option to configure the container with --foo bar
How do I define this in a Azure build pipeline?
I've tried:
command
options
arguments
entrypoint
Service containers must define a CMD or ENTRYPOINT. The pipeline will docker run the provided container without additional arguments.
Check the link below:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/service-containers?view=azure-devops&tabs=yaml
Seems like you need to create a "custom" resource container first. E.g
resources:
containers:
- container: myThing
image: somePublicImage:1.2.3
ports:
- 4223:4222
volumes:
- /docker_vol_config:/config
command: '--foo bar'
which then can be used as a service:
stages:
- stage: Build
jobs:
- job: BuildTestPack
displayName: 'Build, test & pack'
timeoutInMinutes: 5
cancelTimeoutInMinutes: 2
services:
myThing:myThing
I have two jobs viz. build and publish. I want publish to trigger after build is done. So, I am using an external resource gcs-resourcehttps://github.com/frodenas/gcs-resource
Following is my pipeline.yml:
---
resource_types:
- name: gcs-resource
type: docker-image
source:
repository: frodenas/gcs-resource
resources:
- name: proj-repo
type: git
source:
uri: <my uri>
branch: develop
username: <username>
password: <password>
- name: proj-gcr
type: docker-image
source:
repository: asia.gcr.io/myproject/proj
tag: develop
username: _json_key
password: <my password>
- name: proj-build-output
type: gcs-resource
source:
bucket: proj-build-deploy
json_key: <my key>
regexp: Dockerfile
jobs:
- name: build
serial_groups: [proj-build-deploy]
plan:
- get: proj
resource: proj-repo
- task: build
config:
platform: linux
image_resource:
type: docker-image
source: {repository: node, tag: 10.13.0}
inputs:
- name: proj
run:
path: sh
args:
- -exc
- |
<do something>
- put: proj-build-output
params:
file: proj/Dockerfile
content_type: application/octet-stream
- name: publish
serial_groups: [proj-build-deploy]
plan:
- get: proj-build-output
trigger: true
passed: [build]
- put: proj-gcr
params:
build: proj-build-output
I am using the external resource proj-build-output to trigger the next job. I can run the individual jobs without any problem, however the the publish job doesn't automatically get triggered after completion of build job.
Am I missing something?
The regexp of the gcs-resource is misconfigured:
...
regexp: Dockerfile
...
while regexp, as the original S3 resource from which it comes from, wants:
regexp: the pattern to match filenames against within GCS. The first grouped match is used to extract the version, or if a group is explicitly named version, that group is used.
The https://github.com/frodenas/gcs-resource#example-configuration shows its correct usage:
regexp: directory_on_gcs/release-(.*).tgz
This is not specific to the GCS or S3 resource; Concourse needs a "version" to move artifacts from jobs to storage and back. It is one of the fundamental concepts of Concourse. See https://web.archive.org/web/20171205105324/http://concourse.ci:80/versioned-s3-artifacts.html for an example.
As Marco mentioned, the problem was with versioning.
I solved my issue using these two steps:
Enabled versioning on my GCS Bucket https://cloud.google.com/storage/docs/object-versioning#_Enabling
Replaces regexp with versioned_file as mentioned in the docs https://github.com/frodenas/gcs-resource#file-names
I am deploying a simple angular4 application to cloud foundry using static_buildpack. While accessing the application I am always getting nginx 403 issue.
jobs:
- name: app
serial: true
plan:
- get: develop-repo
- task: npm-build
config:
platform: linux
image_resource:
type: docker-image
source:
repository: node
run:
path: sh
args:
- -exec
- |
cd develop-repo
npm install
npm run dist
inputs:
- name: develop-repo
outputs:
- name:
- put: develop
params:
manifest: develop-repo/manifest.yml
current_app_name: app
path: develop-repo
resources:
- name: develop-repo
type: git
- name: develop
type: cf
manifest.yml:
---
applications:
- name: app
instances: 1
memory: 512M
disk_quota: 512M
buildpack: staticfile_buildpack
stack: cflinuxfs2
All I am doing is git clone -> npm build -> cf deploy
Note: All resource variables are rightly set. Just ignored for better readability
After trying out couple of options, I found that by publishing the artifacts to the output folder we can push the app from the output folder
---
jobs:
- name: app
serial: true
plan:
- get: develop
- task: npm-build
config:
platform: linux
image_resource:
type: docker-image
source:
repository: node
inputs:
- name: develop
outputs:
- name: artifacts
run:
path: sh
args:
- -exec
- |
cd develop
npm install
npm run dist
ls
cp -R dist ../artifacts/
- put: deploy-cf
params:
manifest: develop/ci/manifests/manifest-int.yml
path: artifacts/dist
resources:
- name: develop
type: git
source:
uri: <<GITHUB-URI>>
branch:<<GITHUB-BRANCH>>
username:<<GITHUB-USERNAME>>
password: <<GITHUB-PASSWORD>>
- name: deploy-cf
type: cf
source:
api: <<CF-API>>
username: <<CF-USERNAME>>
password: <<CF-PASSWORD>>
organization: <<CF-ORG>>
space: <<CF-SPACE>>
In Concourse CI, by default, the underlying container for a job's task is instantiated and run with user root.
If the container used for my task needs to be executed with a different user (e.g. postgres), how can I do that in Concourse?
Concourse tasks provide a user parameter to explicitly set the user to run its container as.
See http://concourse-ci.org/running-tasks.html#task-run-user .
Here is a sample Concourse pipeline to demonstrate the use of that parameter:
---
jobs:
- name: check-container-user
plan:
- do:
- task: container-user-postgres
config:
platform: linux
image_resource:
type: docker-image
source:
repository: postgres
tag: "latest"
run:
user: postgres
path: sh
args:
- -exc
- |
whoami
echo "Container running with postgres user"