I am attempting to trigger a concourse job from the command line. My pipeline has one resource (a git repo) and one job, which uses that repo. I am seeing:
$ fly -t tutorial trigger-job -j my-pipeline/my-job -w
error: resource not found
However, when I go the web UI and manually trigger the job by pressing the "+" button in the top right, it works fine.
Here is the full pipeline:
resources:
- name: cruise-source
type: git
source:
uri: git#github.com:my-org/cruise.git
branch: develop
jobs:
- name: build-image
public: true
plan:
- get: cruise-source
- task: list-files
config:
platform: linux
image_resource:
type: docker-image
source: {repository: alpine}
inputs:
- name: cruise-source
run:
path: ls
args: [cruise-source]
How can I trigger this job from the CLI?
The "resource not found" you get has nothing to do with the git resource :-) it actually means that the pipeline or job name is wrong. Looking at your pipeline configuration, you should issue
fly -t tutorial trigger-job -j my-pipeline/build-image -w
or if your configuration is different from what you have posted, maybe you have a typo in the name of the pipeline or job.
Related
I am trying to setup my first GitHub Workflow and I am facing many YAML syntax issues even I am using the official documentation.
I am using the below YAML:
# This is a basic workflow to help you get started with Actions
name: TestWorkflowGithub
# Controls when the workflow will run
on:
# Triggers the workflow on push or pull request events but only for the "main" branch
pull_request:
branches:
- 'testbranch/**'
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# The type of runner that the job will run on
runs-on: ubuntu-latest
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
- name: Checkout the code
uses: actions/checkout#v3
- name: Install PMD
run: |
PMD_VERSION=`cat pmd/pmd-version.txt`
wget https://github.com/pmd/pmd/releases/download/pmd_releases%2F6.54.0/pmd-bin-6.54.0.zip
unzip pmd-bin-6.54.0.zip -d ~
mv ~/pmd-bin-$6.54.0 ~/pmd
~/pmd/bin/run.sh pmd --version
# Run PMD scandd
- name: Run PMD scan
run: ~/pmd/bin/run.sh pmd -d force-app -R pmd/ruleset.xml -f text
GitHub is showing me the below error:
You have an error in your yaml syntax on line 14
Note: the line 14 is "runs-on: ubuntu-latest"
Which is the syntax issue in the above YAML file?
You are missing the job identifier:
jobs:
foo: # <-- This
runs-on: ubuntu-latest
steps:
- name: Checkout the code
uses: actions/checkout#v3
steps:
You can use actionlint or vscode-yaml to avoid such syntax issues next time :)
I'm using GitHub actions to copy the artifact to the runner/VM, here I added my VM as a self-hosted runner and ran the workflow directly on the runner. I'm downloading the artifact from artifactory and copying it to the deployment location, Now, I Need to do the same thing on another runner/VM as I have an identical deployment VM for the application.
To achieve this, I have copied the same job and changed the 'runs-on' value with a different runner name which is my second VM. below is my workflow code snippet.
My question is, Instead of 2 jobs, how can we run using a single job for all VMs related to dev? let's say, Assume that I have 4 VMs for the Dev environment and 3 VMs for QA, and 5 VMs for Production
can someone help with this or do we need to continue with the same approach as whatever I'm doing right now?
I have tried matrix but looking to see if there is any other solution
name: Deployment_workflow
on:
workflow_dispatch:
inputs:
dev:
type: boolean
required: false
default: false
qa:
type: boolean
required: false
default: false
jobs:
dev_deploy_1:
if: github.event.inputs.dev =='true'
runs-on: dev-vm-1
steps:
- name: Download the artifact from artifactory
run: |
cd /tmp
curl -u ${ARTIFACTORY_USER}:${ARTIFACTORY_ENCRYPT} -T dep-deploy-1.war "$(ARTIFACTORY_URL)/artifactory/general-artifacts/dep-deploy-1.war"
- name: copy the file from /tmp to deployment location
run: |
cp /tmp/dep-deploy-1.war /var/lib/app_deploy_dir
dev_deploy_2:
if: github.event.inputs.dev =='true'
runs-on: dev-vm-2
steps:
- name: Download the artifact from artifactory
run: |
cd /tmp
curl -u ${ARTIFACTORY_USER}:${ARTIFACTORY_ENCRYPT} -T dep-deploy-2.war "$(ARTIFACTORY_URL)/artifactory/general-artifacts/dep-deploy-2.war"
- name: copy the file from /tmp to deployment location
run: |
cp /tmp/dep-deploy-1.bar /var/lib/app_deploy_dir
I know its not quite simple to do this and tried to explore many approaches but either I couldn't understand it properly or didn't work for me.
I have a concourse job which runs angular build (ng build) and creates /dist folder. This works well.
jobs:
- name: cache
plan:
- get: source
trigger: true
- get: npm-cache
- name: build
plan:
- get: source
trigger: true
passed: [cache]
- get: npm-cache
passed: [cache]
- task: run build
file: source/ci/build.yml
build.yml
---
platform: linux
image_resource:
type: docker-image
source: { repository: alexsuch/angular-cli, tag: '7.3' }
inputs:
- name: source
- name: npm-cache
path: /cache
outputs:
- name: artifact
run:
path: source/ci/build.sh
build.sh
#!/bin/sh
mv cache/node_modules source
cd source
npm rebuild node-saas # temporary fix
npm run build_prod
cp -R dist ../artifact/
I have mentioned output as artifact where I am storing the dist content.
But when I am trying to use this in next job, it doesn't work. Failed with missing input error.
Here is the next job that supposed to consume this dist folder:
jobs:
...
...
- name: list
plan:
- get: npm-cache
passed: [cache, test, build]
trigger: true
- task: list-files
config:
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: artifact
run:
path: ls
args: ['-la', 'artifact/']
Can anyone please help me with this. How I can use the dist folder in above job.
I'm not quite sure why would you want to have different plan definitions for each task but here is the simplest way of doing what you want to do:
jobs:
- name: deploying-my-app
plan:
- get: source
trigger: true
passed: []
- get: npm-cache
passed: []
- task: run build
file: source/ci/build.yml
- task: list-files
file: source/ci/list-files.yml
build.yml
---
platform: linux
image_resource:
type: docker-image
source: { repository: alexsuch/angular-cli, tag: '7.3' }
inputs:
- name: source
- name: npm-cache
path: /cache
outputs:
- name: artifact
run:
path: source/ci/build.sh
list-files.yml
---
platform: linux
image_resource:
type: registry-image
source: { repository: busybox }
inputs:
- name: artifact
run:
path: ls
args: ['-la', 'artifact/']
build.sh
#!/bin/sh
mv cache/node_modules source
cd source
npm rebuild node-saas # temporary fix
npm run build_prod
cp -R dist ../artifact/
Typically you would pass folders as inputs and outputs between TASKS instead of JOBS (althought there's some alternatives)
Concourse is statelessness and that is the idea behind it. But if you want to pass something between jobs the only way to do that is to use a concourse resource and depending on the nature of the project that could be anything from a git repo to a s3 bucket, docker image etc. You can create your own custom resources too.
Using something like s3 concourse resource for example
This way you can push your artifact to an external storage and then use it again on the next jobs on the get step as a resource. But that just may create some unnecesary complexity understanding that what you want to do is pretty straightforward
In my experience I found that sometimes the visual aspect of a job plan in the concourse dashboard gives the impression that a job-plan should by task atomic, which is not always needed
Hope that helps.
I have a concourse pipeline:
groups: []
resources:
- name: source-code
type: git
resource_types: []
jobs:
- name: build
public: true
plan:
- get: source-code
- task: build
privileged: true
config:
platform: linux
image_resource:
type: docker-image
source:
repository: java
tag: openjdk-8-alpine
run:
path: sh
args:
- -exc
- |
set -e -u -x
MVN_VERSION=$(cat pom.xml | grep "^ <version>.*</version>$" | awk -F'[><]' '{print $3}')
**CUSTOM_VERSION = [POM_FILE_VERSION]-build.[NO_OF_COMMITS_IN_BRANCH]**
inputs:
- name: source-code
Using Unix commands, I want to create the custom version:
CUSTOM_VERSION = [POM_FILE_VERSION]-build.[NO_OF_COMMITS_IN_BRANCH]
For [POM_FILE_VERSION], I have found the code:
MVN_VERSION=$(cat pom.xml | grep "^ <version>.*</version>$" | awk -F'[><]' '{print $3}')
But for [NO_OF_COMMITS_IN_BRANCH], i don't know how get that data from git resource. Can anyone help please ?
The Concourse git resource doesn't provide the number of commits in the branch.
I suggest to use something more meaningful like the commit hash, which the Concourse git resource makes available as the .git/short_ref file.
If you really need to use the number of commits (but why?), then you need to run the git command directly in the Concourse task, for example git rev-list --count HEAD.
In the pipeline above, the task is using the openjdk-8-alpine Docker image from the java repository. If that image contains git, then you are all set. If not, I suggest first to see if the java repository offers a variation of openjdk-8-alpine with git installed. If not, you have to create your own Docker image, based off openjdk-8-alpine.
Security note: you have privileged: true specified. Probably you don't need it.
When I try to use template variable, e.g. {{hostname}} as a part of value, it gets wrapped with double quotes.
How to add a variable w/o quotes?
Example:
---
resource_types:
- name: maven
type: docker-image
source:
repository: patrickcrocker/maven-resource
tag: latest
resources:
- name: maven-snapshot
type: maven
source:
url: http://{{hostname}}:8081/repository/maven-snapshots/
- name: repo
type: git
source:
uri: "git#bitbucket.org:foo/bar.git"
branch: master{{hostname}}
And the result for the command fly -t ci set-pipeline --pipeline test --config test.yml --var="hostname=localhost" is as follows (look at "localhost"):
resources:
resource maven-snapshot has been added:
name: maven-snapshot
type: maven
source:
url: http://"localhost":8081/repository/maven-snapshots/
resource repo has been added:
name: repo
type: git
source:
branch: master"localhost"
uri: git#bitbucket.org:foo/bar.git
resource types:
resource type maven has been added:
name: maven
type: docker-image
source:
repository: patrickcrocker/maven-resource
tag: latest
The reason I've included a 3rd-party maven resource is that git resource does not allow {{}} in the uri, leading to the error:
failed to unmarshal configStructure: yaml: line 17: did not find expected key
UPDATE
As of concourse v3.2.0 {{someValue}} syntax is deprecated in favor of ((someValue)). New syntax will understand you are trying to interpolate the string and place the value accordingly.
Replacing {{hostname}} with ((hostname)) will solve your issue:
resources:
- name: maven-snapshot
type: maven
source:
url: http://((hostname)):8081/repository/maven-snapshots/
Concourse does not support this.
The Concourse yaml templating is very primitive and you can not insert variables in the middle of strings.
You will need to set your url parameter as http://localhost:8081/repository/maven-snapshots/ and your branch parameter as localmaster or whatever it should be.
We know this is a problem and we are working on it, but for now you can not set variables in the way you want.
While waiting for this feature from concourse team, I've written this small executable to work around the issue in this GitHub repo:
https://github.com/sercant/inline-yaml
I prepare my config.yml like this:
ftp-username: username
ftp-password: password
ftp-uri: 192.168.1.2
ftp-dir: home/ftp/
ftp-uri-combined: ftp://{{ftp-username}}:{{ftp-password}}#{{ftp-uri}}/{{ftp-dir}}
ftp-uri-combined-html5: {{ftp-uri-combined}}html5
ftp-uri-combined-android: {{ftp-uri-combined}}android
And prepared a create-pipeline.sh:
#!/usr/bin/env sh
TEMP=$(mktemp)
java -jar inline-yaml.jar $3 ${TEMP};
fly -t lite set-pipeline -p $2 -c $1 --load-vars-from ${TEMP};
rm ${TEMP};
Whenever I need to create a pipeline, I ran:
./create-pipeline.sh build-plan.yml build-plan-name config.yml