Concourse: Read git resource meta-data - concourse

I have a concourse pipeline:
groups: []
resources:
- name: source-code
type: git
resource_types: []
jobs:
- name: build
public: true
plan:
- get: source-code
- task: build
privileged: true
config:
platform: linux
image_resource:
type: docker-image
source:
repository: java
tag: openjdk-8-alpine
run:
path: sh
args:
- -exc
- |
set -e -u -x
MVN_VERSION=$(cat pom.xml | grep "^ <version>.*</version>$" | awk -F'[><]' '{print $3}')
**CUSTOM_VERSION = [POM_FILE_VERSION]-build.[NO_OF_COMMITS_IN_BRANCH]**
inputs:
- name: source-code
Using Unix commands, I want to create the custom version:
CUSTOM_VERSION = [POM_FILE_VERSION]-build.[NO_OF_COMMITS_IN_BRANCH]
For [POM_FILE_VERSION], I have found the code:
MVN_VERSION=$(cat pom.xml | grep "^ <version>.*</version>$" | awk -F'[><]' '{print $3}')
But for [NO_OF_COMMITS_IN_BRANCH], i don't know how get that data from git resource. Can anyone help please ?

The Concourse git resource doesn't provide the number of commits in the branch.
I suggest to use something more meaningful like the commit hash, which the Concourse git resource makes available as the .git/short_ref file.
If you really need to use the number of commits (but why?), then you need to run the git command directly in the Concourse task, for example git rev-list --count HEAD.
In the pipeline above, the task is using the openjdk-8-alpine Docker image from the java repository. If that image contains git, then you are all set. If not, I suggest first to see if the java repository offers a variation of openjdk-8-alpine with git installed. If not, you have to create your own Docker image, based off openjdk-8-alpine.
Security note: you have privileged: true specified. Probably you don't need it.

Related

Run GitHub Actions workflows Jobs on 2 runners parallelly at a time

I'm using GitHub actions to copy the artifact to the runner/VM, here I added my VM as a self-hosted runner and ran the workflow directly on the runner. I'm downloading the artifact from artifactory and copying it to the deployment location, Now, I Need to do the same thing on another runner/VM as I have an identical deployment VM for the application.
To achieve this, I have copied the same job and changed the 'runs-on' value with a different runner name which is my second VM. below is my workflow code snippet.
My question is, Instead of 2 jobs, how can we run using a single job for all VMs related to dev? let's say, Assume that I have 4 VMs for the Dev environment and 3 VMs for QA, and 5 VMs for Production
can someone help with this or do we need to continue with the same approach as whatever I'm doing right now?
I have tried matrix but looking to see if there is any other solution
name: Deployment_workflow
on:
workflow_dispatch:
inputs:
dev:
type: boolean
required: false
default: false
qa:
type: boolean
required: false
default: false
jobs:
dev_deploy_1:
if: github.event.inputs.dev =='true'
runs-on: dev-vm-1
steps:
- name: Download the artifact from artifactory
run: |
cd /tmp
curl -u ${ARTIFACTORY_USER}:${ARTIFACTORY_ENCRYPT} -T dep-deploy-1.war "$(ARTIFACTORY_URL)/artifactory/general-artifacts/dep-deploy-1.war"
- name: copy the file from /tmp to deployment location
run: |
cp /tmp/dep-deploy-1.war /var/lib/app_deploy_dir
dev_deploy_2:
if: github.event.inputs.dev =='true'
runs-on: dev-vm-2
steps:
- name: Download the artifact from artifactory
run: |
cd /tmp
curl -u ${ARTIFACTORY_USER}:${ARTIFACTORY_ENCRYPT} -T dep-deploy-2.war "$(ARTIFACTORY_URL)/artifactory/general-artifacts/dep-deploy-2.war"
- name: copy the file from /tmp to deployment location
run: |
cp /tmp/dep-deploy-1.bar /var/lib/app_deploy_dir

Updating GitHub issues from GitHub Actions

I was trying to make a GitHub action using some simple scripts (which I already use locally) that I would like to run inside a docker container.
A new issue should trigger the event to update the said issue with its content based on some processing. An example of this might be:
Say I have a list of labels defined in my script and it checks the issue's title and adds a label to the issue.
I'm still reading the GitHub Action's documentation so I may be not completely informed but the issue I seem to have is that in my local machine these scripts use gh cli for doing such tasks (eg. adding labels). So I was wondering if I need to have the gh installed in that docker container or is there a better way to update the issue? I'm very much willing to make these scripts from scratch again using the GitHub's event payloads and stuff as long as I don't have to write in TypeScript.
I've looked around the documentation and couldn't find anything that talked about updating issues. Also couldn't find a similar question being asked here; it may be that I've missed something so if that is the case direct me to relevant material and I would very much appreciate it.
An option could be (as you said) to install GH in that docker container, and then run GH commands.
Example using a container:
jobs:
build:
runs-on: ubuntu-latest
container:
image: docker://myrepoandimagewithghinstalled
steps:
- name: Github CLI Authentication
run: gh auth login --hostname <your hostname>
- name: Github CLI commands execution samples
run: |
gh command1
gh command2
gh command3
Another option could be to install GH directly on the OS (for exemple ubuntu-latest), authenticate, and then use the "run" option to execute GH command.
Example installing GH on the OS:
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Install Github CLI
run: |
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-key C99B11DEB97541F0
sudo apt-add-repository https://cli.github.com/packages
sudo apt update
sudo apt install gh
- name: Github CLI Authentication
run: gh auth login --hostname <your hostname>
- name: Github CLI commands execution samples
run: |
gh command1
gh command2
gh command3
Finally, you could also create a script consuming the Github API service to update an ISSUE and execute the script using the run option.
Example to execute a Python script in your workflow:
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: checkout repo content
uses: actions/checkout#v2 # checkout the repository content to github runner.
- name: setup python
uses: actions/setup-python#v2
with:
python-version: 3.8 #install the python needed
- name: execute py script # run the run.py to get the latest data
run: |
python run.py
env:
key: ${{ secrets.key }} # if run.py requires passwords..etc, set it as secrets
- name: export index
.... # use crosponding script or actions to help export.

Github actions only on first push

I'm working on a github action that creates an app on ArgoCD. The problem is that I want to execute it only once, the first time that it gets push with the k8s yamls.
Is there any way to restrict the github action to the first push on the repo?
I have been looking to the github triggers, but I was not able to find any relation.
This is a sample of the action:
on: push
name: deploy-argo-app
jobs:
deploy-argo-app:
name: Deploy new app on ArgoCD
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#master
- name: Install Argo Cli
run: |
VERSION=$(curl --silent "https://api.github.com/repos/argoproj/argo-cd/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')
sudo curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/download/$VERSION/argocd-linux-amd64
sudo chmod +x /usr/local/bin/argocd
- name: Create app
run: |
argocd app create guestbook --repo https://github.com/argoproj/argocd-example-apps.git --path guestbook --dest-server https://kubernetes.default.svc --dest-namespace default
I found kind of a workaround.
on: push
name: deploy
jobs:
deploy:
if: github.run_number == 1
Basically "github.run_number" gives you the push number. It will work only on the first push and then it will be ignore.
I have been looking for something similar, but I needed to know the first push on any given branch.
First, you can see everything available to you on your Github context by running this worfklow:
- name: Dump GitHub context
id: github_context_step
run: echo '${{ toJSON(github) }}'
I was able to see that on the first push to any branch the
github.event.before = 0000000000000000000000000000000000000000
and
github.event.created = true
whereas for any subsequent push to that same branch the github.event.before will give a numerical reference to the previous commit, and the github.event.created will be false.
Slightly related, but maybe still useful for people who find themselves on the same hunt I was on!

trigger concourse job via CLI: "resource not found"

I am attempting to trigger a concourse job from the command line. My pipeline has one resource (a git repo) and one job, which uses that repo. I am seeing:
$ fly -t tutorial trigger-job -j my-pipeline/my-job -w
error: resource not found
However, when I go the web UI and manually trigger the job by pressing the "+" button in the top right, it works fine.
Here is the full pipeline:
resources:
- name: cruise-source
type: git
source:
uri: git#github.com:my-org/cruise.git
branch: develop
jobs:
- name: build-image
public: true
plan:
- get: cruise-source
- task: list-files
config:
platform: linux
image_resource:
type: docker-image
source: {repository: alpine}
inputs:
- name: cruise-source
run:
path: ls
args: [cruise-source]
How can I trigger this job from the CLI?
The "resource not found" you get has nothing to do with the git resource :-) it actually means that the pipeline or job name is wrong. Looking at your pipeline configuration, you should issue
fly -t tutorial trigger-job -j my-pipeline/build-image -w
or if your configuration is different from what you have posted, maybe you have a typo in the name of the pipeline or job.

Can I make releases public from a private github repo?

I have an application which is in a private github repo, and am wondering if the releases could be made public so that the app can auto-update itself from github instead of us having to host it.
Additionally I'm wondering if it's possible to use the github api from the deployed app to check for new updates.
A workaround would be to create a public repo, composed of:
empty commits (git commit --allow-empty)
each commit tagged
each tag with a release
each release with the deliveries (the binaries of your private app)
That way, you have a visible repo dedicated for release hosting, and a private repos for source development.
As #VonC mentioned we have to create a second Repository for that. This is not prohibited and i am doing it already. With github workflows i automated this task, I'm using a develop / master branching, so always when I'm pushing anything to the master branch a new version is build and pushed to the public "Realease" Repo.
In my specific use case I'm building an android apk and releasing it via unoffical github api "hub". Some additional advantage of this is you can have an extra issue tracker for foreign issues and bugs.
name: Master CI CD
# using checkout#v2 instead of v1 caus it needs further configuration
on:
pull_request:
types: [closed]
jobs:
UnitTest:
runs-on: ubuntu-latest
if: github.event.pull_request.merged
steps:
- uses: actions/checkout#v2
- name: make executable
run: chmod +x gradlew
- name: Unit tests
run: |
./gradlew test
IncrementVersionCode:
needs: UnitTest
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: set up JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: make executable
run: chmod +x gradlew
- name: increment version
run: ./gradlew incrementVersionCode
- name: Push new version to master
run: |
git config --local user.email "workflow#bot.com"
git config --local user.name "WorkflowBot"
git commit -m "Increment Build version" -a
# maybe better amend commits to avoid bot commits
BuildArtifacts:
needs: IncrementVersionCode
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: set up JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: make executable
run: chmod +x gradlew
- name: Build with Gradle
run: ./gradlew build -x lint
- name: Rename artifacts
run: |
cp app/build/outputs/apk/release/app-release.apk MyApp.apk
- name: Upload Release
uses: actions/upload-artifact#master
with:
name: Release Apk
path: MyApp.apk
- name: Upload Debug
uses: actions/upload-artifact#master
with:
name: Debug Apk
path: app/build/outputs/apk/debug/app-debug.apk
# https://dev.to/ychescale9/running-android-emulators-on-ci-from-bitrise-io-to-github-actions-3j76
E2ETest:
needs: BuildArtifacts
runs-on: macos-latest
strategy:
matrix:
api-level: [21, 27]
arch: [x86]
steps:
- name: checkout
uses: actions/checkout#v2
- name: Make gradlew executable
run: chmod +x ./gradlew
- name: run tests
uses: reactivecircus/android-emulator-runner#v2
with:
api-level: ${{ matrix.api-level }}
arch: ${{ matrix.arch }}
script: ./gradlew connectedCheck
Deploy:
needs: E2ETest
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/master'
steps:
- uses: actions/checkout#v2 # Needed for gradle file to get version information
- name: Get Hub
run: |
curl -fsSL https://github.com/github/hub/raw/master/script/get | bash -s 2.14.1
cd bin
chmod +x hub
cd ..
- name: Get Apk
uses: actions/download-artifact#master
with:
name: Release Apk
- name: Publish
env:
GITHUB_TOKEN: "${{ secrets.RELEASE_REPO_SECRET }}"
run: |
APP_NAME=MyApp
VERSION_NAME=`grep -oP 'versionName "\K(.*?)(?=")' ./app/build.gradle`
VERSION_CODE=`cat version.properties | grep "VERSION_CODE" | cut -d'=' -f2`
FILENAME="${APP_NAME}-v${VERSION_NAME}-${VERSION_CODE}"
TAG="v${VERSION_NAME}-${VERSION_CODE}"
TAG="latest-master"
echo $APP_NAME
echo $VERSION_NAME
echo $VERSION_CODE
echo $FILENAME
echo $TAG
git clone https://github.com/MyUser/MyApp-Releases
cd MyApp-Releases
./../bin/hub release delete "${TAG}" || echo "Failed deleting TAG: ${TAG}" # If release got lost catch error with message
./../bin/hub release create -a "../${APP_NAME}.apk" -m "Current Master Build: ${FILENAME}" -p "${TAG}"
EvaluateCode:
needs: Deploy
runs-on: ubuntu-latest
steps:
- name: Get Hub
run: |
echo "TDOO: Run Jacoco for coverage, and other profiling tools"
The 2022 answer to this question is even more straight-forward.
You'd just need to use the pre-installed gh CLI:
gh release create v0.0.1 foobar.zip -R https://github.com/your/repo-here
This command will create a tag v0.0.1 and a release with the local file foobar.zip attached on the public repository. You can run this in the GitHub Action of any private repository.
The -R argument points to the repository you'd like to create a tag/release on. foobar.zip would be located in your local directory.
One thing is important here: GITHUB_TOKEN must still be set as the token of the repository you'd like to release on!
Full example:
- name: Publish
env:
GITHUB_TOKEN: "${{ secrets.RELEASE_REPO_SECRET }}"
run: |
gh release create v0.0.1 foobar.zip -R https://github.com/your/repo-here
If you're planning to re-release and override existing versions, there is gh release delete as well. The -d flag creates a release as a draft etc. pp. Please take a look at the docs.
I'm using a slightly more advanced approach by setting:
shell: bash
run: $GITHUB_ACTION_PATH/scripts/publish.sh
And in file scripts/publish.sh:
#!/usr/bin/env node
const cp = require('child_process')
const fs = require('fs');
const path = require('path');
const APP_VERSION = JSON.parse(fs.readFileSync('package.json', { encoding: 'utf8' })).version
const TAG = `v${APP_VERSION}`
cp.execSync(`gh release create ${TAG} foobar.zip -R https://github.com/your/repo-name`, { stdio: 'inherit' })
This approach enables you to be able to for example, use Node.js or any other programming language available, to extract a version from the project management file of choice (e.g. package.json) and automatically come up with the right tag version and name.
A simple way to duplicate releases from a private repo to a public one may be this Release & Assets Github Action which can: Create a release, upload release assets, and duplicate a release to other repository.
Then you can use the regular electron-builder updater's support for public repos.