github actions: Push several times, but I want the build to be done only for the last push - github

name: master builder
on:
push:
branches:
- master
~~~
I have a workflow like this. So, Whenever I push to the master branch, the actions run.
But I want the build to work only on the last push.
For example,
master branch - feature1 (person1)
master branch - feature2 (person2)
master branch - feature3 (person3)
In this structure, if features1,2,3 are merged at almost the same time, the build will run 3 times.
But I want the master branch to be built only on the last merge. Just once.
Is there anyway to do this? like.. run the build only once after waiting for about 1 minute when pushing.
This is a sample code that I proceeded in the way you answered. But I get an error "The key 'concurrency' is not allowed". What's wrong?
name: test
on:
push:
branches:
- feature/**
concurrency:
group: ${{ github.ref }}
cancel-in-progress: true
jobs:
~~~

You may try to achieve this with concurrency and cancel-in-progress: true
Concurrency ensures that only a single job or workflow using the same concurrency group will run at a time. A concurrency group can be any string or expression. The expression can only use the github context. For more information about expressions, see "Context and expression syntax for GitHub Actions."
You can also specify concurrency at the job level. For more information, see jobs.<job_id>.concurrency.
When a concurrent job or workflow is queued, if another job or workflow using the same concurrency group in the repository is in progress, the queued job or workflow will be pending. Any previously pending job or workflow in the concurrency group will be canceled. To also cancel any currently running job or workflow in the same concurrency group, specify cancel-in-progress: true.
However
Note: Concurrency is currently in beta and subject to change.
Here is an example workflow:
name: Deploy
on:
push:
branches:
- main
- production
paths-ignore:
- '**.md'
# Ensures that only one deploy task per branch/environment will run at a time.
concurrency:
group: environment-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Extract commit
run: |
echo "Sending commit $GITHUB_SHA for $GITHUB_REPOSITORY"

Related

Pipeline resource triggers doesn't consider branch

I've two Azure DevOps CI pipelines
DataPipeline\Windows - Build
DataPipeline\TestPipeline
The second pipeline is a pipeline resource trigger. I've used this doc as a reference.
When a pull request is created (e.g. with dev branch as source branch and main branch as target branch, not merged yet) both these pipelines are triggered (by default configuration). The second pipeline may pick up old artefacts (this is a different part altogether). A second run for pipeline DataPipeline\TestPipeline is triggered automatically as soon as DataPipeline\Windows - Build completes for a commit in the dev branch. Is this possible? I've specified the branch name in the resources trigger. Is something wrong with the YAML configuration?
My expected behaviour is when a pull request is created for branch name: xyz (not main), DataPipeline\TestPipeline shouldn't be triggered again after DataPipeline\Windows - Build completes.
DataPipeline\TestPipeline.yml
trigger: none
resources:
pipelines:
- pipeline: DataPipeline
source: 'DataPipeline\Windows - Build'
branch: main
trigger:
branches:
include:
- main
name: TestPipeline_$(variable)
jobs:
// template based job

How to run jobs based on the pull request branch

I have 2 jobs in my workflow 'Plan and Apply". I want plan to run when there is pull request on my testing branch and Apply to run when there is a pull request on master branch. Below is the snippet of my code. This workflow doesn't run,I am getting message " This check was skipped" .What I'm i doing wrong?
on:
pull_request:
branches:
- testing
- master
jobs:
plan:
name: "Terraform Plan"
if: ${{ github.head_ref == 'testing'}}
Apply:
name: "Run Terraform Apply"
if: ${{ github.head_ref == 'master'}}
I'd recommend to instead of putting it all into one workflow file, to create two separate ones, where you then only specify the branch you want, for example:
`plan_testing.yml`
on:
pull_request:
branches:
- testing
jobs:
plan:
name: "Terraform Plan"
This achieves the same outcome while keeping them more easily separated and simpler to audit their run history in GitHub Actions.
You can find out more about GitHub Actions and how to configure them by checking out their awesome docs.

How to trigger GitHub Action workflow based on several workflows

Suppose i have three workflows: build_backend, build_frontend and deploy. First two should trigger in parallel, but the third should only trigger when both of those workflows are finished.
Currently the deploy workflow triggers twice -- i suspect that's for each of the two workflows completed.
# .github/workflows/build-xxx.yml
name: Build and Test - Backend
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
# ...
# .github/workflows/deploy.yml
name: Deploy
on:
workflow_run:
workflows:
- "Build and Test - Backend"
- "Build and Test - Frontend"
types:
- completed
branch: master
jobs:
deploy:
# ...
I haven't found the solution in docs:
https://docs.github.com/en/actions/reference/events-that-trigger-workflows#workflow_run
https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions
GitHub actions does not support specific trigger definitions like this for whole workflows.
However, you can use the needs keyword on a job level. So if you could consolidate all of these workflows into one workflow file. It seems like like this could work for you since these workflows all have the same (branch) trigger and the buildxxx are only a single job each.
There is also a GitHub Roadmap item describing, that they are working on adding workflow partials. That would enable you to separte these parts out in the future if you want to, but that is not avilable yet it seems like.

Azure DevOps - Pull Request Workflow and Deployments

I'm a bit confused with how to set this workflow up using pull requests.
I have in place an existing multi-stage YAML build pipeline that in summary does the following:
Runs a build from any branch
Runs a deployment job to a Development Environment/Resource, if the source branch is feature/* or release/* and the build succeeds
Runs a deployment job to a UAT Environment/Resource, if the source branch is release/*
Runs a deployment job to Live/Production Environment/Resource, if the source branch is master
So off the back of CI this workflow seems to work fine, the correct stages are run depending on the branch etc.
I then decided that branch policies and pull requests might be a better option for code quality rather than let any of the main branches be direct committed to, so I started to rework the YAML to account - mainly by removing the trigger to
trigger: none
This now works correctly in line with a branch policy, the build only gets kicked off when a pull request on to develop or master is opened.
However this is then where I'm a bit confused with how this is supposed to work and how I think it works ....
Firstly - is it not possible to trigger the multi-stage YAML off the back of pull requests (using Azure Repos) ? In my head all I want to do is introduce the pull request and branch policies but keep the multi-stage deployments to environments as is.
However, the deployment job stages all get skipped now - but this might be to do with my conditions within the YAML, which are as follows:
- stage: 'Dev'
displayName: 'Development Deployment'
dependsOn: 'Build'
condition: |
and
(
succeeded()
eq(variables['Build.Reason'], 'PullRequest'),
ne(variables['System.PullRequest.PullRequestId'], 'Null'),
or
(
startsWith(variables['Build.SourceBranch'], 'refs/heads/feature/'),
startsWith(variables['Build.SourceBranch'], 'refs/heads/release/')
)
)
jobs:
- deployment: Deploy
pool:
name: 'Development Server Agent Pool'
variables:
Parameters.WebsitePhysicalPath: '%SystemDrive%\inetpub\wwwroot\App'
Parameters.VirtualPathForApplication: ''
Parameters.VirtualApplication: ''
environment: 'Development.Resource-Name'
.....
Is there something I am missing?
Or do I have to remove the multi-stage deployments from YAML and revert back to using Release Pipelines for pull requests (maybe with approval gates??)
Thanks in advance!
It looks like the issue of the conditions within the YAML.
If the pipeline is triggered by a PR. the value variables['Build.SourceBranch'] will be refs/pull/<PR id>/merge. The express in above condtion startsWith(variables['Build.SourceBranch'], 'refs/heads/feature/') will be false, which caused the stage to be skipped. See build variables for more information.
You can try using variables['System.PullRequest.SourceBranch'], which will be evaluated to the value of the source Branch of the PR. Check System variables for more information. See below:
condition: |
and
(
succeeded(),
eq(variables['Build.Reason'], 'PullRequest'),
ne(variables['System.PullRequest.PullRequestId'], ''),
or
(
startsWith(variables['System.PullRequest.SourceBranch'], 'refs/heads/feature/'),
startsWith(variables['System.PullRequest.SourceBranch'], 'refs/heads/release/')
)
)

Reuse portion of github action across jobs

I have a workflow for CI in a monorepo, for this workflow two projects end up being built. The jobs run fine, however, I'm wondering if there is a way to remove the duplication in this workflow.yml file with the setting up of the runner for the job. I have them split so they run in parallel as they do not rely on one another and to be faster to complete. It's a big time difference in 5 minutes vs. 10+ when waiting for the CI to finish.
jobs:
job1:
name: PT.W Build
runs-on: macos-latest
steps:
- name: Checkout Repo
uses: actions/checkout#v1
- name: Setup SSH-Agent
uses: webfactory/ssh-agent#v0.2.0
with:
ssh-private-key: |
${{ secrets.SSH_PRIVATE_KEY }}
- name: Setup JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: Setup Permobil-Client
run: |
echo no | npm i -g nativescript
tns usage-reporting disable
tns error-reporting disable
npm run setup.all
- name: Build PT.W Android
run: |
cd apps/wear/pushtracker
tns build android --env.uglify
job2:
name: SD.W Build
runs-on: macos-latest
steps:
- name: Checkout Repo
uses: actions/checkout#v1
- name: Setup SSH-Agent
uses: webfactory/ssh-agent#v0.2.0
with:
ssh-private-key: |
${{ secrets.SSH_PRIVATE_KEY }}
- name: Setup JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: Setup Permobil-Client
run: |
echo no | npm i -g nativescript
tns usage-reporting disable
tns error-reporting disable
npm run setup.all
- name: Build SD.W Android
run: |
cd apps/wear/smartdrive
tns build android --env.uglify
You can see here the jobs have almost an identical process, it's just the building of the different apps themselves. I'm wondering if there is a way to take the duplicate blocks in the jobs and create a way to only write that once and reuse it in both jobs.
There are 3 main approaches for code reusing in GitHub Actions:
Reusable Workflows
Dispatched workflows
Composite Actions <-- it's the best one in your case
The following details are from my article describing their pros and cons:
🔸 Reusing workflows
The obvious option is using the "Reusable workflows" feature that allows you to extract some steps into a separate "reusable" workflow and call this workflow as a job in other workflows.
🥡 Takeaways:
Nested reusable workflow calls are allowed (up to 4 levels) while loops are not permitted.
Env variables are not inherited. Secrets can be inherited by using special secrets: inherit job param.
It's not convenient if you need to extract and reuse several steps inside one job.
Since it runs as a separate job, you have to use build artifacts to share files between a reusable workflow and your main workflow.
You can call a reusable workflow in synchronous or asynchronous manner (managing it by jobs ordering using needs keys).
A reusable workflow can define outputs that extract outputs/outcomes from executed steps. They can be easily used to pass data to the "main" workflow.
🔸 Dispatched workflows
Another possibility that GitHub gives us is workflow_dispatch event that can trigger a workflow run. Simply put, you can trigger a workflow manually or through GitHub API and provide its inputs.
There are actions available on the Marketplace which allow you to trigger a "dispatched" workflow as a step of "main" workflow.
Some of them also allow doing it in a synchronous manner (wait until dispatched workflow is finished). It is worth to say that this feature is implemented by polling statuses of repo workflows which is not very reliable, especially in a concurrent environment. Also, it is bounded by GitHub API usage limits and therefore has a delay in finding out a status of dispatched workflow.
🥡 Takeaways
You can have multiple nested calls, triggering a workflow from another triggered workflow. If done careless, can lead to an infinite loop.
You need a special token with "workflows" permission; your usual secrets.GITHUB_TOKEN doesn't allow you to dispatch a workflow.
You can trigger multiple dispatched workflows inside one job.
There is no easy way to get some data back from dispatched workflows to the main one.
Works better in "fire and forget" scenario. Waiting for a finish of dispatched workflow has some limitations.
You can observe dispatched workflows runs and cancel them manually.
🔸 Composite Actions
In this approach we extract steps to a distinct composite action, that can be located in the same or separate repository.
From your "main" workflow it looks like a usual action (a single step), but internally it consists of multiple steps each of which can call own actions.
🥡 Takeaways:
Supports nesting: each step of a composite action can use another composite action.
Bad visualisation of internal steps run: in the "main" workflow it's displayed as a usual step run. In raw logs you can find details of internal steps execution, but it doesn't look very friendly.
Shares environment variables with a parent job, but doesn't share secrets, which should be passed explicitly via inputs.
Supports inputs and outputs. Outputs are prepared from outputs/outcomes of internal steps and can be easily used to pass data from composite action to the "main" workflow.
A composite action runs inside the job of the "main" workflow. Since they share a common file system, there is no need to use build artifacts to transfer files from the composite action to the "main" workflow.
You can't use continue-on-error option inside a composite action.
As I know currently there is no way to reuse steps
but in this case, you can use strategy for parallel build and different variation:
jobs:
build:
name: Build
runs-on: macos-latest
strategy:
matrix:
build-dir: ['apps/wear/pushtracker', 'apps/wear/smartdrive']
steps:
- name: Checkout Repo
uses: actions/checkout#v1
- name: Setup SSH-Agent
uses: webfactory/ssh-agent#v0.2.0
with:
ssh-private-key: |
${{ secrets.SSH_PRIVATE_KEY }}
- name: Setup JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: Setup Permobil-Client
run: |
echo no | npm i -g nativescript
tns usage-reporting disable
tns error-reporting disable
npm run setup.all
- name: Build Android
run: |
cd ${{ matrix.build-dir }}
tns build android --env.uglify
For more information please visit https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategy
Since Oct. 2021, "Reusable workflows are generally available"
Reusable workflows are now generally available.
Reusable workflows help you reduce duplication by enabling you to reuse an entire workflow as if it were an action. A number of improvements have been made since the beta was released in October:
You can utilize outputs to pass data from reusable workflows to other jobs in the caller workflow
You can pass environment secrets to reusable workflows
The audit log includes information about which reusable workflows are used
See "Reusing workflows" for more.
A workflow that uses another workflow is referred to as a "caller" workflow.
The reusable workflow is a "called" workflow.
One caller workflow can use multiple called workflows.
Each called workflow is referenced in a single line.
The result is that the caller workflow file may contain just a few lines of YAML, but may perform a large number of tasks when it's run. When you reuse a workflow, the entire called workflow is used, just as if it was part of the caller workflow.
Example:
In the reusable workflow, use the inputs and secrets keywords to define inputs or secrets that will be passed from a caller workflow.
# .github/actions/my-action.yml
# Note the special trigger 'on: workflow_call:'
on:
workflow_call:
inputs:
username:
required: true
type: string
secrets:
envPAT:
required: true
Reference the input or secret in the reusable workflow.
jobs:
reusable_workflow_job:
runs-on: ubuntu-latest
environment: production
steps:
- uses: ./.github/actions/my-action
with:
username: ${{ inputs.username }}
token: ${{ secrets.envPAT }}
With ./.github/actions/my-action the name of the my-action.yml file in your own repository.
A reusable workflow does not have to be in the same repository, and can be in another public one.
Davide Benvegnù aka CoderDave illustrates that in "Avoid Duplication! GitHub Actions Reusable Workflows" where:
n3wt0n/ActionsTest/.github/workflows/reusableWorkflowsUser.yml references
n3wt0n/ReusableWorkflow/.github/workflows/buildAndPublishDockerImage.yml#main