I have a bunch of GitHub actions for services that each build a container if something changes in their corresponding directories. Then I have a final GitHub actions that sets up a cloud environment with Terraform and uses the containers created by the other actions to deploy the microservices and API for my project.
Currently I'm running that final action manually after all the other actions complete. However, I was wondering if there is any way to automate this. I realize I can chain actions but I'm unsure how to handle this since any number of actions might be running and might finish in any order.
You can turn each GitHub actions you have into a composite action for more reusability and readability.
But you can just combine all existing actions into 1 workflow - each in separate job if they need different runners.
Then you can set dependencies between those jobs them using needs syntax.
At the end when they all complete, you can have 1 job that has needs for each of those combined and check if they succeeded and based on that execute what you need.
As an example:
jobs:
job1:
job2:
needs: job1 //if you want to queue jobs then set dependency, if you want to run job1, job2 in parallel, remove this dependency
final_action:
needs: [job1, job2]
if: ${{ needs.job1.result == 'success' && needs.job2.result == 'success' }}
run: //PUBLISH STUFF
notify_error:
needs: [job1, job2]
if: ${{ always() && !cancelled() && (needs.job1.result != 'success' || needs.job2.result != 'success') }}
run: //NOTIFY FAILURE
In one workflow? You can run all actions in parallel in their own job. Then use needs: for the final job to depend on all the other jobs and deploy the application in that job.
Related
We use the "extends" templates feature of Azure yaml pipelines for security purposes. One requirement we have is to not allow the use of a specific task inside a job more than once and if it happens, validation just should fail. I know I can iterate through the job steps and look for the existence of a specific task but is there a way to check how many times a task appears inside a job?
I checked length function but doesn't meet my needs as I'm not able to filter the steps for a specific task.
An example of a check we do today:
${{ each step in parameters.steps }}:
${{ if contains(step.task, 'TaskA') }}:
'TaskA is not allowed etc.' : error
It depends on how you define the task in the yaml. It's recommended to check the definition, and avoid duplication of the tasks directly.
Or you could put the task in main yaml not the template, use sample script here to validate the task in the template, if it exists, fail the pipeline.
What's the difference between GITHUB_REPOSITORY and github.repository? both the value and usage in Github action.
Found the answer from GitHub official docs.
Determining when to use default environment variables or contexts
GitHub Actions includes a collection of variables called contexts and a similar collection of variables called default environment variables. These variables are intended for use at different points in the workflow:
Default environment variables: These variables exist only on the runner that is executing your job. For more information, see "Default environment variables."
Contexts: You can use most contexts at any point in your workflow, including when default environment variables would be unavailable. For example, you can use contexts with expressions to perform initial processing before the job is routed to a runner for execution; this allows you to use a context with the conditional if keyword to determine whether a step should run. Once the job is running, you can also retrieve context variables from the runner that is executing the job, such as runner.os. For details of where you can use various contexts within a workflow, see "Context availability."
The following example demonstrates how these different types of environment variables can be used together in a job:
name: CI
on: push
jobs:
prod-check:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: ubuntu-latest
steps:
- run: echo "Deploying to production server on branch $GITHUB_REF"
In this example, the if statement checks the github.ref context to determine the current branch name; if the name is refs/heads/main, then the subsequent steps are executed. The if check is processed by GitHub Actions, and the job is only sent to the runner if the result is true. Once the job is sent to the runner, the step is executed and refers to the $GITHUB_REF environment variable from the runner.
In Azure pipelines, I can conditionally run a specific task/job. However, if I have a Task group, I don't see any option to specify conditions on its execution.
The only way am able to do this is to create a new Job altogher which runs the task group and specify the conditinal expression in the Job.
This has drawback - Jobs are run on different agents. In this case, a new agent is created only to run that task group under a condition. This slows down the deployment
So - question is - how to conditinally execute a Task Group?
Thanks
According to the doc: Task conditions (such as "Run this task only when a previous task has failed" for a PowerShell Script task) can be configured in a task group and these settings are persisted with the task group.
If you want to custom task group level condition, It is not supported by azure devops for the time being.
I found a ticket on our UserVoice site , which is our main forum for product suggestions. you can vote and add your comments for this feedback. The product team would provide the updates if they view it.
Update1
As a workaround, We can define these tasks in the yaml template and then we can call different templates from a pipeline YAML depending on a condition.
Sample code:
parameters:
- name: experimentalTemplate
displayName: 'Use experimental build process?'
type: boolean
default: false
steps:
- ${{ if eq(parameters.experimentalTemplate, true) }}:
- template: experimental.yml
- ${{ if not(eq(parameters.experimentalTemplate, true)) }}:
- template: stable.yml
I have a workflow for CI in a monorepo, for this workflow two projects end up being built. The jobs run fine, however, I'm wondering if there is a way to remove the duplication in this workflow.yml file with the setting up of the runner for the job. I have them split so they run in parallel as they do not rely on one another and to be faster to complete. It's a big time difference in 5 minutes vs. 10+ when waiting for the CI to finish.
jobs:
job1:
name: PT.W Build
runs-on: macos-latest
steps:
- name: Checkout Repo
uses: actions/checkout#v1
- name: Setup SSH-Agent
uses: webfactory/ssh-agent#v0.2.0
with:
ssh-private-key: |
${{ secrets.SSH_PRIVATE_KEY }}
- name: Setup JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: Setup Permobil-Client
run: |
echo no | npm i -g nativescript
tns usage-reporting disable
tns error-reporting disable
npm run setup.all
- name: Build PT.W Android
run: |
cd apps/wear/pushtracker
tns build android --env.uglify
job2:
name: SD.W Build
runs-on: macos-latest
steps:
- name: Checkout Repo
uses: actions/checkout#v1
- name: Setup SSH-Agent
uses: webfactory/ssh-agent#v0.2.0
with:
ssh-private-key: |
${{ secrets.SSH_PRIVATE_KEY }}
- name: Setup JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: Setup Permobil-Client
run: |
echo no | npm i -g nativescript
tns usage-reporting disable
tns error-reporting disable
npm run setup.all
- name: Build SD.W Android
run: |
cd apps/wear/smartdrive
tns build android --env.uglify
You can see here the jobs have almost an identical process, it's just the building of the different apps themselves. I'm wondering if there is a way to take the duplicate blocks in the jobs and create a way to only write that once and reuse it in both jobs.
There are 3 main approaches for code reusing in GitHub Actions:
Reusable Workflows
Dispatched workflows
Composite Actions <-- it's the best one in your case
The following details are from my article describing their pros and cons:
🔸 Reusing workflows
The obvious option is using the "Reusable workflows" feature that allows you to extract some steps into a separate "reusable" workflow and call this workflow as a job in other workflows.
🥡 Takeaways:
Nested reusable workflow calls are allowed (up to 4 levels) while loops are not permitted.
Env variables are not inherited. Secrets can be inherited by using special secrets: inherit job param.
It's not convenient if you need to extract and reuse several steps inside one job.
Since it runs as a separate job, you have to use build artifacts to share files between a reusable workflow and your main workflow.
You can call a reusable workflow in synchronous or asynchronous manner (managing it by jobs ordering using needs keys).
A reusable workflow can define outputs that extract outputs/outcomes from executed steps. They can be easily used to pass data to the "main" workflow.
🔸 Dispatched workflows
Another possibility that GitHub gives us is workflow_dispatch event that can trigger a workflow run. Simply put, you can trigger a workflow manually or through GitHub API and provide its inputs.
There are actions available on the Marketplace which allow you to trigger a "dispatched" workflow as a step of "main" workflow.
Some of them also allow doing it in a synchronous manner (wait until dispatched workflow is finished). It is worth to say that this feature is implemented by polling statuses of repo workflows which is not very reliable, especially in a concurrent environment. Also, it is bounded by GitHub API usage limits and therefore has a delay in finding out a status of dispatched workflow.
🥡 Takeaways
You can have multiple nested calls, triggering a workflow from another triggered workflow. If done careless, can lead to an infinite loop.
You need a special token with "workflows" permission; your usual secrets.GITHUB_TOKEN doesn't allow you to dispatch a workflow.
You can trigger multiple dispatched workflows inside one job.
There is no easy way to get some data back from dispatched workflows to the main one.
Works better in "fire and forget" scenario. Waiting for a finish of dispatched workflow has some limitations.
You can observe dispatched workflows runs and cancel them manually.
🔸 Composite Actions
In this approach we extract steps to a distinct composite action, that can be located in the same or separate repository.
From your "main" workflow it looks like a usual action (a single step), but internally it consists of multiple steps each of which can call own actions.
🥡 Takeaways:
Supports nesting: each step of a composite action can use another composite action.
Bad visualisation of internal steps run: in the "main" workflow it's displayed as a usual step run. In raw logs you can find details of internal steps execution, but it doesn't look very friendly.
Shares environment variables with a parent job, but doesn't share secrets, which should be passed explicitly via inputs.
Supports inputs and outputs. Outputs are prepared from outputs/outcomes of internal steps and can be easily used to pass data from composite action to the "main" workflow.
A composite action runs inside the job of the "main" workflow. Since they share a common file system, there is no need to use build artifacts to transfer files from the composite action to the "main" workflow.
You can't use continue-on-error option inside a composite action.
As I know currently there is no way to reuse steps
but in this case, you can use strategy for parallel build and different variation:
jobs:
build:
name: Build
runs-on: macos-latest
strategy:
matrix:
build-dir: ['apps/wear/pushtracker', 'apps/wear/smartdrive']
steps:
- name: Checkout Repo
uses: actions/checkout#v1
- name: Setup SSH-Agent
uses: webfactory/ssh-agent#v0.2.0
with:
ssh-private-key: |
${{ secrets.SSH_PRIVATE_KEY }}
- name: Setup JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: Setup Permobil-Client
run: |
echo no | npm i -g nativescript
tns usage-reporting disable
tns error-reporting disable
npm run setup.all
- name: Build Android
run: |
cd ${{ matrix.build-dir }}
tns build android --env.uglify
For more information please visit https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategy
Since Oct. 2021, "Reusable workflows are generally available"
Reusable workflows are now generally available.
Reusable workflows help you reduce duplication by enabling you to reuse an entire workflow as if it were an action. A number of improvements have been made since the beta was released in October:
You can utilize outputs to pass data from reusable workflows to other jobs in the caller workflow
You can pass environment secrets to reusable workflows
The audit log includes information about which reusable workflows are used
See "Reusing workflows" for more.
A workflow that uses another workflow is referred to as a "caller" workflow.
The reusable workflow is a "called" workflow.
One caller workflow can use multiple called workflows.
Each called workflow is referenced in a single line.
The result is that the caller workflow file may contain just a few lines of YAML, but may perform a large number of tasks when it's run. When you reuse a workflow, the entire called workflow is used, just as if it was part of the caller workflow.
Example:
In the reusable workflow, use the inputs and secrets keywords to define inputs or secrets that will be passed from a caller workflow.
# .github/actions/my-action.yml
# Note the special trigger 'on: workflow_call:'
on:
workflow_call:
inputs:
username:
required: true
type: string
secrets:
envPAT:
required: true
Reference the input or secret in the reusable workflow.
jobs:
reusable_workflow_job:
runs-on: ubuntu-latest
environment: production
steps:
- uses: ./.github/actions/my-action
with:
username: ${{ inputs.username }}
token: ${{ secrets.envPAT }}
With ./.github/actions/my-action the name of the my-action.yml file in your own repository.
A reusable workflow does not have to be in the same repository, and can be in another public one.
Davide Benvegnù aka CoderDave illustrates that in "Avoid Duplication! GitHub Actions Reusable Workflows" where:
n3wt0n/ActionsTest/.github/workflows/reusableWorkflowsUser.yml references
n3wt0n/ReusableWorkflow/.github/workflows/buildAndPublishDockerImage.yml#main
I have a travis script deploying to different S3 buckets based on 2 conditions:
1. the branch name
2. the $TRAVIS_BRANCH env variable
... travis stuff
deploy:
- provider: s3
... other config
bucket: my-staging-bucket
on:
repo: MyOrg/my-repo
branch: staging
condition: $TRAVIS_BRANCH = staging
- provider: s3
... other config
bucket: my-prod-bucket
on:
repo: MyOrg/my-repo
branch: production
condition: $TRAVIS_BRANCH = production
It's working as expected:
When I deploy to staging, the first config successfully builds and deploys and I'm given appropriate messaging in Travis' job log.
It also tries to deploy to production and is stopped by the on: conditions, again providing messaging that indicates as much. The resulting log messages look like so, the first two lines indicating successful depoyment to staging and no deployment to production.
-Preparing deploy
-Deploying application
-Skipping a deployment with the s3 provider because a custom condition was not met
This is consistent when the situation is reversed:
-Skipping a deployment with the s3 provider because this branch is not permitted: production
-Skipping a deployment with the s3 provider because a custom condition was not met
...
-Preparing deploy
-Deploying application
This has lead to some confusion amonst the team as the messaging appears to be a false negative, indicating the deployment failed when it's actually functioning as intended. What I would like do is set up Travis so that it only runs the deployment script approprite for that branch and env variable combo.
Is there a way to do that? I was under the impression this was the method for conditional deployment.
If there's no way to prevent both deploy jobs from running, is there a way to at suppress the messaging in the job log?
The best way to do this would be to use Travis' stages and jobs features. Stages are groups of jobs. Jobs inside stages run in parallel. Stages run in sequence, one after the other. Entire stages can be conditional, and stages can also contain conditional jobs. Jobs in a stage can be deploy jobs too (i.e. the entire deploy: in your travis.yml can be nested inside a conditional stage. Most importantly for your goals, conditional stages and their included jobs are silently skipped if the condition is not met.
This is very different to the standard deploy: matrix that you already have. i.e. your current deploy step contains 2 deployments and so you get the message that it is skipping a deployment.
Instead, you can change that into separate deploy stages with conditional jobs.
The downside to using stages like this is that each stage runs in its own VM and so you can't share data from one stage to the next. (i.e build artifacts from previous stages do not propagate to subsequent stages). You can get around this by sharing the build results of a lengthy compile stage via S3, for example.
More information can be found here:
https://docs.travis-ci.com/user/build-stages
I have a working example here in my github: https://github.com/brianonn/travis-test
jobs:
include:
- stage: compile
script: bash scripts/compile.sh
- stage: test
script: bash scripts/test.sh
- stage: deploy-staging
if: branch = staging
name: "Deploy to staging S3"
script: skip
deploy:
provider: script
script: bash scripts/deploy.sh staging
on:
branch: staging
condition: $TRAVIS_BRANCH = staging
- stage: deploy-prod
if: branch = production
name: "Deploy to production S3"
script: skip
deploy:
provider: script
script: bash scripts/deploy.sh production
on:
branch: production
condition: $TRAVIS_BRANCH = production
This produces a Travis job log that is specific to each one of staging and production: