Can't make use of workflow environment variable in github action from marketplace (through build matrix) - mongodb

I'm trying to make use of a workflow environment variable in a marketplace action, using a build matrix but it's not working for some reason.
I basically want to define the database versions just once to avoid repeating them in multiple place in my workflow.
Here's my workflow (minimal reproducible example):
name: dummy
on:
pull_request:
env:
MONGODB_3_6: 3.6.13
MONGODB_4_0: 4.0.13
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
MONGODB: [$MONGODB_4_0, $MONGODB_3_6]
steps:
- uses: actions/checkout#v2
- name: Start MongoDB
uses: supercharge/mongodb-github-action#1.3.0
with:
mongodb-version: ${{ matrix.MONGODB }}
And it's failing with the error below, as if the MONGODB_4_0 wasn't defined.
Interesting fact, without the strategy matrix, I'm able to make it work using the env context(doc):
- name: Start MongoDB
uses: supercharge/mongodb-github-action#1.3.0
with:
mongodb-version: ${{ env.MONGODB_4_0 }}

UPDATED: according tests and comments, I think matrix can't take environment variables and/or dynamic values.
so the best way will be :
matrix:
MONGODB: [3.6.13, 4.0.13]
As #max said you can use a variable for your workflow, so I guess your matrix should be wrong, maybe you can try like that :
MONGODB: [${{ env.MONGODB_4_0 }}, ${{ env.MONGODB_3_6 }}]
You have only one job (test) so you can define your env variables at the job level also.
Variables will be accessible for all the job :
jobs:
test:
runs-on: ubuntu-latest
env:
MONGODB_3_6: 3.6.13
MONGODB_4_0: 4.0.13
For further information : github doc

Related

Invalid workflow file Github Actions (CF CLI)

I'm trying to get this github action to work but once committed it gives me this error:
Invalid workflow file: .github/workflows/main.yml#L1
No steps defined in steps and no workflow called in uses for the following jobs: build
Anyone have any idea what this might depend on?
Below is the code I used:
name: Deploy to Cloud Foundry
on:
push:
branches:
- master
jobs:
build:
runs-on: ubuntu-18.04
# Build your app here
deploy:
runs-on: ubuntu-18.04
needs: build
steps:
- uses: citizen-of-planet-earth/cf-cli-action#master
with:
cf_api: https://api.my-cloud-foundry.com
cf_username: ${{ secrets.CF_USER }}
cf_password: ${{ secrets.CF_PASSWORD }}
cf_org: AwesomeApp
cf_space: Development
command: push -f manifest-dev.yml
Thanks in advance to everyone
Following the Workflow Syntax for Github Actions, you'll identify that some fields are mandatory.
At the workflow level, you need to have at least a trigger (configure with the on field) and a list of jobs specified.
Then, in that list of jobs, you have to specify at least one job, where each of those jobs needs at least the runner and the steps (or the uses for reusable workflow) field configured.
Example of the minimum configurations you would use for a job:
on: push
jobs:
job1:
runs-on: ubuntu-latest
steps:
- name: Print a greeting
run: echo 'Hello World'
job2: # reusable workflow scenario
uses: owner/repo/.github/workflows/reusable-workflow.yml#main

How to optimise github actions in a simple setup-lint-test-e2e-build scenario?

If we were to use minimal resources, doing these steps would look something like:
setup --- checkout, install fixed versions via cache
then trigger in parallel 1 command to run:
lint
test
e2e
all in parallel.
Unfortunately, with the current options I've found out there, the best scenario I can find is to have a separate workflow file for lint, test, e2e that are called by a main workflow and upon success build an artifact.
The issue with this is.
checkout will run 3 times.
install command will run 3 times.
Which can be mitigated with caching, but you're still looking at 10-15 seconds extra per job for those steps.
Is there a way to optimise this and still keep the reusable steps?
Even if I would keep everything in 1 giant file, I'd still define the lint, test, e2e steps as separate jobs and not as 1 single step part of a job.
Can we call a 'workflow' and run it in the same environment on the spot somehow?
This example calls checkout and caches the repo using the $ {{ github.run_id }} to prevent the cache conflicting with other runs. The cache is then used by the child workflows in place of checking out. e2e, test and lint will wait for the cache to be created and then all run in parallel.
Depending on your use case it may be excessive to create a new cache for each run_id, you can adjust this as you need. If speed is what you're after I'd also suggest only caching the files that you need for your child workflows, and not take everything like I've done in the example below.
name: main_caller
on: workflow_dispatch
jobs:
cache_checkout:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- uses: actions/upload-artifact#v3
with:
name: my-checkout-${{ github.run_id }}
path: ${{ github.workspace }}/**/*
run_lint:
needs: cache_checkout
uses: ./.github/workflows/lint.yml
run_test:
needs: cache_checkout
uses: ./.github/workflows/test.yml
./.github/workflows/lint.yml example: (e2e and test will follow this pattern)
name: lint
on: workflow_call
jobs:
lint:
runs-on: ubuntu-latest
steps:
- name: Get Cached Checkout
uses: actions/download-artifact#v3
with:
name: my-checkout-${{ github.run_id }}
- name: Do linting things
run: echo "I'm linting!"
Edited as per comments - all jobs in GitHub run in parallel, but to have cleaner example you could use matrix strategy. To make it work you need it to depend on a 'setup' job that will install everything and create the artifacts (this is done using the needs keyword).
In the following example the install job would setup everything and then the example job, which uses matrix strategy, will run the lint/test/e2e flow in separate parallel jobs after it downloads the required artifacts.
jobs:
install:
runs-on: ubuntu-latest
- uses: actions/checkout#v3
- name: Compile artifacts
run: *install fixed versions/create artifacts*
- uses: actions/upload-artifact#v3
with:
name: my-artifact
path: path/to/my_artifact
example:
runs-on: ubuntu-latest
needs: install
strategy:
matrix:
action: [lint, test, e2e]
steps:
- uses: actions/download-artifact#v3
with:
name: my-artifact
path: path/to/artifact
- name: lint
if: ${{ matrix.lint }}
run: echo "lint"
- name: test
if: ${{ matrix.test }}
run: echo "test"
- name: e2e
if: ${{ matrix.e2e }}
run: echo "e2e"
It will create a workflow like this:

Github Actions Conditional Trigger

I need help figuring something, I am trying to trigger 2 different workflows based on 2 different release tags. I want prod-* to trigger the production workflow and dev-* for the development workflow.
The problem is both tags trigger both workflows and I have no idea how to fix this
(I've canceled both actions but they triggered as you can see)
The tags element is not valid for release events. In consequence, the workflow is triggered for every release event of type published no matter the tag. There is no direct filter for tags with the release event as there is for push and pull_request events.
So you can leverage the if conditional on jobs in combination with the github.ref in the context which contains the tag of the release.
name: Deploy
on:
release:
types: [published]
jobs:
deploy-dev:
name: Deploy to development
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/dev-')
steps:
# [...]
deploy-prod:
name: Deploy to production
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/prod-')
steps:
# [...]
The published trigger is used in both cases here (independently of the tag), because your workflow will start for published OR tags (with the informed pattern) events.
To perform an operation only for a specific tag, you would have to extract the tag version from the $GITHUB_REF (env-var), for example using a step as below with an output in a first job:
- name: Get the version
id: get_tag_version
run: echo ::set-output name=version::${GITHUB_REF/refs\/tags\//}
And then use an if condition on 2 others jobs to check if the tag version contains prod- or dev- (needing the first job) to perform the operation you want for each scenario.
Here is an complete example of what could be used:
name: Example
on:
release:
types: [published]
jobs:
job1:
runs-on: ubuntu-latest
outputs:
tag_version: ${{ steps.get_tag_version.outputs.version }}
steps:
- name: Get the version
id: get_tag_version
run: echo ::set-output name=version::${GITHUB_REF/refs\/tags\//}
job2: # will be executed on for dev- tag
runs-on: ubuntu-latest
needs: [job1]
if: contains( needs.job1.outputs.tag_version , 'dev-')
steps:
[...]
job3: # will be executed on for prod- tag
runs-on: ubuntu-latest
needs: [job1]
if: contains( needs.job1.outputs.tag_version , 'prod-')
steps:
[...]
I coded a workflow to test the implementation above and it worked as expected creating a prod-2 release tag:
workflow file implementation
workflow run
EDIT: Note that you could also use a startWith instead of a contains function for the if expression.

Nested templates (calling a yaml file from another yaml file) in GitHub Actions

Does GitHub action support nested templates? For example, here is an example of Azure Pipeline yaml where it calls another yaml file:
- job: BuildFunctions
steps:
- ${{ each func in parameters.functionApps }}:
- template: yaml/build-functionapps.yml
parameters:
Is it possible to call a yaml file from another yaml file in GitHub actions?
You can use composite run steps actions. These are actions that are solely defined in YAML (documentation).
You can now specify containers, other composite actions (up to a depth of 9) and node actions in additional to the previously available run steps
node actions likely refers to leaf actions, i.e. actions that don't call any other actions.
Source: https://github.com/actions/runner/issues/646#issuecomment-901336347
Workflow
[...]
jobs:
job:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: ./.github/workflows/composite-action
[...]
Composite run steps action
.github/workflows/composite-action/action.yml (same repository as the workflow)
name: "My composite action"
description: "Checks out the repository and does something"
runs:
using: "composite"
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v2
with:
node-version: 12
- run: npm test
shell: bash
- run: |
echo "Executing action"
shell: bash
Old limitations:
What does composite run steps currently support?
For each run step in a composite action, we support:
name
id
run
env
shell
working-directory
In addition, we support mapping input and outputs throughout the action.
See docs for more info.
What does Composite Run Steps Not Support
We don't support setting conditionals, continue-on-error, timeout-minutes, "uses" [remark: i.e. using other actions], and secrets on individual steps within a composite action right now.
(Note: we do support these attributes being set in workflows for a step that uses a composite run steps action)
Source: https://github.com/actions/runner/issues/646
I think using the composite action pattern, you can achieve what you want.
You need to define the steps which you think will be reused in other places, and make it parameterized, by providing inputs. In my opinion, it's more powerful than how templates work in gitlab or in other similar platforms.
This way, you are defining a function, which can take inputs, and get stuff done for you, based on those inputs.
Also, even though the docs suggest that, you should create your leaf action as a separate public repo, and use it in your base action- it's not necessary, you can simply have a structure like below(taken the example from one of our live workflow), and use those leaf actions in your workflow-
.github
- actions
- deploy-to-k8s
- action.yaml
- publish-image
- action.yaml
- workflows
- deploy-from-pr.yaml <-- this will make use of all the actions defined
Here's how the deploy-from-pr.yaml workflow looks like-
name: deploy-from-pr
on:
pull_request:
branches:
- master
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
deploy-from-pr:
name: Deploy from PR to Development
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Set version
id: release
run: echo ::set-output name=version::$(git describe --always)
# custom action to build and push image
- name: Build & publish image
uses: ./.github/actions/publish-image # see how it's referred from same repo directly
with:
registry: ${{ env.REGISTRY }}
registry_username: ${{ github.REGISTRY_USERNAME }}
registry_password: ${{ secrets.REGISTRY_PASSWORD }}
image_name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tag: ${{ steps.release.outputs.version }}
# custom action to deploy into kubernetes
- name: Deploy to kubernetes
uses: ./.github/actions/deploy-to-k8s # see how it's referred from same repo directly
with:
digitalocean_token: ${{ secrets.DIGITALOCEAN_TOKEN }}
cluster_name: ${{ secrets.CLUSTER_NAME }}
overlay_path: .k8s/overlays/dev
image_name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tag: ${{ steps.release.outputs.version }}
Github Gist
You can check the deploy-to-k8s/action.yaml, to see how it's written.
No, it is not. I asked the exact same question in the GitHub Forum:
Is it possible to create / publish Actions without Docker or JS by having the code in the Workflow Syntax / YML?
As mentioned in the document: Currently, types of actions only lists
Docker container and JavaScript, so there is no such feature to
achieve your requirement.
Source: https://github.community/t/how-to-create-ready-to-use-action-without-docker-js/124889/2
This would have eased creating templates for users as system administrator.
You can also use reusable workflows.

switch environment variable in github actions

I use github actions for integration tests.
The problem is, that the tests should not run on multiple instances with the same configuration in parallel (the test would fail).
But, it can be run once with let's say configuration 1 and once with configuration 2 in parallel.
As this blog post describes, it is not possible to secure that a workflow does not run in parallel.
Is there any way to switch configurations, that configuration 1 and configuration 2 alternately?
In that case, it would not be that likely that the workflow workflows with the same configuration runs in parallel (I could add more configurations if needed).
For example, this could be done by a global and writable (for the workflow) variable that is alternately 1 or 2 and the workflow picks that configuration.
Example workflow(the secret confToSwitch should be switched):
name: test
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: Set up JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: "load configuration"
run: echo "configuration=$conf" >> ./conf
env:
conf: ${{ secrets.confToSwitch }}
- name: "integration tests"
run: "mvn -B integration-test"
You can try a matrix configuration with:
name: test
on: [push]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
token: [token1, token2, etc...]
steps:
- uses: actions/checkout#v1
- name: Set up JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: "load configuration"
run: echo "configuration=$conf" >> ./conf
env:
conf: ${{ matrix.token }}
- name: "integration tests"
run: "mvn -B integration-test"
This will create N jobs where N is the number of tokens in the list and each job with conf: ${{ matrix.token }} will resolve to a token in the list for the current job.
I think it may also be possible to store your tokens as secrets and setup the matrix like:
strategy:
matrix:
token: ["${{secrets.token1}}", "${{secrets.token2}}", etc...]
However, I haven't tested this.
EDIT
I found a trick to make the secrets tokens work:
Create your secrets and call them token1, token2, etc
Create your matrix configuration using the tokens i.e. the names of the secrets:
strategy:
matrix:
token: [token1, token2]
In your job's env, create the following environment variable:
env:
token: ${{secrets[matrix.token]}}
Now the actual value for the token for each build matrix is stored inside the environment variable ${{env.token}} (when operating within an expression context) or $token (in bash).
The environment variable will still remain a secret, so you don't loose anything.