So assume I have 2 workflows, one for building and testing for all pushes, then one for creating a release and packaging the app then adding it to the release, but this should only run on tagged pushes.
Currently I've found loads of posts which tell me how to only run for tags:
on:
push:
tag: *
So with this in place I then needed to check how I can have the 2nd workflow depend on the first one. I could find a brief bit of information which implied I should do:
on:
workflow_run:
workflows: ["first_one"]
types:
- completed
However I cannot find any examples which combine the two, and I assume this is a common use case, so im baffled as to how to do it.
In Azure devops I would probably have two stages with second stage dependsOn the first one with a condition on it for only tags.
So can anyone provides way to do this or at least best way to structure it so I can get test feedback on all pushes and create releases with artifact automatically on tags?
If these two workflows are only used in one project, I would structure it like the following: Create a job for building and testing (or make it two jobs) and then another one to package and release. All in one file.
The workflow would look like so:
on:
push:
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- run: build.sh && test.sh
package-and-release:
runs-on: ubuntu-latest
if: startsWith(github.event.ref, 'refs/tags/')
needs: [build-and-test]
steps:
- run: package.sh && release.sh
With this setup, the second job package-and-release is run after the first one and only if the push is triggered by a tag.
If you have this setup in multiple repositories, I would recommend using reusable workflows and either have one big reusable workflow that basically contains both of the above jobs, or have one reusable workflow per job. Note that "job" in this case can also be more than one job: While a reusable workflow is referenced from a job, it can itself contain more than one job.
The condition to only trigger the package & release on tag pushes could then be either inside the reusable workflow or outside. I would make that decision based on wether all of the users will do it this way or not.
Related
I'm not sure if this is a bug or a breaking change happened as of yesterday, I have a pretty simple setup calling three reusable workflows:
name: pr-checks
on:
pull_request:
branches: "**"
jobs:
lint:
name: Call Lint
uses: ./.github/checks/check-lint.yaml
test:
name: Call Test
uses: ./.github/checks/check-test.yaml
e2e:
name: Call E2E
uses: ./.github/checks/check-e2e.yaml
But this throws
"invalid value workflow reference: no version specified"
as of now, even though identical workflows have worked yesterday.
When reusing workflows like this at the 'job' level - it is not necessary to specify version, in fact, it used to error out if I specified the version.
Screenshots attached as I think this doesn't make much sense.
I did click on 're-run all jobs and it re-ran successfully.
However, without any discenrable difference and after also removing the build step just to be sure there's nothing weird happening there:
As you can see in your 2 screenshots, one is referring to the .github/workflows directory (the one which worked), and the other to the .github/checks directory (the one which didn't).
Short answer: If you change the workflow folder back to workflows instead of checks, it should work as expected.
Long answer: It seems there is a confusion between the syntax of two different concepts:
local actions (using an action in the same repo)
reusable workflows (reusing the same workflow in different workflows)
LOCAL ACTIONS
To access local actions (folders with action.yml file) from your workflow, you need to use the actions/checkout first, to allow it to access the other repository folders and files.
Example:
steps:
- uses: actions/checkout#v3 # Necessary to access local action
- name: Local Action Call
uses: ./.github/actions/local-action #path/to/action
I've made a POC here some time ago if you want to have a look.
REUSABLE WORKFLOWS
Now, if you want to use reusable workflows, the issue is different:
As with other workflow files, you locate reusable workflows in the
.github/workflows directory of a repository. Subdirectories of the
workflows directory are not supported.
GitHub documentation reference
In that case, according to this other section from the documentation:
You reference reusable workflow files using one of the following
syntaxes:
{owner}/{repo}/.github/workflows/{filename}#{ref} for reusable
workflows in public repositories.
./.github/workflows/{filename} for reusable workflows in the same repository.
{ref} can be a SHA, a release tag, or a branch name.
Example:
lint:
name: Call Lint
uses: ./.github/workflows/check-lint.yaml#{SHA/TAG/BRANCH}
or
lint:
name: Call Lint
uses: ./.github/workflows/check-lint.yaml
Here is another POC for the workflow call using this reusable workflow
CONCLUSION
It's like you were trying to call a reusable workflow as if it was a local action, which won't work as reusable workflows need to be located in the .github/workflows directory.
Note that you could eventually add the #branch-name at the end of the workflow call to be sure to use the workflow from the branch you want to test if the reusable workflow is already present on the default branch.
Is there any way we can run multiple jobs in a single runner or share the Github actions workspace between jobs?
In my organization, development teams use multiple reusable workflows created and managed by multiple teams. Team build creates and manages build.yaml that builds applications. My team creates and manages analysis.yaml that does data analysis on application builds and archives the built artifacts.
Developments teams are planning to use both of our workflows in their application workflow. For my team's workflow to work, my workflow needs to access the built code (target directory for maven builds, build directory for gradle builds and node_modules for npm builds).
Is there a way to run my reusable workflow on the runner where the code is built?
Is there a way I can get access to the workspace where the code is built (I searched other answers and learnt that I can use the upload action and cache action). Are there other ways I can accomplish this and run my reusable workflow on the build runner itself?
Will I accomplish this better with a composite action rather than using a reusable workflow?
I have the following example.
build.yaml
on:
workflow_call
inputs:
build:
description: Build type
required: true
jobs:
Build:
runs-on: self-hosted
steps:
- name: Building apps
- if: ${{inputs.build=='gradle'}}
run: |
gradle build
analysis.yaml
on:
workflow_call
inputs:
analysis:
description: Build type
required: true
type: boolean
jobs:
Build:
runs-on: self-hosted
steps:
- name: Building apps
- if: ${{inputs.analysis}}
run: |
#Run ML build analysis
#Archive the build artifacts
workflow.yaml
on:
push:
branches: [main]
jobs:
Build:
uses: buildteam/.github/workflows/build.yaml#main
with:
build: gradle
Analysis:
uses: analysis/.github/workflows/analysis.yaml#main
with:
analysis: true
No, as far as I know it is not possible to share workspace between jobs, at least for now.
Is there a way to run my reusable workflow on the runner where the
code is built?
Is there a way I can get access to the workspace where the code is
built (I searched other answers and learnt that I can use the upload
action and cache action). Are there other ways I can accomplish this
and run my reusable workflow on the build runner itself?
You could force that by using unique runner labels. By defining unique label on a runner you can then reference that runner in both jobs with runs-on
Will I accomplish this better with a composite action rather than
using a reusable workflow?
This is more of opinion based question, but I'll answer it anyway. Yes, I definitely think composite action would be a better solution in this case, especially because both build and analysis workflows seem simple and don't use secrets. But you should check the docs for differences between these two, there are some limitations.
I want to trigger a workflow that deploys the build. I want to make sure of the following:
CI workflow is completed
branch is develop
there are some changes in src/**
on:
workflow_run:
workflows: ['CI']
branches: [develop]
paths: ['src/**']
types:
- completed
It seems the above approach is not supported 😢, is there any other way?
Considering the workflows+branches combo should work, but paths: does not exist in the workflow_run context, you might consider using actions/verify-changed-files
That way, you can trigger an intermediate workflow which would check which files have changed.
Then the next step can be run only if the previous step (file check) has succeeded (using success())
I have around 10 individual micro-services which are mostly cloud functions for various data processing jobs, which all live in a single github repository.
The goal is to trigger the selective deployment of these service to Google Cloud Functions, on push to a branch - when an individual function has been updated.
I must avoid the situation in which update of a single service causes the deployment of all the cloud functions.
My current repository structure:
/repo
--/service_A
----/function
----/notebook
--/service_B
----/function
----/notebook
On a side note, what are the pros/cons of using Github Actions VS Google Cloud Build for such automation?
GitHub Actions supports monorepos with path filtering for workflows. You can create a workflow to selectively trigger when files on a specific path change.
https://help.github.com/en/articles/workflow-syntax-for-github-actions#onpushpull_requestpaths
For example, this workflow will trigger on a push when any files under the path service_A/ have changed (note the ** glob to match files in nested directories).
on:
push:
paths:
- 'service_A/**'
You could also run some script to discover which services were changed based on git diff and trigger corresponding job via GitHub REST API.
There could be two workflows main.yml and services.yml.
Main workflow will be configured to be started always on push and it will only start script to find out which services were changed. For each changed service repository dispatch event will be triggered with service name in payload.
Services workflow will be configured to be started on repository_dispatch and it will contain one job for each service. Jobs would have additional condition based on event payload.
See showcase with similar setup:
https://github.com/zladovan/monorepo
It's not a Monorepo
If you only have apps, then I'm sorry... but all you have is a repo of many apps.
A monorepo is a collection of packages that you can map a graph of dependencies between.
Aha, I have a monorepo
But if you have a collection of packges which depend on each other, then read on.
apps/
one/
depends:
pkg/foo
two/
depends:
pkg/bar
pkg/foo
pkg/
foo/
bar/
baz/
The answer is that you switch to a tool that can describe which packages have changed between the current git ref and some other git ref.
The following two examples runs the release npm script on each package that changed under apps/* and all the packges they would depend on.
I'm unsure if the pnpm method silently skips packages that don't have a release target/command/script.
Use NX Dev
Using NX.dev, it will work it out for you with its nx affected command.
you need a nx.json in the root of your monorepo
it assumes you're using the package.json approach with nx.dev, if you have project.json in each package, then the target would reside there.
your CI would then look like:
pnpx nx affected --target=release
Pnpm Filtering
Your other option is to switch to pnpm and use its filtering syntax:
pnpm --filter "...{apps/**}[origin/master]" release
Naive Path Filtering
If you just try and rely on "which paths" changed in this git commit, then you miss out on transient changes that affect the packages you actually want to deploy.
If you have a github action like:
on:
push:
paths:
- 'app/**'
Then you won't ever get any builds for when you only push commits that change anything in pkg/**.
Other interesting github actions
https://github.com/marketplace/actions/nx-check-changes
https://github.com/marketplace/actions/nx-affected-dependencies-action
https://github.com/marketplace/actions/nx-affected-list (a non nx alternative here is dorny/paths-filter
https://github.com/marketplace/actions/nx-affected-matrix
Has Changed Path Action might be worth a try:
name: Conditional Deploy
on: push
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
fetch-depth: 100
- uses: marceloprado/has-changed-path#v1
id: service_A_deployment
with:
paths: service_A
- name: Deploy front
if: steps.service_A_deployment.outputs.changed == 'true'
run: /deploy-service_A.sh
I am using TravisCI's Build Stages to separate my job to two stages:
Build and test on multiple environments.
Build and deploy if stage 1 pass.
And I want to Travis to run the jobs on commits to two GitHub branches, master and dev, and pull requests to master and dev. However, I only want to run the stage two when a commit to master happens.
Is there a way to completely ignore stage 2 in commits to branches that are not master, and on pull requests?
This is what my .travis.yml looks like at the time of writing this:
https://github.com/SammyIsra/photostream-react/blob/c354a62c3cc963b345a5c2fb95658c90ddc39d21/.travis.yml
Update:
this seems to not be possible as of yet. However, the TravisCI team may be working on something like that, as of this comment on the Build Stages feedback board. Whenever I learn that it was added as a feature, or that it will definitely not be possible ever, I will change this question.
There doesn't seem to be a way to ignore specific branches when triggering jobs in the build stages section, however it looks like there's a way to only fire your deployments on pushes to master.
In your build stage script section, you could wrap the npm run build command in a quick Bash if statement to test the environment variable which shows what branch you're on:
if [ TRAVIS_BRANCH == "master" ]; then npm run build; fi
Then, in your Surge deploy section, you can restrict the deployment to a particular branch with:
deploy:
...
on: master
And basically the same for the NPM deployment:
deploy:
...
on:
branch: master
Note: I've not tried any of this myself, so it may or may not work in combination with the new build stages functionality.