Github actions suddenly throwing "no version specified" for reusable workflows that worked yesterday - github

I'm not sure if this is a bug or a breaking change happened as of yesterday, I have a pretty simple setup calling three reusable workflows:
name: pr-checks
on:
pull_request:
branches: "**"
jobs:
lint:
name: Call Lint
uses: ./.github/checks/check-lint.yaml
test:
name: Call Test
uses: ./.github/checks/check-test.yaml
e2e:
name: Call E2E
uses: ./.github/checks/check-e2e.yaml
But this throws
"invalid value workflow reference: no version specified"
as of now, even though identical workflows have worked yesterday.
When reusing workflows like this at the 'job' level - it is not necessary to specify version, in fact, it used to error out if I specified the version.
Screenshots attached as I think this doesn't make much sense.
I did click on 're-run all jobs and it re-ran successfully.
However, without any discenrable difference and after also removing the build step just to be sure there's nothing weird happening there:

As you can see in your 2 screenshots, one is referring to the .github/workflows directory (the one which worked), and the other to the .github/checks directory (the one which didn't).
Short answer: If you change the workflow folder back to workflows instead of checks, it should work as expected.
Long answer: It seems there is a confusion between the syntax of two different concepts:
local actions (using an action in the same repo)
reusable workflows (reusing the same workflow in different workflows)
LOCAL ACTIONS
To access local actions (folders with action.yml file) from your workflow, you need to use the actions/checkout first, to allow it to access the other repository folders and files.
Example:
steps:
- uses: actions/checkout#v3 # Necessary to access local action
- name: Local Action Call
uses: ./.github/actions/local-action #path/to/action
I've made a POC here some time ago if you want to have a look.
REUSABLE WORKFLOWS
Now, if you want to use reusable workflows, the issue is different:
As with other workflow files, you locate reusable workflows in the
.github/workflows directory of a repository. Subdirectories of the
workflows directory are not supported.
GitHub documentation reference
In that case, according to this other section from the documentation:
You reference reusable workflow files using one of the following
syntaxes:
{owner}/{repo}/.github/workflows/{filename}#{ref} for reusable
workflows in public repositories.
./.github/workflows/{filename} for reusable workflows in the same repository.
{ref} can be a SHA, a release tag, or a branch name.
Example:
lint:
name: Call Lint
uses: ./.github/workflows/check-lint.yaml#{SHA/TAG/BRANCH}
or
lint:
name: Call Lint
uses: ./.github/workflows/check-lint.yaml
Here is another POC for the workflow call using this reusable workflow
CONCLUSION
It's like you were trying to call a reusable workflow as if it was a local action, which won't work as reusable workflows need to be located in the .github/workflows directory.
Note that you could eventually add the #branch-name at the end of the workflow call to be sure to use the workflow from the branch you want to test if the reusable workflow is already present on the default branch.

Related

Trying to refer a Reusbale Workflow from a Private repo

I am trying to find a workaround for using a Reusable workflow between two private repos: (It is not supported natively)
Context: I am trying to use a central repo (private) to contain all the Workflows and refer to that location from other repos (private).
Legend:
Central Repo: Contains all the Reusable Workflows. (If we edit a file in here we should not have to edit each and every repo's workflow files)
Calling Repo: A repo which runs Workflows, This repo will try to refer the workflows from the Central repo.
name: Calling Reusable Workflows
on: [ push ]
jobs:
download-remote-repo:
runs-on: ubuntu-latest
steps:
- name: Get private repo with action
uses: actions/checkout#v2
with:
repository: kalanatd/central-repo
ref: main
token: ${{ secrets.PAT_TOKEN }}
- name: List Files After remote Checkout
run: |
pwd
ls -a // I can see the checked out remote repo files in here
ls -al .github/workflows
cat ./.github/workflows/workflow.yml // Can read the file without any issue
- name: Run a workflow file
uses: ./.github/workflows/workflow.yml
Note: In aboe scenario ./.github/workflows/workflow.yml is dynamically checked out to the Runner's workspace. Even though 'run' commands can see that file 'uses' command does not use it.
[1] Error: Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/runner/work/caller-repo/caller-repo/workflow.yml'. Did you forget to run actions/checkout before running your local action?
[2] https://github.github.io/actions-cheat-sheet/actions-cheat-sheet.pdf
Note:
I learned that RWs can not be called under steps which means there is no way to download a RW from another place and use that workflow, right?
I am trying to find a workaround for using a Reusable workflow between two private repos: (It is not supported natively)
Actually... it is, at least on GitHub (and later on a private GHE instance)
GitHub Actions – Sharing actions and reusable workflows from private repositories is now GA (Dec. 2022)
The actions and reusable workflows from private repositories can now be shared with other private repositories within the same organization, user account, or enterprise.
See managing the repository settings and managing the enterprise repository settings to allow access to workflows in other repositories.
We have also added the API support to configure Actions share policy.
Refer to API support or API support for Enterprise for more details.
Learn more about Sharing actions and workflows from your private repository, Sharing actions and workflows with your organization, and Sharing Actions and workflows with your enterprise.

How to run a custom command unique to a PR on merging a PR?

I am wondering if there is any way to do the following. Say I have an "open data" repo, which allows people to submit content. The repo saves all the data, and the changes to the structured JSON/YAML is reviewed in a PR. But then because I am in a serverless system (like Vercel), I need to upload the changes to the data to the production database, on merge of the branch. So there should be required a custom data migration in the PR, which runs when the PR is approved and merged.
How can that be accomplished? All I can imagine as a solution is having a special "code block" in markdown with some JSON config explaining what script to run for the data migration, and you add that marked code snippet as a comment to the PR, then parse the PR comments and figure out what script to run from that. But that would be of course (seemingly) a super hack, so is there a right way to do something like this?
The other option is to have to run the script/command manually after you merge the PR, but ideally there would be a more automatic way of doing it.
GitHub itself can run code on various events through actions. Actions are configured through YAML files in the directory .github/workflows in the repository. Some actions relative to a branch use the workflow files from that branch, while “global” actions use the workflow files from the default branch (typically called main, or master for older repositories).
For example, this workflow runs bin/update-production-database whenever the main branch is updated (whether from a pull request merge or by pushing directly):
name: Update database
on:
push:
branches:
- main
jobs:
update-database:
runs-on: ubuntu-latest
steps:
- run: bin/update-production-database
See more examples in Deploying with GitHub Actions. To pass the credentials needed to access the database, set up an encrypted secret.
To only run the job on a PR merge and not on other pushes, see Running your workflow when a pull request merges.
If you use Vercel (which I know nothing about), it claims it “automatically deploys your GitHub projects” so there may be a built-in solution there (either using actions so that the trigger comes from GitHub, or using some Vercel-owned server which polls GitHub).

Having a github workflow dependent on another workflow AND it being a tagged push

So assume I have 2 workflows, one for building and testing for all pushes, then one for creating a release and packaging the app then adding it to the release, but this should only run on tagged pushes.
Currently I've found loads of posts which tell me how to only run for tags:
on:
push:
tag: *
So with this in place I then needed to check how I can have the 2nd workflow depend on the first one. I could find a brief bit of information which implied I should do:
on:
workflow_run:
workflows: ["first_one"]
types:
- completed
However I cannot find any examples which combine the two, and I assume this is a common use case, so im baffled as to how to do it.
In Azure devops I would probably have two stages with second stage dependsOn the first one with a condition on it for only tags.
So can anyone provides way to do this or at least best way to structure it so I can get test feedback on all pushes and create releases with artifact automatically on tags?
If these two workflows are only used in one project, I would structure it like the following: Create a job for building and testing (or make it two jobs) and then another one to package and release. All in one file.
The workflow would look like so:
on:
push:
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- run: build.sh && test.sh
package-and-release:
runs-on: ubuntu-latest
if: startsWith(github.event.ref, 'refs/tags/')
needs: [build-and-test]
steps:
- run: package.sh && release.sh
With this setup, the second job package-and-release is run after the first one and only if the push is triggered by a tag.
If you have this setup in multiple repositories, I would recommend using reusable workflows and either have one big reusable workflow that basically contains both of the above jobs, or have one reusable workflow per job. Note that "job" in this case can also be more than one job: While a reusable workflow is referenced from a job, it can itself contain more than one job.
The condition to only trigger the package & release on tag pushes could then be either inside the reusable workflow or outside. I would make that decision based on wether all of the users will do it this way or not.

How to set up actions in GitHub for new user?

I just got a GitHub account and writing small scripts in Python which I am learning.
While adding my code to GitHub I noticed there is an option to run tests/validation on my code but mine is empty.
I googled around and found that lint and black and are good checks.
I found this Action that I want to add - https://github.com/marketplace/actions/python-quality-and-format-checker
There is a "script" and a "config" that I think I need to add/update somewhere. Also when I click "Use latest version" it tells me to add the code into some .yml.
Can anyone assist me in installing this Action or point me in the right direction? Also, how can I use this Action on all my repositories/code?
=======================================
EDIT:
This link has the instructions - https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow
place yaml or yml in this directory -> .github/workflows
For this Action: https://github.com/marketplace/actions/python-quality-and-format-checker
the code inside the file will look like this:
on: [push, pull_request]
name: Python Linting
jobs:
PythonLinting:
name: Python linting
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#master
- name: Konstruktoid Python linting
uses: konstruktoid/action-pylint#master
thanks to: bertrand martel
pylint is part of the new GitHub Super Linter (github/super-linter):
Introducing GitHub Super Linter: one linter to rule them all
The Super Linter is a source code repository that is packaged into a Docker container and called by GitHub Actions. This allows for any repository on GitHub.com to call the Super Linter and start utilizing its benefits.
When you’ve set your repository to start running this action, any time you open a pull request, it will start linting the code case and return via the Status API.
It will let you know if any of your code changes passed successfully, or if any errors were detected, where they are, and what they are.
This then allows the developer to go back to their branch, fix any issues, and create a new push to the open pull request.
At that point, the Super Linter will run again and validate the updated code and repeat the process.
And you can set it up to only int new files if you want.
Update August 2020:
github/super-linter issue 226 has been closed with PR 593:
This Pr will add:
Black python linting
Updated tests

Deploy individual services from a monorepo using github actions

I have around 10 individual micro-services which are mostly cloud functions for various data processing jobs, which all live in a single github repository.
The goal is to trigger the selective deployment of these service to Google Cloud Functions, on push to a branch - when an individual function has been updated.
I must avoid the situation in which update of a single service causes the deployment of all the cloud functions.
My current repository structure:
/repo
--/service_A
----/function
----/notebook
--/service_B
----/function
----/notebook
On a side note, what are the pros/cons of using Github Actions VS Google Cloud Build for such automation?
GitHub Actions supports monorepos with path filtering for workflows. You can create a workflow to selectively trigger when files on a specific path change.
https://help.github.com/en/articles/workflow-syntax-for-github-actions#onpushpull_requestpaths
For example, this workflow will trigger on a push when any files under the path service_A/ have changed (note the ** glob to match files in nested directories).
on:
push:
paths:
- 'service_A/**'
You could also run some script to discover which services were changed based on git diff and trigger corresponding job via GitHub REST API.
There could be two workflows main.yml and services.yml.
Main workflow will be configured to be started always on push and it will only start script to find out which services were changed. For each changed service repository dispatch event will be triggered with service name in payload.
Services workflow will be configured to be started on repository_dispatch and it will contain one job for each service. Jobs would have additional condition based on event payload.
See showcase with similar setup:
https://github.com/zladovan/monorepo
It's not a Monorepo
If you only have apps, then I'm sorry... but all you have is a repo of many apps.
A monorepo is a collection of packages that you can map a graph of dependencies between.
Aha, I have a monorepo
But if you have a collection of packges which depend on each other, then read on.
apps/
one/
depends:
pkg/foo
two/
depends:
pkg/bar
pkg/foo
pkg/
foo/
bar/
baz/
The answer is that you switch to a tool that can describe which packages have changed between the current git ref and some other git ref.
The following two examples runs the release npm script on each package that changed under apps/* and all the packges they would depend on.
I'm unsure if the pnpm method silently skips packages that don't have a release target/command/script.
Use NX Dev
Using NX.dev, it will work it out for you with its nx affected command.
you need a nx.json in the root of your monorepo
it assumes you're using the package.json approach with nx.dev, if you have project.json in each package, then the target would reside there.
your CI would then look like:
pnpx nx affected --target=release
Pnpm Filtering
Your other option is to switch to pnpm and use its filtering syntax:
pnpm --filter "...{apps/**}[origin/master]" release
Naive Path Filtering
If you just try and rely on "which paths" changed in this git commit, then you miss out on transient changes that affect the packages you actually want to deploy.
If you have a github action like:
on:
push:
paths:
- 'app/**'
Then you won't ever get any builds for when you only push commits that change anything in pkg/**.
Other interesting github actions
https://github.com/marketplace/actions/nx-check-changes
https://github.com/marketplace/actions/nx-affected-dependencies-action
https://github.com/marketplace/actions/nx-affected-list (a non nx alternative here is dorny/paths-filter
https://github.com/marketplace/actions/nx-affected-matrix
Has Changed Path Action might be worth a try:
name: Conditional Deploy
on: push
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
fetch-depth: 100
- uses: marceloprado/has-changed-path#v1
id: service_A_deployment
with:
paths: service_A
- name: Deploy front
if: steps.service_A_deployment.outputs.changed == 'true'
run: /deploy-service_A.sh