I am trying to find a workaround for using a Reusable workflow between two private repos: (It is not supported natively)
Context: I am trying to use a central repo (private) to contain all the Workflows and refer to that location from other repos (private).
Legend:
Central Repo: Contains all the Reusable Workflows. (If we edit a file in here we should not have to edit each and every repo's workflow files)
Calling Repo: A repo which runs Workflows, This repo will try to refer the workflows from the Central repo.
name: Calling Reusable Workflows
on: [ push ]
jobs:
download-remote-repo:
runs-on: ubuntu-latest
steps:
- name: Get private repo with action
uses: actions/checkout#v2
with:
repository: kalanatd/central-repo
ref: main
token: ${{ secrets.PAT_TOKEN }}
- name: List Files After remote Checkout
run: |
pwd
ls -a // I can see the checked out remote repo files in here
ls -al .github/workflows
cat ./.github/workflows/workflow.yml // Can read the file without any issue
- name: Run a workflow file
uses: ./.github/workflows/workflow.yml
Note: In aboe scenario ./.github/workflows/workflow.yml is dynamically checked out to the Runner's workspace. Even though 'run' commands can see that file 'uses' command does not use it.
[1] Error: Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/runner/work/caller-repo/caller-repo/workflow.yml'. Did you forget to run actions/checkout before running your local action?
[2] https://github.github.io/actions-cheat-sheet/actions-cheat-sheet.pdf
Note:
I learned that RWs can not be called under steps which means there is no way to download a RW from another place and use that workflow, right?
I am trying to find a workaround for using a Reusable workflow between two private repos: (It is not supported natively)
Actually... it is, at least on GitHub (and later on a private GHE instance)
GitHub Actions – Sharing actions and reusable workflows from private repositories is now GA (Dec. 2022)
The actions and reusable workflows from private repositories can now be shared with other private repositories within the same organization, user account, or enterprise.
See managing the repository settings and managing the enterprise repository settings to allow access to workflows in other repositories.
We have also added the API support to configure Actions share policy.
Refer to API support or API support for Enterprise for more details.
Learn more about Sharing actions and workflows from your private repository, Sharing actions and workflows with your organization, and Sharing Actions and workflows with your enterprise.
Related
Has the Following Feature has been implemented for Gihub Repos yet?strong text
Multi-repo triggers
You can specify multiple repositories in one YAML file and cause a pipeline to trigger by updates to any of the repositories. This feature is useful, for instance, in the following scenarios:
You consume a tool or a library from a different repository. You want to run tests for your application whenever the tool or library is updated.
You keep your YAML file in a separate repository from the application code. You want to trigger the pipeline every time an update is pushed to the application repository.
With this update, multi-repo triggers will only work for Git repositories in Azure Repos. They don't work for GitHub or Bitbucket repository resources.
SAMPLE :
trigger:
main
resources:
repositories:
- repository: tools
type: git
name: MyProject/tools
ref: main
trigger:
branches:
include:
- main
- release
As per Microsoft official sprint 173 updates 2020, this is achiveable using resources tag inside your yaml.
Here is an example that shows how you can setup an auto trigger inside your yaml pipeline based on any change in any other repos inside the same project and even other projects inside Azure DevOps as well.
Sample:
trigger:
- main
resources:
repositories:
- repository: tools
type: git
name: MyProject/tools
ref: main
trigger:
branches:
include:
- main
- release
In the above code snippet:
main branch in the self repo containing the YAML file
main or release branches in tools repo
Here is the link for official documentation form Microsoft for further details.
Hope that solution works for you.
Repository resource triggers only work for Azure Repos Git repositories in the same organization at present. They do not work for GitHub or Bitbucket repository resources.
Refer to this official doc for details: https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#triggers
If you would like this feature to be supported, create a feature request: https://developercommunity.visualstudio.com/AzureDevOps/suggest
Overview
The GitHub documentation suggests that actions can be shared between repositories within an enterprise. However, when I try to use an action from inside my enterprise from another repository I get the error:
Error: Unable to clone https://github.com/Org1/action refs/heads/workflow-test: repository not found
Current set up
We have two organizations Org1 and Org2 that are both in the same enterprise. Org1 contains an repository containing an action: Org1/action. Org2 contains a repository which would like to consume this action: Org2/consumer.
I have followed this documentation: https://docs.github.com/en/enterprise-cloud#latest/actions/creating-actions/sharing-actions-and-workflows-with-your-enterprise.
Org1/action
Visibility is set to internal although it was originally private.
I have set Settings>Actions>General>Access to Accessible from repositories in the 'ENTERPRISE NAME' enterprise
Org2/consumer
This repository is set to allow all actions and reusable workflows.
Problem
There is no good documentation on GitHub which explains how to consume actions from within the enterprise.
This is what we currently have in Org2/consumer:
Name: Load test
on: [push]
jobs:
run_action:
runs-on: self-hosted
name: Action
steps:
- name: Trigger Load Test
uses: Org1/action#workflow-test # the action is on this branch
id: action
When we run this action we get the error displayed in the Overview section. This seems like an issue with how the runner is authorizing with GitHub when pulling the repository.
I would love to know if anyone has overcome this issue, or if they have found a work around.
Thanks in advance.
I'm trying to set up a pipeline that will trigger, when a commit is made in a repository that exists in a different organization.
In my own org, I've created a git repo with a yaml pipeline file in the main branch.
With the below setup, I can checkout the code from the other organization if I run the pipeline manually. But it is not triggered when a commit is pushed to that repository.
Looking at the documentation, this should be possible?
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#triggers
resources:
repositories:
- repository: OtherOrgRepo # In a different organization
endpoint: OtherOrgConnection
type: git
name: proj/reponame
ref: develop
trigger:
- develop
pool:
vmImage: ubuntu-latest
steps:
- checkout: OtherOrgRepo
The token used for the service connection has full access.
Is this not supported, or am I missing a step?
I guess I just need to read the big blue box:
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#triggers
Repository resource triggers only work for Azure Repos Git repositories in the same organization at present. They do not work for GitHub or Bitbucket repository resources.
I did however manage to trigger a classic pipeline by using a generic Git service connection, which will poll for changes at an interval.
Trying to use pulumi preview with Github actions. The workflow file is giving me the below error.
Invalid workflow file: .github/workflows/pull_request.yml#L12
error parsing called workflow "peopleticker/ccc/.github/workflows/pull_request.yml#master": workflow was not found.
I'm using the workflow file below kindly check:
name: Call a reusable workflow
permissions:
id-token: write
contents: read
on:
pull_request:
jobs:
call-workflow-passing-data:
uses: peopleticker/ccc/.github/workflows/pull_request.yml#master
According to GitHub document:
doc
You can visit repository Setting -> Actions -> Access: Choose "Accessible from repositories in the organization"
You can't use reusable workflows from other private repositories.
Reusable workflows stored within a private repository can only be used by workflows within the same repository.
source
You can use private repo workflows but you have to give access on org level.
Github give access that.
Related documentaion: github-doc
Example Video Explanation: video-expl
I have around 10 individual micro-services which are mostly cloud functions for various data processing jobs, which all live in a single github repository.
The goal is to trigger the selective deployment of these service to Google Cloud Functions, on push to a branch - when an individual function has been updated.
I must avoid the situation in which update of a single service causes the deployment of all the cloud functions.
My current repository structure:
/repo
--/service_A
----/function
----/notebook
--/service_B
----/function
----/notebook
On a side note, what are the pros/cons of using Github Actions VS Google Cloud Build for such automation?
GitHub Actions supports monorepos with path filtering for workflows. You can create a workflow to selectively trigger when files on a specific path change.
https://help.github.com/en/articles/workflow-syntax-for-github-actions#onpushpull_requestpaths
For example, this workflow will trigger on a push when any files under the path service_A/ have changed (note the ** glob to match files in nested directories).
on:
push:
paths:
- 'service_A/**'
You could also run some script to discover which services were changed based on git diff and trigger corresponding job via GitHub REST API.
There could be two workflows main.yml and services.yml.
Main workflow will be configured to be started always on push and it will only start script to find out which services were changed. For each changed service repository dispatch event will be triggered with service name in payload.
Services workflow will be configured to be started on repository_dispatch and it will contain one job for each service. Jobs would have additional condition based on event payload.
See showcase with similar setup:
https://github.com/zladovan/monorepo
It's not a Monorepo
If you only have apps, then I'm sorry... but all you have is a repo of many apps.
A monorepo is a collection of packages that you can map a graph of dependencies between.
Aha, I have a monorepo
But if you have a collection of packges which depend on each other, then read on.
apps/
one/
depends:
pkg/foo
two/
depends:
pkg/bar
pkg/foo
pkg/
foo/
bar/
baz/
The answer is that you switch to a tool that can describe which packages have changed between the current git ref and some other git ref.
The following two examples runs the release npm script on each package that changed under apps/* and all the packges they would depend on.
I'm unsure if the pnpm method silently skips packages that don't have a release target/command/script.
Use NX Dev
Using NX.dev, it will work it out for you with its nx affected command.
you need a nx.json in the root of your monorepo
it assumes you're using the package.json approach with nx.dev, if you have project.json in each package, then the target would reside there.
your CI would then look like:
pnpx nx affected --target=release
Pnpm Filtering
Your other option is to switch to pnpm and use its filtering syntax:
pnpm --filter "...{apps/**}[origin/master]" release
Naive Path Filtering
If you just try and rely on "which paths" changed in this git commit, then you miss out on transient changes that affect the packages you actually want to deploy.
If you have a github action like:
on:
push:
paths:
- 'app/**'
Then you won't ever get any builds for when you only push commits that change anything in pkg/**.
Other interesting github actions
https://github.com/marketplace/actions/nx-check-changes
https://github.com/marketplace/actions/nx-affected-dependencies-action
https://github.com/marketplace/actions/nx-affected-list (a non nx alternative here is dorny/paths-filter
https://github.com/marketplace/actions/nx-affected-matrix
Has Changed Path Action might be worth a try:
name: Conditional Deploy
on: push
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
fetch-depth: 100
- uses: marceloprado/has-changed-path#v1
id: service_A_deployment
with:
paths: service_A
- name: Deploy front
if: steps.service_A_deployment.outputs.changed == 'true'
run: /deploy-service_A.sh