I'm using circleCI version 2 and my config.yml like this:
version: 2
jobs:
a:
steps:...
b:
steps:...
workflows:
version: 2
main_pipeline:
jobs:
- a
- b
I want only to build when a change happens in the directory.
job a for folder a
job b for folder b
when folder a changes, build only job a.
function trigger_job() {
job_name=$1
curl --user ${CIRCLE_API_TOKEN}: \
--data build_parameters[CIRCLE_JOB]=$job_name \
--data revision=$CIRCLE_SHA1 \
https://circleci.com/api/v1.1/project/github/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME/tree/$CIRCLE_BRANCH
I use this function for trigger job and find different by git-diff like this git diff-tree --name-only $(git log -n 2 --pretty=format:"%H") | grep project
CircleCI now has support for dynamic configs (instead of a single static config file), and with that a path-filtering orb is available that can detect changes under a folder and set pipeline parameters accordingly.
See https://circleci.com/docs/2.0/using-dynamic-configuration/#execute-specific-workflows-or-steps-based-on-which-files-are-modified
You can also check out our config
Related
After following guides like this one I am able to successfully run dependabot against my Azure DevOps repo and it auto creates PRs. The issue is I have some customizations I need to make such as ignoring specific packages as the dependabot documentation says can be done here are not working.
Not sure if it is the way I am composing the options object or something else, but no values seem to be honored.
Here is what my Azure DevOps Pipeline looks like:
trigger:
- main
jobs:
- job: dependabot
displayName: Dependabot Execution
pool:
vmImage: 'ubuntu-latest'
variables:
- name: DIRECTORY_PATH
value: /MyApp/
- name: PACKAGE_MANAGER
value: nuget
- name: PROJECT_PATH
value: someDomain/someProject/_git/my-app
- name: OPTIONS
value: |
{"ignore":[{"dependency-name":"NLog*"}]}
# {"ignore_conditions":[{"dependency-name":"NLog*"}]} # also tried and did not work
steps:
- script: git clone https://github.com/dependabot/dependabot-script.git
displayName: Clone Dependabot config repo
- script: |
cd dependabot-script
docker build -t "dependabot/dependabot-script" -f Dockerfile .
displayName: Build Dependabot Image
- script: |
docker run --rm -e AZURE_ACCESS_TOKEN='$(PAT)' \
-e GUTHUB_ACCESS_TOKEN='$(GHPAT)' \
-e PACKAGE_MANAGER='$(PACKAGE_MANAGER)' \
-e PROJECT_PATH='$(PROJECT_PATH)' \
-e DIRECTORY_PATH='$(DIRECTORY_PATH)' \
-e OPTIONS='$(OPTIONS)' \
dependabot/dependabot-script
displayName: Run Dependabot
And here is the output when the pipeline runs:
Running with options: {:ignore=>[{:"dependency-name"=>"NLog*"}]}
Fetching nuget dependency files for someDomain/someProject/_git/my-app
Parsing dependencies information
- Updating NLog (from 5.1.0)… submitted
- Updating System.Data.SqlClient (from 4.8.4)… submitted
Done
Finishing: Run Dependabot
As you can see, 2 PRs are created, which is great, except the NLog one should have been ignored/skipped. I have also tried other options such as commit-message prefix and it did not take either.
Any help is appreciated!
I'm new to GitHub actions, I want to use PowerShell and save the output of a command to a file in my repository, but I don't know what's the right way to do it.
name: get hash
on: [workflow_dispatch]
jobs:
build:
name: Run Script
runs-on: windows-latest
steps:
- uses: actions/checkout#v3
- name: Script
run: ./script.ps1
shell: pwsh
and using get-filehash command I'm trying to learn creating a simple Github workflow that saves the hash of a file to a text file in my repository.
this is the script.ps1 file content
$wc = [System.Net.WebClient]::new()
$pkgurl = 'File on the web'
$FileHashSHA512 = Get-FileHash -Algorithm SHA512 -InputStream ($wc.OpenRead($pkgurl))
$FileHashSHA512.Hash > .\sss.txt
I know it's not practical but I want to learn basics of workflows. the command works on my computer but I just don't get how Github workflows work when it comes to saving files on my repository.
I also want to make this command work in the same workflow:
Invoke-WebRequest -Uri $pkgurl -OutFile .\file.zip
but again don't know how to make the workflow save the file in the repository's root.
since these are very basic, I'm trying to create/write the workflow files myself and not use other people's pre-made actions.
The actions runner is just a PC that clones the repo and runs a few commands in it. So when you create a new file on the runner, you have to add it to the repo on the runner, create a commit and push that to the repo over on GitHub.
It might help to understand that the repo in GitHub and the runner in GitHub actions are two completely seperate things. It's not like the actions runner somehow runs 'inside' your repo.
So one way to save filer is to run:
git add fileyoujustcreated.txt
git commit -m "updating file"
git push
Here is a sample GitHub action I use to update a file in my repo each time the content changes:
Composite action that changes a set of files and commits them
Workflow that calls the composite action
Alternatively, you can edit files in GitHub using the REST API, so you could invoke:
curl \
-X PUT \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer <YOUR-TOKEN>"\
-H "X-GitHub-Api-Version: 2022-11-28" \
https://api.github.com/repos/OWNER/REPO/contents/PATH \
-d '{"message":"my commit message","committer":{"name":"Monalisa Octocat","email":"octocat#github.com"},"content":"bXkgbmV3IGZpbGUgY29udGVudHM="}'
Or the equivalent in PowerRhell.
You can use the ${{ GitHub.token }} variable to authenticate.
See:
https://docs.github.com/en/rest/repos/contents?apiVersion=2022-11-28#create-or-update-file-contents
I am relatively new to GitHub workflows and testing. I am working in a private GitHub repository with a dozen colleagues. We want to avoid using services like CircleCI for the time being and see how much we can do with just the integrated GitHub actions, since we are unsure about the kind of access a third party service would be getting to the repo.
Currently, we have two workflows (each one tests the same code for a separate Python environment) that get triggered on push or pull request in the master branch.
The steps of the workflow are as follows (the full workflow yml file is given at the bottom):
Install Anaconda
Create the conda environment (installing dependencies)
Patch libraries
Build a 3rd party library
Run python unit tests
It would be amazing to know immediately which part of the code failed given some new pull requests. Right now, every aspect of the codebase gets tested by a single python file run_tests.py. I was thinking of splitting up this file and creating a workflow per aspect I want to test separately, but then I would have to create a whole new environment, patch the libraries and build the 3rd party library every time I want to conduct a single test. These tests already take quite some time.
My question is now: is there any way to avoid doing that? Is there a way to build everything on the Linux server and re-use that, so that they don't need to be rebuilt every test? Is there a way to display a badge per python test that fails/succeeds, so that we can give more information than just "everything passed" or "everything failed". Is such a thing better suited for a service like CircleCI (or other recommendations are also welcome)?
Here is the full yml file for the workflow for the Python 3 environment. The Python2 one is identical except for the anaconda environment steps.
name: (Python 3) install and test
# Controls when the workflow will run
on:
# Triggers the workflow on push or pull request events but only for the master branch
push:
branches: [ master ]
pull_request:
branches: [ master ]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest
defaults:
run:
shell: bash -l {0}
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout#v2
# Install Anaconda3 and update conda package manager
- name: Install Anaconda3
run: |
wget https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh --quiet
bash Anaconda3-2020.11-Linux-x86_64.sh -b -p ~/conda3-env-py3
source ~/conda3-env-py3/bin/activate
conda info
# Updating the root environment. Install dependencies (YAML)
# NOTE: The environment file (yaml) is in the 'etc' folder
- name: Install ISF dependencies
run: |
source ~/conda3-env-py3/bin/activate
conda-env create --name isf-py3 --file etc/env-py3.yml --quiet
source activate env-py3
conda list
# Patch Dask library
- name: Patch dask library
run: |
echo "Patching dask library."
source ~/conda3-env-py3/bin/activate
source activate env-py3
cd installer
python patch_dask_linux64.py
conda list
# Install pandas-msgpack
- name: Install pandas-msgpack
run: |
echo "Installing pandas-msgpack"
git clone https://github.com/abast/pandas-msgpack.git
# Applying patch to pandas-msgpack (generating files using newer Cython)
git -C pandas-msgpack apply ../installer/pandas_msgpack.patch
source ~/conda3-env-py3/bin/activate
source activate env-py3
cd pandas-msgpack; python setup.py install
pip list --format=freeze | grep pandas
# Compile neuron mechanisms
- name: Compile neuron mechanisms
run: |
echo "Compiling neuron mechanisms"
source ~/conda3-env-py3/bin/activate
source activate env-py3
pushd .
cd mechanisms/channels_py3; nrnivmodl
popd
cd mechanisms/netcon_py3; nrnivmodl
# Run tests
- name: Testing
run: |
source ~/conda3-env-py3/bin/activate
source activate env-py3
export PYTHONPATH="$(pwd)"
dask-scheduler --port=38786 --dashboard-address=38787 &
dask-worker localhost:38786 --nthreads 1 --nprocs 4 --memory-limit=100e15 &
python run_tests.py
Many thanks in advance
Tried:
Building everything in a single github workflow, testing everything in the same workflow.
Expected:
Gaining information on specific steps that failed or worked. Displaying this information as a badge on the readme page.
Actual result:
Only the overall success status can be displayed as badge. Only the success status of "running all tests" is available.
I have this workflow in a repo called terraform-do-database and I'm trying to use a reusable workflow coming from the public repo foo/git-workflows/.github/workflows/tag_validation.yaml#master
name: Tag Validation
on:
pull_request:
branches: [master]
push:
branches:
- '*' # matches every branch that doesn't contain a '/'
- '*/*' # matches every branch containing a single '/'
- '**' # matches every branch
- '!master' # excludes master
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
jobs:
tag_check:
uses: foo/git-workflows/.github/workflows/tag_validation.yaml#master
And this is the reusable workflow file from the public git-workflows repo that has the script that should run on it. What is happening is that the workflow is trying to use a script inside the repo terraform-do-database
name: Tag Validation
on:
pull_request:
branches: [master]
workflow_call:
jobs:
tag_check:
# The type of runner that the job will run on
runs-on: ubuntu-latest
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout#v3
# Runs a single command using the runners shell
- name: Verify the tag value
run: ./scripts/tag_verify.sh
So the question: How can I make the workflow use the script stored in the git-worflows repo instead of the terraform-do-database?
I want to have a single repo where I can call the workflow and the scripts, I don't want to have everything duplicated inside all my repos.
I have found that if I wrap the script into a composite action. I can use GitHub context github.action_path to locate the scripts.
Example:
run: ${{ github.action_path }}/scripts/foo.sh
One way to go about this is perform a checkout inside your reusable workflow that essentially clones the content of the repo where your scripts are and only then you can access it. It's not the cleanest solution but it works.
Perform a second checkout, to clone your repo that has the reusable workflow into a dir reusable-workflow-repo
- name: Checkout reusable workflow dir
uses: actions/checkout#v3
with:
repository: <your-org>/terraform-do-database
token: ${{ secrets.GIT_ACCESS_TOKEN }}
path: reusable-workflow-repo
Now you have all the code you need inside reusable-workflow-repo. Use ${GITHUB_WORKSPACE} to find the current path and simply append the path to the script.
- name: Verify the tag value
run: ${GITHUB_WORKSPACE}/reusable-workflow-repo/scripts/tag_verify.sh
I was able to solve it adding a few more commands to manually download the script and execute it.
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout#v3
# Runs a single command using the runners shell
- name: Check current directory
run: pwd
- name: Download the script
run: curl -o $PWD/tag_verify.sh https://raw.githubusercontent.com/foo/git-workflows/master/scripts/tag_verify.sh
- name: Give script permissions
run: chmod +x $PWD/tag_verify.sh
- name: Execute script
run: $PWD/tag_verify.sh
Following Kaleby Cadorin example but for the case where the script is in a private repository
- name: Download & run script
run: |
curl --header "Authorization: token ${{ secrets.MY_PAT }}" \
--header 'Accept: application/vnd.github.raw' \
--remote-name \
--location https://raw.githubusercontent.com/COMPANY/REPO/BRANCH/PATH/script.sh
chmod +x script.sh
./script.sh
Note: GITHUB_TOKEN doesn't seem to work here, a PAT is required.
According to this thread on github-community the script needs to be downloaded/checked out separatly.
The "reusable" workflow you posted is not reusable in this sense, because since it is not downloading the script the workflow can only run within its own repository (or a repository that already has the script).
So, the problem I have is that right now CircleCI doesn't trigger builds on PRs, and I have checks that I'd like to run only on PRs. There is this option:
Only build pull requests
By default, we will build all the commits for this project. Once
turned on, we will only build branches that have associated pull
requests open. Note: For your default branch, we will always build all
commits.
However, this is not what I need, because I still need some checks to run on every commit, on all branches, not just the default one.
This is what I have in my circle.yml file:
test:
override:
- if [[ ! -z $CI_PULL_REQUEST ]] ; then yarn test:selenium ; fi
This only gets triggered if there's another push made to the branch after opening a PR, because only then is the build triggered.
I just haven't been able to find a workaround for this, does anyone have any ideas?
There is NOW a way if you are on github, and you are willing to run a quick github action to tickle CircleCI. The action is pretty quick so doesn't consume much github actions time.
Setup an action by saving the following in .github/workflows/main.yml
name: CI
# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
pull_request:
types: [opened,reopened]
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
- name: Tickle CircleCI
env:
CIRCLE_BRANCH: ${{ github.head_ref }}
CIRCLE_TOKEN: ${{ secrets.CIRCLE_TOKEN }}
run: |
curl -X POST \
-H "Circle-Token: ${CIRCLE_TOKEN}" \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-d "{\"branch\":\"${CIRCLE_BRANCH}\"}" \
https://circleci.com/api/v2/project/<project_slug>/pipeline
Replace <project_slug> with the slug for your project which is gh/<repo_owner>/<repo_name>.
Now go setup a token in CircleCI and add it to secrets in github actions as CIRCLE_TOKEN.
Next add a step to your .circleci/config.yml in any job you want to be conditionally run as the very first step. We always run tests on master
so this takes that into account. Feel free to adjust to your needs.
- run:
name: Check for PR or master
command: |
if [ -z "$CIRCLE_PULL_REQUEST" ]; then
if [ "$CIRCLE_BRANCH" != "master" ]; then
echo "Not a PR or master, halting"
circleci step halt
fi
fi
If a PR is not open a job will run on CircleCI and then immediately stop. This isn't super expensive in terms of CircleCI time either, so overall this lets us run all commits to master and anything in an open PR for reasonable cost.
When you open a PR the github action will run and trigger CircleCI to rerun but since a PR is now open it will run completely.
The one downside to this approach is that you WILL get a green check on commits run before the PR is open indicating tests passed but they did not, however as soon as the github action runs the last commit, the one you care about, will change back to yellow as CircleCI reruns the job.
Future commits are run only by CircleCI and the github action does not run at all costing you nothing in github actions once the PR is open.
If you don't want use github actions and just want something that you can click a few buttons to configure check our PR Triggers using Swissknife
Discalimer: I built this since I needed this functionality and a few other things.