using `git commit —no-verify` for pre-commit azure pipeline - azure-devops

I see that I can use pre-commit with pipelines, is there a way to set up the yaml file for azure pipeline to use git commit --no-verify when if fails for specific cases? or is there a way to troubleshoot the pipeline when the issue occurs?
this is what I have for the yaml file
pool:
vmImage: ubuntu-18.04
variables:
PRE_COMMIT_HOME: $(Pipeline.Workspace)/pre-commit-cache
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: ${{ parameters.python }}
- script: |
echo "##vso[task.setvariable variable=PY]$(python -VV)"
displayName: set version variables
- task: CacheBeta#0
inputs:
key: pre-commit | .pre-commit-config.yaml | "$(PY)"
path: $(PRE_COMMIT_HOME)
- script: python -m pip install --upgrade pre-commit
displayName: install pre-commit
- script: pre-commit run --all-files --show-diff-on-failure
displayName: run pre-commit

Check the documentation here:
Not all hooks are perfect so sometimes you may need to skip execution
of one or more hooks. pre-commit solves this by querying a SKIP
environment variable. The SKIP environment variable is a comma
separated list of hook ids. This allows you to skip a single hook
instead of --no-verifying the entire commit.
$ SKIP=flake8 git commit -m "foo"

Related

How to pass Dependabot OPTIONS properties to dependabot-script in Azure DevOps Pipeline

After following guides like this one I am able to successfully run dependabot against my Azure DevOps repo and it auto creates PRs. The issue is I have some customizations I need to make such as ignoring specific packages as the dependabot documentation says can be done here are not working.
Not sure if it is the way I am composing the options object or something else, but no values seem to be honored.
Here is what my Azure DevOps Pipeline looks like:
trigger:
- main
jobs:
- job: dependabot
displayName: Dependabot Execution
pool:
vmImage: 'ubuntu-latest'
variables:
- name: DIRECTORY_PATH
value: /MyApp/
- name: PACKAGE_MANAGER
value: nuget
- name: PROJECT_PATH
value: someDomain/someProject/_git/my-app
- name: OPTIONS
value: |
{"ignore":[{"dependency-name":"NLog*"}]}
# {"ignore_conditions":[{"dependency-name":"NLog*"}]} # also tried and did not work
steps:
- script: git clone https://github.com/dependabot/dependabot-script.git
displayName: Clone Dependabot config repo
- script: |
cd dependabot-script
docker build -t "dependabot/dependabot-script" -f Dockerfile .
displayName: Build Dependabot Image
- script: |
docker run --rm -e AZURE_ACCESS_TOKEN='$(PAT)' \
-e GUTHUB_ACCESS_TOKEN='$(GHPAT)' \
-e PACKAGE_MANAGER='$(PACKAGE_MANAGER)' \
-e PROJECT_PATH='$(PROJECT_PATH)' \
-e DIRECTORY_PATH='$(DIRECTORY_PATH)' \
-e OPTIONS='$(OPTIONS)' \
dependabot/dependabot-script
displayName: Run Dependabot
And here is the output when the pipeline runs:
Running with options: {:ignore=>[{:"dependency-name"=>"NLog*"}]}
Fetching nuget dependency files for someDomain/someProject/_git/my-app
Parsing dependencies information
- Updating NLog (from 5.1.0)… submitted
- Updating System.Data.SqlClient (from 4.8.4)… submitted
Done
Finishing: Run Dependabot
As you can see, 2 PRs are created, which is great, except the NLog one should have been ignored/skipped. I have also tried other options such as commit-message prefix and it did not take either.
Any help is appreciated!

How to trigger different Azure CD pipelines depending on git branch

The environment which I'm deploying to has 3 stages
dev
uat
production
The git repository for the project has dev and main branches.
On push to dev, the CI pipeline runs, creates an image and pushes the image to a registry.
Next the CD release pipeline is triggered and auto deploys to the dev environment.
I want the next stage to be triggered by the merge from dev to main,
so I can't see how I can use the same CD pipeline - I'd love to be shown that this is actually possible.
So for now, I'm going with 2 CI pipelines and 2 cd pipelines as follows:
dev-branch
CI (auto triggered by push to dev)
CD
dev-stage (auto triggered by CI above)
main-branch
CI (auto triggered by push to main)
CD
uat-stage (auto triggered by CI above)
prod-stage
The problem that I'd like to solve with this setup is :
can this be done with a single CI pipeline, suppressing the CD pipeline runs based on the source branch in git which triggered the build?
My CI pipeline yaml is as follows, I'd like to use this one file for both main and dev,
rather than having to duplicate it with the only difference being the individual triggers for dev and main.
trigger:
branches:
include:
- dev
- main
variables:
- group: devops-project-group
- template: app-variables.yml
pool:
vmImage: ubuntu-latest
steps:
- checkout: self
- bash: |
echo 'starting build ...'
you can use conditions:
- bash: |
echo 'starting MASTER build ...'
condition: and(succeeded(), variables['Build.SourceBranch'], 'refs/heads/master')
alternatively you can do ifs:
steps:
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/master') }}:
- bash: master step1
- bash: master step2
etc
${{ else }}:
- bash: dev step1
- bash: dev step2
etc
-
or you can determine branch in the bash step and do something accordingly

Checkov scan particular folder or PR custom branch files

Trying to run Checkov (for IaC validation) via Azure DevOps YAML pipelines, for ARM template files stored in Azure DevOps version control. The code below:
trigger: none
pool:
vmImage: ubuntu-latest
stages:
- stage: 'runCheckov'
displayName: 'Checkov - Scan ARM files'
jobs:
- job: 'RunCheckov'
displayName: 'Checkov solution'
steps:
- bash: |
docker pull bridgecrew/checkov
workingDirectory: $(System.DefaultWorkingDirectory)
displayName: 'Pull bridgecrew/checkov image'
- bash: |
docker run \
--volume $(pwd):/scripts bridgecrew/checkov \
--directory /scripts \
--output junitxml \
--soft-fail > $(pwd)/CheckovReport.xml
workingDirectory: $(System.DefaultWorkingDirectory)
displayName: 'Run checkov'
- task: PublishTestResults#2
inputs:
testRunTitle: 'Checkov run results'
failTaskOnFailedTests: false
testResultsFormat: 'JUnit'
testResultsFiles: 'CheckovReport.xml'
searchFolder: '$(System.DefaultWorkingDirectory)'
mergeTestResults: false
publishRunAttachments: true
displayName: 'Publish Test results'
The problem - how to change the path/folder of ARM templates to scan. Now it scans all ARM templates found under my whole repo1, regardless what directory value I set.
Also, how to scan PR files committed to custom branch during PR review, so it would trigger the build but the build would scan only those files in the custom branch. I know how to set to trigger build via DevOps repository settings, but again, how to assure build pipeline uses/scan particular PR commit files, not whole repo1 (and master branch).
I recommend you use the Docker image bridgecrew/checkov to set up a container job to run the Checkov scan. The container job will run all the tasks of the job into the Docker container started from this image.
In the container job, you can check out the source repository into the container, then use a script task (such as Bash task) to run the related Checkov CLI to do the files scan. On the script task, you can use the 'workingDirectory' option to specify the path/folder where the command lines run in. Normally, the command lines will only act on files which are in the specified directory and its subdirectories.
If you want to only scan the files in a specific branch in the job, you can clone/checkout the specific branch to the working directory of the job in the container, then like as above mentioned, use the related Checkov CLI to scan files under the specified directory.
[UPDATE]
In the pipeline job, you can try to call the Azure DevOps REST API "Commits - Get Changes" to get all the changed files and folders for the particular commit.
Then use the Checkov CLI with the parameter --directory (-d) or --file (-f) to scan the specified file or folder.

Azure DevOps error 'Git fetch failed with exit code 1: could not read Username

I have a Git repo on Azure DevOps and I am trying to fetch the origin/main branch. However, the command failed with Bash existed code 128.
fatal: could not read Username for 'https://xxx.xxx.com': terminal
prompts disabled
##[error]Bash exited with code '128'.
This is my release pipeline definition:
# Release pipeline
pr: none
trigger: none
variables:
vmImageName: 'ubuntu-latest'
pool:
vmImage: $(vmImageName)
jobs:
- job: release
workspace:
clean: all
steps:
# Install Node
- task: NodeTool#0
inputs:
versionSpec: '16.14.x'
displayName: 'Use node 16.14.x'
# Authenticate private VivaCommonUX feed packages
- task: npmAuthenticate#0
displayName: 'npm Authenticate .npmrc'
inputs:
workingFile: .npmrc
# Install Yarn package manager
- task: geeklearningio.gl-vsts-tasks-yarn.yarn-installer-task.YarnInstaller#3
inputs:
versionSpec: '1.22.x'
displayName: 'Use Yarn 1.22.x'
# Check Node, NPM, Yarn version
- script: |
echo "Node Version: "
node -v
echo "NPM Version: "
npm -v
echoe "Yarn Version: "
yarn -v
displayName: 'Check Node, NPM, Yarn version'
- script: |
git fetch -- origin main
displayName: 'fetch origin main'
The reason why I did git fetch -- origin main is to debug the issue. It performs a git fetch -- origin main under the hood.
Any reason why this happens?
Turns out an additional step is required for the pipeline have access to the system token. More information to enable git command in Azure DevOps here.
Add a checkout session with persistCredentials set to true.
steps:
- checkout: self
persistCredentials: true

Avoid git clean with Azure Devops self-hosted Build Agent

I have a YAML build script in an Azure hosted git repository which gets triggered across 7 build agents running on a local VM. Every time this runs, the build performs a git clean which takes a significant amount of time due to a large node_modules folder which takes a long time to clean up.
The MSDN page here seems to suggest this is configurable but shows no detail of how to configure it. I can't tell whether this is a setting that should be specified on the agent, the YAML script, within DevOps on the pipeline, or where.
Is there any other documentation I'm missing or is this not possible?
Update:
The start of the YAML file is here:
variables:
BUILD_VERSION: 1.0.0.$(Build.BuildId)
buildConfiguration: 'Release'
process.clean: false
jobs:
###### ######################################################
###### 1 - Build and publish .NET
#############################################################
- job: net_build_publish
displayName: .NET build and publish
pool:
name: default
steps:
- script: echo $(BUILD_VERSION)
- task: DotNetCoreCLI#2
displayName: dotnet build $(buildConfiguration)
inputs:
command: 'build'
projects: |
myrepo/**/API/*.csproj
arguments: '-c $(buildConfiguration) /p:Version=$(BUILD_VERSION)'
The complete yaml is a lot longer, but the output from the first job includes this output in a Checkout task
Checkout myrepo#master to s
View raw log
Starting: Checkout myrepo#master to s
==============================================================================
Task : Get sources
Description : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version : 1.0.0
Author : Microsoft
Help : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Syncing repository: myrepo (Git)
Prepending Path environment variable with directory containing 'git.exe'.
git version
git version 2.26.2.windows.1
git lfs version
git-lfs/2.11.0 (GitHub; windows amd64; go 1.14.2; git 48b28d97)
git config --get remote.origin.url
git clean -ffdx
Removing myrepo/Data/Core/API/bin/
Removing myrepo/Data/Core/API/customersettings.json
Removing myrepo/Data/Core/API/obj/
Removing myrepo/Data/Core/Shared/bin/
Removing myrepo/Data/Core/Shared/obj/
....
We have another job further down which runs npm install and npm build for an Angular project, and every build in the pipeline is taking 5 minutes to perform the npm install step, possibly because of this git clean when retrieving the repository?
Click on your pipeline to show the run history
Click Edit
Click the 3 dot kebab menu
Click Triggers
Click YAML
Click Get Sources
Set Clean to False and Save
To say this is obfuscated is an understatement!
I can't say what affect this will have though, I think the agent reuses the same folder each time a pipeline runs and I'm not Node.js developer so I don't know what leaving old node_modules hanging around will do!
P.S. what people were saying about pipeline caching I don't think is what you were asking, also pipeline caching zips up the cached folder and uploads it to your artifacts storage, it then downloads it each time, if you only have 1 build agent then actually not doing a git clean might be more efficent I'm not 100%
As I mentioned below. You need to calculate hash before you run npm install. If hash is the same as the one kept close to node_modules you can skip installing dependencies. This may help you achieve this:
steps:
- task: PowerShell#2
displayName: 'Calculate and save packages.config hash'
inputs:
targetType: 'inline'
pwsh: true
script: |
# generates a hash of package-lock.json
$newHash = Get-FileHash -Algorithm MD5 -Path (Get-ChildItem package-lock.json)
$hashPath = "$(System.DefaultWorkingDirectory)/cache-npm/hash.txt"
if(Test-Path -path $hashPath) {
if(Compare-Object -ReferenceObject $(Get-Content $hashPath) -DifferenceObject $newHash) {
Write-Host "##vso[task.setvariable variable=NodeModulesAreUpToDate;]true"
$newHash > $hashPath
Write-Host ("Hash File saved to " + $hashPath)
} else {
# files are the same
Write-Host "no need to install node_modules"
}
} else {
$newHash > $hashPath
Write-Host ("Hash File saved to " + $hashPath)
}
$storedHash = Get-Content $hashPath
Write-Host $storedHash
workingDirectory: '$(System.DefaultWorkingDirectory)/cache-npm'
- script: npm install
workingDirectory: '$(Build.SourcesDirectory)/cache-npm'
condition: ne(variables['NodeModulesAreUpToDate'], true)
git clean -ffdx will clean any change untracked by source control in the source. You may try Pipeline caching, which can help reduce build time by allowing the outputs or downloaded dependencies from one run to be reused in later runs, thereby reducing or avoiding the cost to recreate or redownload the same files again. Check the following link:
https://learn.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops#nodejsnpm
variables:
npm_config_cache: $(Pipeline.Workspace)/.npm
steps:
- task: Cache#2
inputs:
key: 'npm | "$(Agent.OS)" | package-lock.json'
restoreKeys: |
npm | "$(Agent.OS)"
path: $(npm_config_cache)
displayName: Cache npm
In the checkout step, it allows us to set the boolean option clean to true or false. The default is true so it runs git clean by default.
Below is a minimal example with clean set to false.
jobs:
- job: Build_Job
timeoutInMinutes: 0
pool: 'PoolOne'
steps:
- checkout: self
clean: false
submodules: recursive
- task: PowerShell#2
displayName: Make build
inputs:
targetType: 'inline'
script: |
bash -c 'make'
More documentation and related options can be found here