I have an Azure DevOps pipeline step failing running the OWASP dependency check. I want to find what dependencies need to be updated.
The logs that are written during the dependency check pipeline step say:
[INFO] Writing report to: e:\vsts\a\7567\TestResults\dependency-check\dependency-check-report.html
I assume this dependency-check-report.html is where it will tell me what dependencies need to be updated. But I do not understand where this e:\vsts\a\7567\TestResults\ location is, as this step is being run in DevOps. Is this somewhere in DevOps? I cannot seem to find it anywhere. "Download logs" on the pipeline page doesn't seem to have it either.
where this e:\vsts\a\7567\TestResults\ location is
When you run the pipeline in Azure DevOps, this path represents the local path of the machine where the agent locates.
In your case, the agent is self-hosted agent. You go to the local machine where the agent locates and find the dependency-check-report.html in e:\vsts\a\7567\TestResults\dependency-check.
On the other hand, you can use the Publish Pipeline Artifacts task to upload the target file to Pipeline artifacts.
For example:
steps:
- task: dependency-check-build-task#6
displayName: 'Dependency Check'
inputs:
projectName: test
scanPath: test
continueOnError: true
- task: PublishPipelineArtifact#1
displayName: 'Publish Pipeline Artifact'
inputs:
targetPath: '$(Common.TestResultsDirectory)'
artifact: drop
Note: You need to set the continueOnError: true in OWASP dependency check task.
In this case, the dependency-check-report.html on agent machine will be uploaded to Azure Artifacts.
For example:
Related
I have an Azure DevOps Build pipeline that publishes the entire repository as an artifact to be used with the Release pipeline.
# Publish artifacts to be used in release
- task: PublishBuildArtifacts#1
displayName: 'publish artifacts'
inputs:
PathtoPublish: '$(System.DefaultWorkingDirectory)'
ArtifactName: 'TerraformModule'
publishLocation: 'Container'
The build pipeline triggers the creation of a release pipeline where I try to deploy the terraform configuration.
I can successfully run terraform init in this pipeline but when I try to run plan or apply, I get the following error:
Looking at the screenshot, it looks like it tries to execute the command from /usr/local/bin instead of what I specified in the step? Confused by this. Below is the yaml for my plan step:
steps:
- task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV3#3
displayName: 'terraform plan'
inputs:
provider: aws
command: plan
workingDirectory: '/home/vsts/work/r1/a/_terraform/TerraformModule/Projects/Potentium/Prod'
environmentServiceNameAWS: 'AWS-Terraform-Build'
I manually changed workingDirectory to where the Artifacts from the build pipeline were downloaded to. See log below for example:
2022-08-14T23:41:31.3359557Z Downloaded TerraformModule/Projects/Potentium/Prod/main.tf to /home/vsts/work/r1/a/_terraform/TerraformModule/Projects/Potentium/Prod/main.tf
The plan step in my build pipeline executes without any issues so I have a feeling it is something to do with the artefacts/extraction that is occurring in the download step. Looking for any advice.
I've had similar issues with the extraction phase, when using ExtractFiles#1 doing a similar thing with terraform. I think there's a bug in it, I could not get it to extract files back to the root of System.DefaultWorkingDirectory unless the root folder was included in the archiv, I am using ArchiveFiles#2. So I was ending up with /opt/az_devops/_work/*/s/s
My solution, was to shell out a command to do the extraction. No problems extracting to the root of System.DefaultWorkingDirectory
Just remember if you're running a subsequent terraform plan, by default the working directory System.DefaultWorkingDirectory will change between runs. So ensure you use these variables rather than an explicit reference.
Is there any possibility to get the URL of a published artifact in the yml pipeline, so it can be used in further pipeline tasks/steps?
Sadly, the Microsoft Docs on the two tasks don't give any hints if the published path value is available in any way.
- task: PublishBuildArtifacts#1
inputs:
pathToPublish: report.html
artifactName: HtmlReport
It depends on where you're using the artifacts - Deployment jobs will typically automatically download the artifacts into the $(Pipeline.Workspace) folder with the same name as you declare in the build task.
So in your case it would be located at $(Pipeline.Workspace)\HtmlReport
You can also use the Download Build Artifacts task to download a specific artifact:
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'HtmlReport`
This is useful if you have multiple published artifacts and you only want to download one of them in a later stage. There are other options if you wish to download an artifact from a different pipeline.
Note that the Publish Build Artifacts task is now deprecated and you are recommended to use the newer Publish Pipeline Artifacts and matching Download Pipeline Artifacts tasks:
We recommend upgrading from build artifacts (PublishBuildArtifacts#1 and DownloadBuildArtifacts#0) to pipeline artifacts (PublishPipelineArtifact#1 and DownloadPipelineArtifact#2) for faster performance.
In azure devops i am trying to create a multistage release definition via yaml. Build is done via classic editor and the artifacts are uploaded to azure pipelines. so i want to access a specific artifact for deployment
- task: DownloadPipelineArtifact#2 displayName: 'Download Pipeline Artifact' inputs: buildType: specific project: 'vvxxxxxx-vxxv-xxxv-vxxx-xxxxxxvvxxvv' definition: 5 buildVersionToDownload: specific pipelineId: 'SSE_XXXXXXXXXXXXXXXXXX_Auto-import_dev_20200423.4' artifactName: Service targetPath: '$(Pipeline.Workspace)'
When i try it via classic release using task 'Download Pipeline Artifact' it's successful but when I try it via yaml it's failing with error "##[error]Run Id is not valid: SSE_XXXXXXXXXXXXXXXXXX_Auto-import_dev_20200423.4"
if there is anyother way to get the artifact from a pipeline would be helpful and also instead of hardcoding pipelineId I want to make it dynamic as well.
Download Artifact from other pipeline in Multistage YAML
The value of the pipelineId should be the ID of the build pipeline, which you want to download, rather than the name/title of the build pipeline.
Find the build pipeline you want to download, click on a build record you want to download, you could see it in the web address bar of the browser:
also instead of hardcoding pipelineId I want to make it dynamic as
well.
If you don't want hard code the pipelineId/runid in YAML definition, you can consider to pass queue variable as a work around.
For example:
- task: DownloadPipelineArtifact#2
inputs:
source: 'specific'
artifact: 'drop'
path: $(Build.SourcesDirectory)/bin
project: 'AndroidBuild'
pipeline: 12
runVersion: 'specific'
runId: $(buildid)
In above definition, buildid is the variable, and you can configure its value at queue time:
This do not need you to do any modification to the pipeline when you want to choosing another runId, just pass the value at queue time.
Hope this helps.
I'm learning how to build the infrastructure provisioning pipelines in Azure DevOps and looking for some step-by-step tutorial to deploy for instance the vnet and a VM in Azure using both methods - ARM templates as well as Terraform
Any help highly appreciated,
Thanks,
Andrey
ARM template:
Firstly, you need have one template which it can be deployed to azure to create VM and Vnet. Just refer to this sample: azure-quickstart-templates. Then do some changes into the JSON scripts based on your actual demands.
The CI/CD structure in azure devops also very easy.
1) Build:
Since the template scripts do not need any build process, it only need 2 tasks to pack and publish the repos as artifact. So that release can use them.
- task: CopyFiles#2
displayName: 'Copy Files to: $(build.artifactstagingdirectory)'
inputs:
SourceFolder: '$(Build.SourcesDirectory)'
TargetFolder: '$(build.artifactstagingdirectory)'
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: drop'
inputs:
PathtoPublish: '$(build.artifactstagingdirectory)'
2) Release:
In release pipeline, it just need one task: Azure Resource Group Deployment task.
This blog has detailed description on how to configure the task.
Terraform:
Same with ARM, here you also need one terraform scripts.
1) Build
For tefrraform, the build configuration same with ARM. Just need publish artifacts.
2) Relesase
See this detailed blog: Terraform deployment with Azure DevOps
I've been struggling with an issue in Azure DevOps where I can build successfully an artifact through a build pipeline (I use the publish artifact task). I can see that the artifact is built successfully, as I can download it and I can reference it later in the release pipeline.
The issue is that when the hosted agent is started, no artifacts are downloaded to the machine, and my $(System.DefaultWorkingDirectory) is always empty. I am out of ideas on why in the initialization of the hosted agent, no artifacts are actually being downloaded....
I had the same issue this week. I set up my build with the 'Publish pipeline artifact' task and ran into the same issue as you, i replaced this with 'Publish build artifact' task in my build and everything worked out then.
I've read that 'Publish pipeline artifact' is only intended for uploading/downloading artifacts within the same pipeline, so thats why this won't get downloaded automatically in your release. Alternatively you could use the 'Download Pipeline Artifact' task in your release to explicitly download your artifact