Download and publish artifact only if exist in azure pipeline - azure-devops

Using task of azure pipeline is it possible to download and publish artifact only if exists?
-> My first task is a build - but that build is a bit special, it will check if there is something that have changed since the last commit. If nothing has changed, nothing will be build.
-> The second task is the download artifact, but in the case nothing was build, there will be nothing to download.
Because no folder has been created by the build, my CI failed.
Is there a way to "tell" the download task: "download only if exist".
Or continue to task 3 only if folder created otherwise finish.
Thank you

You can use for this purpose variables and set them dynamically based on the fact if you are going to create your artifact or not
steps:
- bash: |
echo "Test here if you have folder which will you use to create an artifact and then set Yes or No"
echo "##vso[task.setvariable variable=doThing]Yes" #set variable doThing to Yes
name: DetermineResult
- script: echo "Job Foo ran and doThing is Yes."
condition: eq(variables['doThing'], 'Yes')
- script: echo "Skip this one"
condition: ne(variables['doThing'], 'Yes')

Based on your requirement, you can add a task before downloading artifacts task to determine whether there is a folder, and then decide whether to run the download artifacts task.
Here are the examples:
Check if the folder exists:
steps:
- powershell: |
$Folder = '$(Build.ArtifactStagingDirectory)'
"Test to see if folder [$Folder] exists"
if (Test-Path -Path $Folder) {
echo "##vso[task.setvariable variable=test]Yes"
} else {
echo "##vso[task.setvariable variable=test]No"
}
displayName: 'PowerShell Script'
- task: DownloadBuildArtifacts#1
displayName: 'Download Build Artifacts'
inputs:
downloadType: specific
condition: ne(variables['test'], 'Yes')
Check if the folder is empty:
steps:
- powershell: |
$directoryInfo = Get-ChildItem $(Build.ArtifactStagingDirectory) | Measure-Object
$directoryInfo.count
echo $directoryInfo.count
if ( $directoryInfo.count -eq 0 )
{
echo "##vso[task.setvariable variable=test]Yes"
}
else
{
echo "##vso[task.setvariable variable=test]No"
}
displayName: 'PowerShell Script'
- task: DownloadBuildArtifacts#1
displayName: 'Download Build Artifacts'
inputs:
downloadType: specific
condition: ne(variables['test'], 'Yes')

Related

How can I check that an artifact is built in ADO

I'm building an ADO pipeline and want to check that an artifact has been created before continuing with the build stage. I'm trying to create a variable to tell me if the file exists but when I run the pipeline I get an error saying the file path is invalid.
This is the relevant ymal code:
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: $(Build.ArtifactStagingDirectory) # dist or build files
ArtifactName: 'www' # output artifact named www
- task: PowerShell#2
inputs:
script: |
$fileExists = Test-Path -Path "$(Build.ArtifactStagingDirectory)/www/dist/apps/poc/index.html"
Write-Output "##vso[task.setvariable variable=FileExists]$fileExists"
- stage: deploy
condition: and(succeeded(), eq(variables['FileExists'], True))
This is the error that I get when I run the pipeline:
##[error]Invalid file path '/home/vsts/work/1/s'. A path to a .ps1 file is required.
##[error]Invalid file path '/home/vsts/work/1/s'. A path to a .ps1 file is required.
From your YAML sample, the Powershell task definition format is not correct if you are using inline script type.
You need to define the targetType field.
For example: targetType: 'inline'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
$fileExists = Test-Path -Path "$(Build.ArtifactStagingDirectory)/www/dist/apps/poc/index.html"
Write-Output "##vso[task.setvariable variable=FileExists]$fileExists"

Azure DevOps Security Scan as part of build and pull request on only incremental changes

we are trying to build an Azure Devops pipeline (yaml based pipeline) where the build is triggered on a pull request but what we want to do is we only want to execute a step (security scanning thru veracode extension) on the incremental changes i.e. lets say I have base code with version "X" and I make few changes to it (modify couple of files, add couple of files, etc) and now whenever I execute a pull-request I want a build to be triggered but as part of the build the security scan step should be executed only on the incremental changes instead of the entire code base of the build. I am not sure how we can achieve this as part of the YAML pipeline (step) where some filter value is automatically applied (may be some REST API to get incremental changes in that build) so the step only executes on those changed files.
I have tried the git diff HEAD command with following script just to output changes from last build but this is not returning me anything
steps:
- task: AzurePowerShell#5
displayName: 'Check changed files'
inputs:
azureSubscription: 'mysubscription'
scriptType: 'InlineScript'
Inline: |
$url = "https://dev.azure.com/xxxx/KKKK/_apis/build/latest/2083?apir-version=6.0-preview.1&branchName=myevbrnch"
$response = (Invoke-RestMethod -Uri $url -Method GET -Headers $AzureDevOpsAuthenicationHeader)
$files = (git diff HEAD $response.sourceVersion --name-only)
$temp=$files -split ' '
$count=$temp.Length
echo "Total changed $count files"
For ($i=0; $i -lt $temp.Length; $i++)
{
$name=$temp[$i]
echo "this is $name file"
if ($name -like "SecOpsTest/*")
{
Write-Host "##vso[task.setvariable variable=MicroserviceAUpdated]True"
}
}
azurePowerShellVersion: 'latestVersion'
pwsh: true
Any help on how we can achieve this thru the YAML pipeline?
Thank you
I was able to get the differences between the last build and the current change, I had to enable tagging on the build and with that enabled I introduced the following powershell script to check for changes between build (found this tag comparison code written by Shamrai Alexander:
steps:
- task: AzurePowerShell#5
displayName: 'Check changed files'
inputs:
azureSubscription: 'mysubscription'
scriptType: 'InlineScript'
Inline: |
$last_tag = git describe --tags --abbrev=0 --match "[0-9]*"
$changes = git diff --name-only --relative --diff-filter AMR $last_tag HEAD .
if ($changes -is [string]){
echo "Nothing changed $changes" }
else
{
if ($changes -is [array])
{
foreach ($change in $changes){
if ($change -like '*Azure.Infra/SecOpsTest/*') {
echo "this is changed file $change"
}
}
}
}
azurePowerShellVersion: 'latestVersion'
pwsh: true

##[error]Required: 'ConnectedServiceNameARM' input --- Azure DevOps

##[error]Required: 'ConnectedServiceNameARM' input
This is in Azure DevOps using YAML inline script.
Need help with what to enter to fix this error? I am really new at YAML. This is a inline YAML and what tried seems to break the YAML script. The ConnectedServiceNameARM is just the Azure Subscription name? My service connection in azure devops has a working azure subscription name so I am wondering what is wrong?
Also need this YAML code to run so that the output file is placed in agent/_work/_tasks folder and not the artifacts folder. How would I move the file from the _tasks/Powershell folder to something that can be copied to share?
trigger:
- main
pool:
name: 'CloudUiPath001'
demands:
- agent.name -equals UiPathAgent01
steps:
- task: AzurePowerShell#3
displayName: 'Azure PowerShell script: InlineScript'
inputs:
ScriptType: InlineScript
Inline: |
$filePath='C:\Program Files (x86)\UiPath\Studio'
$dir=(New-Object -com scripting.filesystemobject).getFolder($filePath).ShortPath
$ProjectFilePath= "$(System.DefaultWorkingDirectory)/_TESTREPO7/project.json"
$ExecutableFilePath=$dir+"\UiPath.Studio.CommandLine.exe"
$OutputFilePath=".\$(Get-Date -Format 'yyyy-MM-dd-HH-mm-ss')-Workflow-Analysis.json"
#This was an attempt to write the filename to a pipeline variable: Feel free to continue on this path if possible
Write-Host "##vso[task.setvariable variable=jsonFilePath]$OutputFilePath"
Write-Output "$(Get-Date -Format 'HH:mm:ss') - STARTED - Workflow Analyzer CLI Script"
$Command = "$ExecutableFilePath analyze -p $ProjectFilePath"
Invoke-Expression $Command | Out-File -FilePath $OutputFilePath
Write-Output "$(Get-Date -Format 'HH:mm:ss') - COMPLETED - Workflow Analyzer CLI Script"
azurePowerShellVersion: LatestVersion
How do I fix his error within a INLINE YAML script. I am new to YAML and when I tried to enter a input I got errors.
##[error]Required: 'ConnectedServiceNameARM' input
According to your AzurePowerShell task definition, you don’t seem to specify the azureSubscription field.
steps:
- task: AzurePowerShell#5
displayName: 'Azure PowerShell script: InlineScript'
inputs:
azureSubscription: 'xxx'
ScriptType: InlineScript
Inline: xxx
azurePowerShellVersion: 'LatestVersion'
You can click the Settings shown in the figure below to specify the subscription.
About Azure PowerShell task, please refer to this document for details.
to go around this error change from task: AzurePowerShell#5 to pwsh:
- pwsh: |
InlineScript
displayName: 'Azure PowerShell script: InlineScript'

Access YAML variable in inline PowerShell script

If I have a YAML Pipeline;
variables:
- name : myVariable
value : 'abcd'
and if I want to run some inline powershell - how can i access that value (abcd) in the powershell script;
I expected this to work - but it didnt;
- task: PowerShell#2
displayName: "Do the thing"
inputs:
targetType: 'inline'
script:
write-host $(myVariable)
we could refer to this doc to specify variables at the pipeline, stage, or job level.
YAML build definition:
pool:
vmImage: 'vs2017-win2016'
variables:
- name : myVariable
value : 'abcd'
steps:
- task: PowerShell#2
displayName: "Do the thing"
inputs:
targetType: 'inline'
script:
write-host $(myVariable)
Result:
Not sure if this works in Azure DevOps, but in GitLab it's:
$VAR_NAME
or
$env:VAR_NAME
I'm using these two line in existing yaml and they work fine:
- $PKG_VERSION = (Get-ChildItem -Path . -Filter *.version).basename
- Write-Host $PKG_VERSION

Releasing only changed files with Pipelines

I am working on setting up a Azure DevOps project to update our website.
I set up a test website, and got it all working so that it would publish automatically whenever I did a Git Push.
The problem I'm having is that the test website has 2 files, and the real website has many many more, totalling in a little over 500MB.
I'm hoping there is a way to get it to only push out the files that changed, and not every single file.
My build pipeline is using the following script:
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: ArchiveFiles#2
displayName: 'ArchiveFiles'
inputs:
rootFolderOrFile: '$(System.DefaultWorkingDirectory)'
includeRootFolder: false
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifacts: drop'
And the release pipeline is doing a IIS Web App Deploy.
We did face the same issue and had to check a few solutions on Stackoverflow and Google and finally worked out on the following solution, which helped us pick only the last modified files based on GIT and then publish only those files instead of pushing the whole lot.
# HTML
# Archive your static HTML project and save it with the build record.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
# Updated to pick only modified files and push them for publish by adding "task: PowerShell" (to optimize CICD process)
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
#We fist check the latest modified files based on GIT DIFF command
#Then we loop through the files using files-split
#Next we split the file URL to get the file name and parent folder structure of the file separately
#Based on the file path, we check the path in output directory and create it if its not present
#Once the folder structure is available in the destination, we then copy the file to that location using copy-item command
#Force parameter is used copy-item to overwrite any existing files
script: $files=$(git diff HEAD HEAD~ --name-only);
$temp=$files-split ' ';
$count=$temp.Length;
echo "Total changed $count files";
For ($i=0; $i -lt $temp.Length; $i++)
{
$name=$temp[$i];
echo "Modified File - $name file";
$filepath = ($name.Split('/'))[0..($name.Split('/').count-2)] -join '/';
echo "File path - $filepath";
$destFile = Join-Path $(Build.ArtifactStagingDirectory) $name;
$destinationpath = Split-Path $destFile ;
echo "Destination path - $destinationpath";
if (!(Test-Path -Path $destinationpath)) {
New-Item $destinationpath -ItemType Directory
}
Copy-Item $name -Destination $destFile -Recurse -Force
}
Get-ChildItem -Path $(Build.ArtifactStagingDirectory) -Recurse -Force
- task: ArchiveFiles#2
inputs:
rootFolderOrFile: '$(Build.ArtifactStagingDirectory)'
includeRootFolder: false
- task: PublishBuildArtifacts#1
Please note that the code above works perfectly fine, only only copy pasting due to tabs/spaces at the start of every line, we need to validate the code before actually running the build.
Happy coding..!
Releasing only changed files with Pipelines
There is no such out of box method to only pick the modified files when using copy task or PublishBuildArtifacts.
As workaround, we could add powershell task to deletes all files (recursive) which have timestamp that is < (Now - x min), with this way Artifact directory contains of ONLY CHANGED files.
Check the details here: TFS 2017 - how build/deliver only changed files?.
I tried to develop this script using my lame PowerShell technology, but I did not success.
Hope this helps.
I was able to create an ADO powershell task based on the suggestion of another answer in this thread.
- task: PowerShell#2
inputs:
targetType: 'inline'
script: 'get-childitem $(build.artifactStagingDirectory) -recurse | Foreach {
$lastupdatetime=$_.LastWriteTime;
$nowtime=get-date;
if(($nowtime - $lastupdatetime).totalhours -le 24) {
Write-Host $_.fullname
}
else{
remove-item $_.fullname -Recurse -Force
}
}'
failOnStderr: true
workingDirectory: '$(build.artifactStagingDirectory)'
The script portion of the task should be in one line, but I expanded it for easier reading.
If you are getting error "unknown revision" then increase the shallow fetch depth of your Azure DevOps YAML pipeline. See the official documentation.