We have an Azure DevOps pipeline which uses self hosted Windows agents with Azure DevOps server 2019. The pipeline runs our front-end tests without any problems. However, occasionally our linting step will find problems that it throws as warnings (such as unused variables). This is what we want it to do but the issue is that these warnings were not being elevated. So the only way to see them was to look in the build execution.
This we were able to resolve by adding a vso formatter to the linting command: npm run nx run-many -- --target="lint" --all --skip-nx-cache=true --parallel --format=vso. So now the warnings are thrown like this:
As shown in the green box the warnings are displaying properly. However, in the red circles the status of the build, job, and linting task are success. Is there a way I can mark this build, job, and task as warning so we know to take a further look? Thank you for any help, please let me know if I can provide additional information.
You can add a powershell task at the end of the pipeline, and then run the Rest API(Timeline - Get) to traverse the warning messages in the previous task. Finally, you can use the logging command to set the pipeline status
Here is PowerShell Sample:
$token = "PAT"
$url=" https://{instance}/{collection}/{project}/_apis/build/builds/$(build.buildid)/timeline?api-version=5.0"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($token)"))
$count = 0
$response = Invoke-RestMethod -Uri $url -Headers #{Authorization = "Basic $token"} -Method Get -ContentType application/json
ForEach( $issues in $response.records.issues )
{
if($issues.type -eq "warning")
{
echo $issues.Message
$count ++
}
}
echo $count
if($count -ne 0 )
{
Write-Host "##vso[task.complete result=SucceededWithIssues;]"
}
Result:
There is Azure DevOps Extension/Task created by Microsoft "Build Quality Checks"
So you can use it to set up an additional option to force specific build qulity.
Related
On Azure DevOps I have a build/deploy YAML pipeline which builds my Function App and deploys this to a Linux Function App in Azure (hosted on an app-service plan).
However, recently I noticed an issue where the pipeline was showing the function app was successfully deployed, but when going into the FunctionApp in Azure and clicking on the Functions tab on the left, it shows "No results", but there should be 4 functions in there.
I want to have a step at the end of the deploy pipeline which checks that 4 functions exist in this FunctionApp, and to fail the pipeline run if this is not the case.
I know this will most likely be a task in the pipeline using Azure CLI or Powershell, but not sure how I would go around writing the script. Any help would be immensely appreciated.
Thanks all,
You could use an Azure CLI task and call the Rest API to list the functions withint the FunctionApp. I don't think the native Az CLI and Azure PowerShell functions expose the functions within the FunctionApp/
URI = "https://management.azure.com/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.Web/sites/<functionAppName>/functions?api-version=2015-08-01"
az rest -u $URI --method get | jq '.value[].name'
steps:
- powershell: |
#get token
$TENANTID="xxx"
$APPID="xxx"
$PASSWORD="xxx"
$result=Invoke-RestMethod -Uri https://login.microsoftonline.com/$TENANTID/oauth2/token?api-version=1.0 -Method Post -Body #{"grant_type" = "client_credentials"; "resource" = "https://management.core.windows.net/"; "client_id" = "$APPID"; "client_secret" = "$PASSWORD" }
$token=$result.access_token
##set Header
$Headers=#{
'authorization'="Bearer $token"
'host'="management.azure.com"
}
$functions = Invoke-RestMethod -Uri "https://management.azure.com/subscriptions/<subcription id>/resourceGroups/<resource group name>/providers/Microsoft.Web/sites/<function app name>/functions?api-version=2015-08-01" -Headers $Headers -ContentType "application/json" -Method GET
if($functions.value.Count -eq 4) {
# make pipeline to succeed
Write-Host 'Function deployment success.'
exit 0
}
else {
Write-Host 'Function deployment failed.'
exit 1
}
displayName: 'Check whether the function app deployment completed.'
If you have concerns, let me know.
I've been scouring for the answer to this for the last several days. There's many fields labeled "status" in the TC API and for the life of me, I can't figure out which one returns the status listed above.
I'm currently calling...
$status = (Invoke-RestMethod -Uri %teamcity.serverUrl%/httpAuth/app/rest/builds/id:%teamcity.build.id% -Method Get -Headers $header).build.status
...within PowerShell. This returns SUCCESS unless the build has an execution timeout. However, as you can see from the screenshot above, the build failed. Obviously, the call above doesn't return the right status. The failure is due to a specific build step failing. If any of the build steps fail, then the entire build fails. Not sure if that helps!
The problem doesn't seem to be in the endpoint itself. You are requesting the correct field. The same can also be done with the following requests:
GET /app/rest/builds?locator=id:%teamcity.build.id%&fields=build(status)
GET /app/rest/builds/%teamcity.build.id%/status
The first request will return you an XML or JSON structure depending on your Accept header. The second response will return you plain text, so set the headers accordingly.
This returns SUCCESS unless the build has an execution timeout. However, as you can see from the screenshot above, the build failed. Obviously, the call above doesn't return the right status. The failure is due to a specific build step failing. If any of the build steps fail, then the entire build fails.
You are requesting the build status during the build execution. It is possible, that when the build step that checks the status is running, the status of the build is still SUCCESS. Try moving the build step to the very end of this build and configure it to execute 'Always, even if build stop command was issued'
Ok, so this is what was happening. I have an build step in TeamCity that runs Katalon (our test automation software). Even though the tests were failing, TeamCity was not updating the build status until after the last step, which happened to be my PS script that called the API then sent the info over to MS Teams. I had to setup a Failure Condition in TeamCity for the Katalon build step that looks in the build log for the specific text "(Katalon) failed". This immediately fails the build, but the rest of the build steps continue to run. Now, when my last step runs, it pulls the correct status and pushes it to MS Teams. For anyone that is interested, this is the full PowerShell script for that build step.
function Get-AuthenticationHeader
{
param($User, $Password)
$pair = "$($User):$($Password)"
$encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($pair))
"Basic $encodedCreds"
}
$authHeader = Get-AuthenticationHeader -User "%system.teamcity.auth.userId%" -Password "%system.teamcity.auth.password%"
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls -bor [Net.SecurityProtocolType]::Tls11 -bor [Net.SecurityProtocolType]::Tls12
$header = #{
"Authorization"="$authHeader"
}
$response = (Invoke-RestMethod -Method GET -Headers $header -ContentType "application/json" -Uri "%teamcity.serverUrl%/httpAuth/app/rest/builds/%teamcity.build.id%")
$status = $response.build.status
$name = "%system.teamcity.buildConfName%"
$teamcityAgent = "%teamcity.agent.name%"
$testResults = $response.build.webUrl
Write-Host "%teamcity.build.id%"
Write-Host $status
Write-Host $name
Write-Host $testResults
if ( $status -eq "SUCCESS" ) {
$color = "00ff00"
}
else {
$status = "FAILURE"
$color = "ff0000"
}
$body = #{
title= "API Test Automation";
text= "<pre><b><i><font color=$color>$status</font></i></b> - $name<br>Tests ran on <i>$teamcityAgent</i><br>Test results are HERE</pre>"
} | ConvertTo-Json
Invoke-WebRequest -Method POST -ContentType "application/json" -Uri "<YOUR MS TEAMS WEBHOOK HERE>" -Body $body -UseBasicParsing
I have a pipeline with a number of stages, e.g.:
Canary -> Dev -> UAT -> Prod
For each stage we have approvals set so a team member has to allow a build to proceed to the next stage. Sometimes we have long periods between stages being approved (e.g. someone needs to do some testing and feedback outside of devops we can proceed), therefore it can be hard to remember we have an old run pending approval unless we specifically check - which is prone to human error.
But if I have one build say queued at UAT awaiting push to production how can I prevent a second run which includes the changes from the first run (as they are both running off master) being allowed to proceeded.
E.g. I need newer jobs to be aware of older runs still pending approval to proceed.
Thanks!
There is no default way to prevent newer runs if old one queued, but you can try the following workaround:
You could add a script in the pipeline to check state of the latest pipeline run. If the state is inProgress, cancel the build. If the state is completed, proceed the build.
You should use Runs - List api to get state of the latest pipeline run. The script looks like:
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $user,$token)))
$uri = "https://dev.azure.com/$org/$projectName/_apis/pipelines/$pipelineid/runs?api-version=6.0-preview.1"
$result = Invoke-RestMethod -Uri $uri -Method Get -Body $json -ContentType "application/json" -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)}
$latestrun = $result.value.state[0]
Write-Host "$latestrun"
Then you could refer the following case to cancel the build if $latestrun is inProgress:
Is it possible to cancel a Azure DevOps pipeline Job programmatically?
I'm having issues when trying to display all runs for one of my Pipelines. I'm running on premise Azure Devops 2020 and I get a perpetual spinning Loading dialogue. I don't have this issue in any of my other pipelines. It seems that if I filter by run state in the problematic pipeline I am able to get past this and view some of the runs but for other states in my case "Succeeded" and "Succeeded with Issues" I continue to get the spinning loading symbol. Any advice?
Here are some suggestions:
Suggestion 1
Clean the cache or load the page from a different brower and restart the Azure DevOps Server and the SQL Server machine.
Suggestion 2
Create a new pipeline that has the same settings as the affected pipeline.
Suggestion 3
You can use the REST API Builds - List to get all runs of your pipeline.
GET https://dev.azure.com/{organization}/{project}/_apis/build/builds?api-version=6.0
Because the cause of this question may be that the runs list is too long. You can use the REST API Builds-Delete to delete some of the runs you don't need to see whether the question can be solved.
DELETE https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}?api-version=6.0
Suggestion 4
You can refer to a similar question on the Developer Community. The Jan Selbach's comment offers a solution.
1.Run the following SQL script to find the LeaseId of duplicate rows. Please change collection Db name according to your’s.
SELECT LeaseId
FROM [AzureDevOps_DefaultCollection].[Build].[tbl_RetentionLease]
WHERE
partitionId > 0 AND
LeaseId NOT IN (
SELECT MIN(LeaseId) as LeaseId
FROM [AzureDevOps_DefaultCollection].[Build].[tbl_RetentionLease]
WHERE PartitionId > 0
GROUP BY OwnerId,RunId
)
2.Concatenate the LeaseIds into a comma separated list
3.Run the following PowerShell script by filling in the org, project, pat, and leaseId by passing in the comma separated list of LeaseIds
(remember if on-prem change the https://dev.azure.com in the $uri to
your on-prem server)
$org = "";
$project = "";
$pat = "";
$encoded = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes([string]::Format("{0}:{1}", "", $pat)));
$accessToken = "Basic $encoded";
$leaseId = "";
$uri = "https://dev.azure.com/$org/$project/_apis/build/retention/leases?ids=$leaseId&api-version=6.0-preview";
try {
$response = Invoke-RestMethod -uri $uri -method DELETE -Headers #{ Authorization = $accessToken } -ContentType "application/json"
$response
}
catch {
$errorDetails = ConvertFrom-Json $_.ErrorDetails
Write-Host "StatusCode: $($_.Exception.Response.StatusCode)`nExceptionType: $($errorDetails.typeKey)`nExceptionMessage: $($errorDetails.message)"
#StackTrace: $($errorDetails.stackTrace)"
}
Our build artifact is an Octopus nuget package. When the build is released it lands into the QA stage where the artifact is deployed through Octopus. This octopus consumes it directly from the Azure Artifacts nuget feed.
If the deployment and the subsequent tests are successful we want to promote the artifact to the Release view of the Azure Artifacts nuget feed, because we think it gives us a different nuget URL that can be used by another Octopus serving the next stage (for historic reasons we have dedicated Octopus per stage - working to change that, but it takes time).
We can promote manually, but we want to do it automatically. How can this be done?
We are testing it on on-premises TFS 2019 RC2.
EDIT 1
The suggested plugin does not seem to install on on-premises TFS 2019 RC2:
Using PowerShell...
$organisationName = '' # Name of organisation
$projectName = '' # Name of project
$feedName = '' # Name of Azure Artifacts feed
$viewName = 'Release' # I believe this can also be Prerelease, but I've not tried it
# List of names of packages within Azure Artifacts feed to be promoted
$packagesToPromote = #('')
# Need a personal access token for this script to work
# PAT token should be assigned to Packaging (Read, Write and Manage) scopes
$azureArtifactsPAT = ''
$AzureArtifactsPAT_Base64 = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($azureArtifactsPAT)"))
$restAPICallHeader = #{ Authorization = "Basic $AzureArtifactsPAT_Base64" }
$feedBaseURL = "https://feeds.dev.azure.com/$organisationName/$projectName/_apis/packaging/feeds"
$packageBaseURL = "https://pkgs.dev.azure.com/$organisationName/$projectName/_apis/packaging/feeds"
$feedIdURL = "$feedBaseURL/$feedName/?api-version=5.1-preview.1"
$feedIdResponse = (Invoke-RestMethod -Method Get -Uri $feedIdUrl -Headers $restAPICallHeader -ContentType 'application/json')
$feedId = $feedIdResponse.id
$viewIdURL = "$feedBaseURL/$feedId/views/$viewName/?api-version=5.1-preview.1"
$viewIdResponse = (Invoke-RestMethod -Method Get -Uri $viewIdUrl -Headers $restAPICallHeader -ContentType 'application/json')
$viewId = $viewIdResponse.id
$restAPICallBodyJson = #{
views = #{
op = 'add'
path = '/views/-'
value = "$viewId"
}
}
$restAPICallBody = (ConvertTo-Json $restAPICallBodyJson)
foreach ($packageName in $packagesToPromote) {
$packageQueryUrl = "$feedBaseURL/$feedId/packages?api-version=5.1-preview.1&packageNameQuery=$packageName"
$packagesResponse = (Invoke-RestMethod -Method Get -Uri $packageQueryUrl -Headers $restAPICallHeader -ContentType 'application/json')
$latestPackageVersion = ($packagesResponse.value.versions | ? { $_.isLatest -eq $True } | Select -ExpandProperty version)
$encodedPackageVersion = [System.Web.HttpUtility]::UrlEncode($latestPackageVersion)
Write-Host "Package Name: $packageName"
Write-Host "Package Version: $latestPackageVersion"
$releaseViewURL = $packageBaseURL `
+ "/$($feedId)" `
+ "/nuget/packages/$packageName" `
+ "/versions/$encodedPackageVersion" `
+ "?api-version=5.1-preview.1"
$response = Invoke-RestMethod -Method Patch -Uri $releaseViewURL -Headers $restAPICallHeader -ContentType 'application/json' -Body $restAPICallBody
Write-Host $response
}
For reference, the script above uses the following API calls:
Feed Management - Get Feed
https://learn.microsoft.com/en-us/rest/api/azure/devops/artifacts/feed%20%20management/get%20feed?view=azure-devops-rest-5.1
Feed Management - Get Feed View
https://learn.microsoft.com/en-us/rest/api/azure/devops/artifacts/feed%20%20management/get%20feed%20view?view=azure-devops-rest-5.1
Artifact Details - Get Packages
https://learn.microsoft.com/en-us/rest/api/azure/devops/artifacts/artifact%20%20details/get%20packages?view=azure-devops-rest-5.1
NuGet - Update Package Version
https://learn.microsoft.com/en-us/rest/api/azure/devops/artifactspackagetypes/nuget/update%20package%20version?view=azure-devops-rest-5.1
As per Azure DevOps documentation the marketplace task Promote package to Release View is the recommended way to accomplish this from a CI/CD pipeline.
The repository can be found on Github.
Edit:
Since you are on-prem with a version that this task doesn't support. I would say that the comments about using the REST api would be the route you need to go in something like a powershell script.
Having never used the REST Api for this task I'm not exactly sure how the body is supposed to look for the request. However, it seems to be documented here.
My understanding of the JSON Patch object is limited, but I would think you might use the replace operation.
{ "op": "replace", "path": "/view", "value": "#Release" }
This article may also be helpful, but I still don't see anything that would relate to the from identifier on the JsonPatchObject definition in the REST Api documentation.
I also recently struggled with trying to implement version using TFS. I've produced some PowerShell scripts (adapting other scripts out on the web) to do package versioning.
https://gist.github.com/oceanexplorer/6a91930419b35c1923974af265777a5f
https://gist.github.com/oceanexplorer/35e0f26962018dc8578c745060365c15
The first step is my build pipeline I use the "Update AssemblyInfo" task to set the build version which then gets embedded into the DLL's.
https://marketplace.visualstudio.com/items?itemName=sebastianlux.UpdateAssemblyInfo
Initially I embedded the above scripts with my project to get things going but eventually in my release pipeline I then have a task that deploys these build scripts via a "NuGet Install" task which effectively pulls them from a feed and unzips them.
In the release pipeline I then have a task "Version Package" which is a custom PowerShell script that calls functions defined in the two gists above, what these do is to unzip the NuGet packages that have been created from the build pipeline and placed in the artifact directory, applies the correct versioning to the package and zips it back up. I have used the following build number format in my build pipeline:
$(version.major).$(version.minor).$(version.patch).$(Date:yyyyMMdd)$(Rev:r)-CI
1.0.0.201902051-CI
This will produce a semantic build number format of:
1.0.0-alpha.201902051
I call the scripts using an inline PowerShell task
##-------------------------------------------
## Import Build Scripts
##-------------------------------------------
gci -Recurse "$(System.DefaultWorkingDirectory)\scripts\*.psm1" | ForEach-Object { Import-Module $_.FullName }
##-------------------------------------------
## Version Files
##-------------------------------------------
Expand-NugetPackages -packagesDirectory "$(artifact.directory)" -Verbose
Add-VersionToAssemblies -suffix "$(Release.EnvironmentName)" -semVer "2.0" -artifactsToApplyTo "nuspec" -isRelease $(isRelease) -Verbose
Compress-NugetPackages -packagesDirectory "$(artifact.directory)" -Verbose
Then a NuGet push task to push the package
Them another inline PowerShell script which sets the release view for the package feed:
##-------------------------------------------
## Import Build Scripts
##-------------------------------------------
gci -Recurse "$(System.DefaultWorkingDirectory)\scripts\*.psm1" | ForEach-Object { Import-Module $_.FullName }
##-------------------------------------------
## Set Package Quality
##-------------------------------------------
Set-PackageQuality -feedName "Libraries" -packageId $(nuget.packageId) -packageVersion $env:semanticVersion -packageQuality $(Release.EnvironmentName)
Something changed recently in the ADO RestApi.
What worked for me was sending a PATCH request like:
curl --location --request PATCH 'https://pkgs.dev.azure.com/YOUR_ORGANIZATION/YOUR_PROJECT/_apis/packaging/feeds/YOUR_FEEDNAME/upack/packages/YOUR_PACKAGENAME/versions/YOUR_PACKAGEVERSION?api-version=5.1-preview.1' \
--header 'Authorization: Basic YOUR_TOKEN' \
--header 'Content-Type: application/json' \
--data-raw '{
"views": {
"op": "add",
"path": "/views/-",
"value": "Release"
}
}'