Publish files with UniversalPackages#0 - "The path provided is invalid." - azure-devops

I'm trying to publish a file (Helloworld.txt) to my Universal Packages directory to pass on to a different stage within my Release Piepline (using UniversalPackages#0).
I cannot for the life of me figure out how to make it work.
I'm trying to follow this KB but it leaves me confused:
https://learn.microsoft.com/en-us/azure/devops/pipelines/artifacts/universal-packages?view=azure-devops&tabs=yaml
For example, I can't determine what vstsFeedPublish is and don't know if vstsFeedPackagePublish can be a new name I make up on the spot or if it is actually the name of an existing file/folder.
What "path" am I entering incorrectly that's making it fail?
Stage Deployment Steps:
Helloworld Output to TXT:
cd $(Build.ArtifactStagingDirectory)
"$(System.DefaultWorkingDirectory)/Helloworld Build/Helloworld EXE Folder/Helloworld.exe" >> WriteLineOutput.txt
I cd into $(Build.ArtifactStagingDirectory) first because that is the default folder UniversalPackages#0 looks to publish files, so I want WriteLineOutput.txt to be generated there.
Publish Output TXT to Universal Dir (YAML):
steps:
- task: UniversalPackages#0
displayName: 'Publish Output TXT to Universal Dir'
inputs:
command: publish
vstsFeed: '0a3a9abd-83fd-495f-967b-e986c523f2d2'
vstsPackageVersion: 1
vstsFeedPublish: '0a3a9abd-83fd-495f-967b-e986c523f2d2'
vstsFeedPackagePublish: 'writelineoutput-txt'
versionOption: minor
packagePublishDescription: 'TXT output from Helloworld.exe'
Working Directory Structure:
"Publish Output TXT to Universal Dir" Task Output:
2019-12-02T03:35:35.2264352Z ##[section]Starting: Publish Output TXT to Universal Dir
2019-12-02T03:35:35.2381247Z ==============================================================================
2019-12-02T03:35:35.2381328Z Task : Universal packages
2019-12-02T03:35:35.2381359Z Description : Download or publish Universal Packages
2019-12-02T03:35:35.2381412Z Version : 0.160.1
2019-12-02T03:35:35.2381461Z Author : Microsoft Corporation
2019-12-02T03:35:35.2381490Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks
2019-12-02T03:35:35.2381520Z ==============================================================================
2019-12-02T03:35:36.0651445Z SYSTEMVSSCONNECTION exists true
2019-12-02T03:35:36.6058302Z Downloading: https://0t3vsblobprodcus362.vsblob.vsassets.io/artifacttool/artifacttool-win10-x64-Release_0.2.128.zip?sv=2017-04-17&sr=b&sig=tHZQU3V2DuXcC0Y1xnmzB7Zw7kMdjJSijWVDiztc9UE%3D&spr=https&se=2019-12-02T04%3A35%3A37Z&sp=r&P1=1575261037&P2=11&P3=2&P4=LS6Ffab5P%2bb8Q9r3aGsGLlK9ELRD6bRxxlTkDc5aEc8%3d
2019-12-02T03:35:39.3862184Z Caching tool: ArtifactTool 0.2.128 x64
2019-12-02T03:35:40.4414172Z SYSTEMVSSCONNECTION exists true
2019-12-02T03:35:41.4415128Z Publishing package: writelineoutput-txt, version: 0.1.0 using feed id: 65dc653c-5c3b-771c-b308-34b199d8fcee, project: null
2019-12-02T03:35:41.4469092Z [command]C:\hostedtoolcache\windows\ArtifactTool\0.2.128\x64\ArtifactTool.exe universal publish --feed 65dc653c-5c3b-771c-b308-34b199d8fcee --service https://vsrm.dev.azure.com/sawtooth-capstone/ --package-name writelineoutput-txt --package-version 0.1.0 --path D:\a\r1\a\$(Build.ArtifactStagingDirectory) --patvar UNIVERSAL_PUBLISH_PAT --verbosity None --description "TXT output from Helloworld.exe"
2019-12-02T03:35:44.0492154Z {"#t":"2019-12-02T03:35:43.3152933Z","#m":"ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 23e076ea-5122-4c06-b92a-2aef5974defd","#i":"8778ba0f","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.315Z"}
2019-12-02T03:35:44.0493421Z {"#t":"2019-12-02T03:35:43.7170274Z","#m":"Ensuring that the package does not yet exist...","#i":"40e01e14","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.717Z"}
2019-12-02T03:35:44.0494251Z {"#t":"2019-12-02T03:35:43.8900269Z","#m":"Package does not yet exist","#i":"c781eca5","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.890Z"}
2019-12-02T03:35:44.0494769Z {"#t":"2019-12-02T03:35:43.8913512Z","#m":"Pushing content...","#i":"3aa40378","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.891Z"}
2019-12-02T03:35:44.0495859Z {"#t":"2019-12-02T03:35:43.9060246Z","#m":"DedupManifestArtifactClient will correlate http requests with X-TFS-Session 23e076ea-5122-4c06-b92a-2aef5974defd","#i":"09a6f3ce","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.906Z"}
2019-12-02T03:35:44.0496478Z {"#t":"2019-12-02T03:35:44.0301714Z","#m":"The path provided is invalid.","#i":"05178f7d","#l":"Error","SourceContext":"ArtifactTool.Program","UtcTimestamp":"2019-12-02 03:35:44.030Z"}
2019-12-02T03:35:44.0776082Z ##[error]Error: An unexpected error occurred while trying to push the package. Exit code(16) and error({"#t":"2019-12-02T03:35:43.3152933Z","#m":"ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 23e076ea-5122-4c06-b92a-2aef5974defd","#i":"8778ba0f","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.315Z"}
{"#t":"2019-12-02T03:35:43.7170274Z","#m":"Ensuring that the package does not yet exist...","#i":"40e01e14","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.717Z"}
{"#t":"2019-12-02T03:35:43.8900269Z","#m":"Package does not yet exist","#i":"c781eca5","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.890Z"}
{"#t":"2019-12-02T03:35:43.8913512Z","#m":"Pushing content...","#i":"3aa40378","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.891Z"}
{"#t":"2019-12-02T03:35:43.9060246Z","#m":"DedupManifestArtifactClient will correlate http requests with X-TFS-Session 23e076ea-5122-4c06-b92a-2aef5974defd","#i":"09a6f3ce","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.906Z"}
{"#t":"2019-12-02T03:35:44.0301714Z","#m":"The path provided is invalid.","#i":"05178f7d","#l":"Error","SourceContext":"ArtifactTool.Program","UtcTimestamp":"2019-12-02 03:35:44.030Z"})
2019-12-02T03:35:44.0789627Z ##[error]Packages failed to publish
2019-12-02T03:35:44.0898947Z ##[section]Finishing: Publish Output TXT to Universal Dir
Key Lines:
2019-12-02T03:35:44.0496478Z {"#t":"2019-12-02T03:35:44.0301714Z","#m":"The path provided is invalid.","#i":"05178f7d","#l":"Error","SourceContext":"ArtifactTool.Program","UtcTimestamp":"2019-12-02 03:35:44.030Z"}
2019-12-02T03:35:44.0776082Z ##[error]Error: An unexpected error occurred while trying to push the package. Exit code(16) and error({"#t":"2019-12-02T03:35:43.3152933Z","#m":"ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 23e076ea-5122-4c06-b92a-2aef5974defd","#i":"8778ba0f","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.315Z"}
EDIT:
Well, I read it's best practice to use the Build pipeline to publish artifacts and the Release pipeline to run/test them, so I decided to just publish the artifact from the Build pipeline instead of trying to make the same Helloworld.txt again during the Release pipeline. But I'm willing to take the inevitable downvotes if someone can help me figure out how to publish new files created in the Release pipeline into the Universal packages directory.

I'd better suggest you call exe to generate the txt file in Build. Then publish them as artifacts to release pipeline. Then you can easily pick the HelloWord.txt file in Release pipeline.
This task can be used both in Build and Release pipeline. The difference between apply this task in Build and release just be the Path to file(s) to publish value which you very confused.
For Release pipeline:
If what you want is publish the HelloWord.txt into package, just click , and then select the corresponding file HelloWord.txt from it.
So, what its value should be $(System.DefaultWorkingDirectory)/{Source alias}/{artifact name}/s/HelloWorld.txt.
Note: This way is applied when you put the txt generated in Build pipeline, and then publish them as artifact and used in release pipeline.
Similar in Build pipeline, you can also select the file from that button. For its YAML format, just input the file name like:
inputs:
command: publish
publishDirectory: 'azure-pipelines.txt'
vstsFeedPublish: '*****'
vstsFeedPackagePublish: merlin
versionOption: custom
versionPublish: 0.0.1
packagePublishDescription: published in 2019/12/2
If azure-pipelines.txt file is under a folder, just input it as a {folder name}/azure-pipelines.txt.
In your issue, if you want to put the file generate and the package publish both in Build pipeline. Since I do not very clear know how's your HellowWorld.exe script like. The location of HelloWorld.txt is decided by your script. If you did not define the generated file location in script, as default, this txt file path should same with exe. According to the pic you shared, seem its path is HelloWorld EXE Folder/HelloWorld.txt. I suggest you can use private agent firstly to run this Build to confirm its(.txt) path.
vstsFeedPublish should be the existing feed you have created in your project. Here it can not be the new name that dose not exist before the pipeline executed.
vstsFeedPackagePublish is the package name you want it stored in feed. For me as sample, here I name it as merlin. After the release finished:
You can specify a new name or use the exists one. Use a new name means create a new package in Feed, and use the exists one just means the version incremental.

Related

How to exclude artifact files after web deployment?

We have a build/release pipeline that is finally working correctly, but the developer asked that we exclude the stage config files (Web.Dev.config, Web.Test.config, Web.Prod.config) as well as the artifact archive itself from the site/wwwroot.
As you can see, every time we deployed, these zip files have been getting stored in the site root as well. They aren't harmful but it doesn't look good:
This is the Release App Service Web Deploy YAML:
steps:
- task: AzureRmWebAppDeployment#4
displayName: 'Azure App Service Deploy: project-123'
inputs:
azureSubscription: 'Azure Dev Service Connection'
WebAppName: 'project-123'
packageForLinux: '$(System.DefaultWorkingDirectory)/Project123 Dev Build Artifact/Release'
enableCustomDeployment: true
enableXmlTransform: true
How do we exclude those files after successful deployment?
Kudu dir structure:
Building on #theWinterCoder answer, Unfortunately, there doesn’t appear to be a way to honor the MSDeploySkipRules defined in the csproj file. Instead, files and folders can be skipped by defining the AdditionalArguments parameter of the Azure App Service Deploy (AzureRmWebAppDeployment) task.
Since there doesn’t appear to be any official documentation for the -skip rules, and the MSDeploy.exe documentation that Azure Pipelines references is out-of-date, in 2012, richard-szalay wrote a useful article, “Demystifying MSDeploy skip rules”, which provides a lot of details for anyone requiring additional control.
Brief Explanation:
The dirPath argument represents the Web Deploy Provider to skip a directory whilst the filePath argument is used to skip an individual file.
The dirPath starts at wwwroot.
For ASP.NET Core applications, there’s another wwwroot under wwwroot; as such, the absolutePath in that case would look like this: absolutePath=wwwroot\\somefoldername which would map to D:\home\site\wwwroot\wwwroot\somefoldername
Solution:
Therefore, since I’m skipping files, i set the web deploy provider to filePath, and since we’re not using .NET Core, we set absolutePath to Web.Dev.config. That would map to D:\home\site\wwwroot\Web.Dev.config.
The same thing applies for the zip artifact, however, if we don’t prepend \\ before the wildcard it will fail with following error:
Error: Error: The regular expression '.zip’ is invalid. Error: parsing ".zip" - Quantifier {x,y} following nothing. Error count: 1.
-skip:objectName=filePath,absolutePath=Web.Dev.config
-skip:objectName=filePath,absolutePath=Web.Prod.config
-skip:objectName=filePath,absolutePath=Web.Test.config
-skip:objectName=filePath,absolutePath=\\*.zip
or with regular expression:
-skip:objectName=filePath,absolutePath="Web.Dev.config|Web.Prod.config|Web.Test.config|\\*.zip"
Thats it 😃
You can add an additional arguments line to the yml that will tell it to skip certain files. It will look something like this:
AdditionalArguments: '-skip:objectName=dirPath,absolutePath=wwwroot\\Uploads'
More details can be found in this thread

Is there a way to pipe the smoke test output outside the agent?

I have a release pipeline with a QA/Smoke Test stage, that generates XML files containing test results.
If I run this manually on my machine, obviously I have access to the XML files and I can see the details but on the agent I cannot since we don't have access to those Microsoft hosted agents to view the files.
Is there a way to pipe the files "out" in the task for viewing? maybe there's a third marketplace task that can achieve that?
Here's the deployment result:
2021-06-06T23:34:19.1260519Z Results File: D:\a\r1\a\qa-automation\TestResults\CurrentReport\Logs\junit.xml
2021-06-06T23:34:19.2448029Z Results File: D:\a\r1\a\qa-automation\TestResults\.\CurrentReport\Logs\detailedLogs.xml
2021-06-06T23:34:19.2533810Z
2021-06-06T23:34:19.2596243Z Failed! - Failed: 22, Passed: 2, Skipped: 0, Total: 24, Duration: 52 m 11 s - EED.dll (netcoreapp3.1)
Here's the stage YAML:
steps:
- script: |
git clone https://.../qa-automation.git -b master
cd qa-automation
testrun.bat --cat "EDSmoke" --env dev
displayName: 'Clone qa-automation repo'
Is there a way to pipe the files "out" in the task for viewing? maybe there's a 3rd marketplace task that can achieve that?
You can try with following task:
Write-host "##vso[task.uploadfile]<PathOfTheFiles>\<filename>"
Like:
Write-host "##vso[task.uploadfile]$(System.DefaultWorkingDirectory)\qa-automation\TestResults\CurrentReport\Logs\junit.xml"
View and download attachments associated with releases
Would you like to upload additional logs or diagnostics or images when
running tasks in a release? This feature enables users to upload
additional files during deployments. To upload a new file, use the
following agent command in your script:
Write-host "##vso[task.uploadfile]"
The file is then available as part of the release logs. When you
download all the logs associated with the release, you will be able to
retrieve this file as well.
You can also add a powershell script task in your release definition to read the smoke test output and output it to the console. Then you will be see the content of the log files from "Logs" tab powershell script step. And you can also click "Download all logs as zip" to download the smoke test result files.

Variable substitution in config, JSON files in Azure DevOps Pipeline

I am a bit new in Azure DevOps. I know there is way that we can do XML transformation and JSON variable replacement. We can define key, value and json variable in library and that would update in release pipeline. Is there any way like just replacing a variable value in any file (config, json). for example I define a library value (hello = world), and Release pipeline task would find and replace $hello in config, json file and replace it with "world".
I am trying to use replace tokens as below.
Yes I was looking at same. Looks like its not replacing the values. Do you see any configuration issue as below
steps:
- task: qetza.replacetokens.replacetokens-task.replacetokens#3
displayName: 'Replace tokens in **/*.config **/*.json'
inputs:
targetFiles: |
**/*.config
**/*.json
verbosity: detailed
tokenPrefix: '{'
tokenSuffix: '}'
log as below
2020-05-11T18:44:01.6149125Z ##[section]Starting: Replace tokens in **/*.config **/*.json
2020-05-11T18:44:01.6363261Z ==============================================================================
2020-05-11T18:44:01.6363986Z Task : Replace Tokens
2020-05-11T18:44:01.6364452Z Description : Replace tokens in files
2020-05-11T18:44:01.6364873Z Version : 3.6.0
2020-05-11T18:44:01.6365252Z Author : Guillaume Rouchon
2020-05-11T18:44:01.6365919Z Help : v3.6.0 - [More Information](https://github.com/qetza/vsts-replacetokens-task#readme)
2020-05-11T18:44:01.6366694Z ==============================================================================
2020-05-11T18:44:02.2020864Z pattern: \{\s*((?:(?!\{)(?!\s*\}).)*)\s*\}
2020-05-11T18:44:02.2247835Z replaced 0 tokens out of 0 in 0 file(s) in 0.04 seconds.
2020-05-11T18:44:03.1202699Z ##[section]Finishing: Replace tokens in **/*.config **/*.json
Library value
myhello = Hello
Value in appsettngs.config
<section name="{myhello}" type="Exceptionless.ExceptionlessSection, Exceptionless" />
Yes. You can use token replace task. Please check this extension
I used your configuration (I just changed target folder to find a place wher I keep files) with this file.
steps:
- task: replacetokens#3
displayName: 'Replace tokens in *.config *.json'
inputs:
targetFiles: |
stackoverflow/23-token-replace/*.config
stackoverflow/23-token-replace/*.json
verbosity: detailed
tokenPrefix: '{'
tokenSuffix: '}'
config file:
<configuration>
<configSections>
<section name="sampleSection"
type="System.Configuration.SingleTagSectionHandler" />
<section name="{myhello}" type="Exceptionless.ExceptionlessSection, Exceptionless" />
</configSections>
<sampleSection setting1="Value1"
setting2="value two"
setting3="third value" />
</configuration>
And all went fine. Here you have a log for a build.
replacing tokens in: /home/vsts/work/1/s/stackoverflow/23-token-replace/appsettings.config
using encoding: ascii
myhello: Hello
1 tokens replaced out of 1
replaced 1 tokens out of 1 in 1 file(s) in 0.033 seconds.
All looks fine. Are you sure you have a correct targetFiles? **/*.config means that extension will look for any-folder-name/any-file-name.config located in your root folder.
The error replaced 0 tokens out of 0 in 0 file(s) in 0.04 seconds indicates the config or json files were not found in the default working directory.
Below are the possible reasons that the config/json files not found in the default working directory.
1, Your artifacts published from build pipeline is zipped. If the config/json files reside in the zipped artifacts, they cannot be found by the replacetoken task.
So you need to check if the artifacts downloaded in release pipeline is a zip file. If it is zipped, you need to add the Extract files task to unzip it before replacetoken task.
You can easily check the artifacts by clicking the 3dots highlighted in below screenshot.
2, If the config/json files are not in the default working directory. You need to specify in Root directory field the path to the folder where the config/json files reside.
Hope above helps!

How to receive Revision in Azure Pipelines YAML build definition

I created a new build with Azure Pipelines (Azure DevOps) and it worked really well.
Usually, you use $(Rev:.r) to get the revision in the build. Unfortunately, it seems the variable isn't replaced/set in the build steps. The only place where you can use it is the name: property in the YAML document.
Now I set it in the name and extract it in some PowerShell, which isn't necessary if you can get it via an environment variable.
How do I get the Revision (like $(Rev)) in the new builds (outside of the name: property in the YAML document)?
(The Build Agents running on-premise, inside Docker - but this shouldn't affect the things above)
You can't get the revision number without parsing, it is not stored as a separate field somewhere or in an environment variable.
The $(Rev:.r) portion instructs Azure DevOps to come up with the first number that makes the build number unique (and, in that specific example, put a dot in front of it).
Like you said, the only way is to use PowerShell script to get the value:
$buildNumber = $Env:BUILD_BUILDNUMBER
$revision= $buildNumber.Substring($buildNumber.LastIndexOf('.') + 1)
Edit:
You can install the Get Revision Number extension that does it.
Another possible solution to the above problem could be to use counter expression for ex: we difine the variable and use it in a task to build nuget package.
variables:
counterVar: $[counter($(versionVariable),0)]
.......
- task: CmdLine#2
inputs:
script: >
nuget pack ClassLibrary1/ClassLibrary1.csproj
-OutputDirectory $(Build.ArtifactStagingDirectory)
-NonInteractive
-Properties Configuration=release
-Version $(versionVariable).$(counterVar)
-Verbosity Detailed
-IncludeReferencedProjects
Here versionVariable is a custome variable defined in pipelines->variables.And the seed value is 0(2nd param to counter).
It works as below
Let's assume the versionVariable is 1.19
Build Run 1 counterVar will be 0.
Build Run 2 counterVar will be 1.
Now say we change the versionVariable to 1.20
Build Run 3 counterVar will be 0.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops
Check the counter expression in above link it reset its value for diff prefix.
P.S. Benefit of using counter over $(Rev:r) is that it can start from 0 unlike $(Rev:r)

VSTS Agent cannot overwrite nor remove the DLL when used by IIS

VSTS agent for AZure devOPS is not able to remove or overwrite the file when a URL has been invoked . However when the IIS worker process is killed and the Release is redeployed it is working fine .
I have enabled both overwrite and clean option . But it is still unable to deploy.
2018-11-17T13:05:23.2176452Z ##[section]Starting: Copy Files to: D:\IIS Hosting\CICD
2018-11-17T13:05:23.2179160Z ==============================================================================
2018-11-17T13:05:23.2179236Z Task : Copy Files
2018-11-17T13:05:23.2179314Z Description : Copy files from source folder to target folder using match patterns (The match patterns will only match file paths, not folder paths)
2018-11-17T13:05:23.2179373Z Version : 2.117.2
2018-11-17T13:05:23.2179438Z Author : Microsoft Corporation
2018-11-17T13:05:23.2179495Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkID=708389)
2018-11-17T13:05:23.2179554Z ==============================================================================
2018-11-17T13:05:23.4602460Z found 7 files
2018-11-17T13:05:23.4602904Z Cleaning target folder: D:\IIS Hosting\CICD
2018-11-17T13:05:23.4650798Z ##[error]Unhandled: Failed rmRF: EPERM: operation not permitted, unlink 'D:\IIS Hosting\CICD\AZSync.Web.dll'
2018-11-17T13:05:23.4659907Z ##[section]Finishing: Copy Files to: D:\IIS Hosting\CICD
Gives an error -
The solution was to have a task to copy app_offline.htm to the IIS hosting directory which would stop dotnet core from running . Copy/ overwrite all files and then remove app_offline.htmusing a delete file task.