Azure DevOps Pipeline not finding edmx resources - entity-framework

I have a .Net Core 3.1 MSTest project that calls a separate class library that connects to my database, which was done database-first so I have a .edmx file. It all runs fine locally, but when I push it out to my Azure DevOps Pipeline I start getting a Unable to load the specified metadata resource. exception. I've tested this out by putting in a bit of code to print out all the resources in the assembly
var resources = (Assembly with EDMX).Assembly.GetManifestResourceNames();
System.Console.WriteLine("There are " + resources.Length + " resources in the assembly");
foreach (var resource in resources)
{
System.Console.WriteLine(resource);
}
The result on my local computer prints out what I expect
There are 3 resources in the assembly
Model.csdl
Model.msl
Model.ssdl
However, the same run on my Pipeline shows 0 resources
There are 0 resources in the assembly
My build process locally and on my pipeline are the exact same
task: PowerShell#2
inputs:
targetType: 'inline'
script: |
& "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\msbuild.exe" $(Build.SourcesDirectory)\MyProject\MyProject.sln
dir $(Build.SourcesDirectory)\MyProject -include ('*.csdl', '*.msl', '*.ssdl') -recurse
& "C:\hostedtoolcache\windows\dotnet\dotnet.exe" test $(Build.SourcesDirectory)\MyProject\Project.Tests\Project.Tests.csproj
Just to make sure the resources actually exist on my Azure Build agent I've added that 2nd powershell command to find my .csdl, .msl, and .ssdl files, and sure enough they do exist
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 5/30/2020 3:54 PM 30772 Model.csdl
-a---- 5/30/2020 3:54 PM 10993 Model.msl
-a---- 5/30/2020 3:54 PM 23351 Model.ssdl
These files are located in $(Build.SourcesDirectory)\MyProject\Project.Models\obj\Debug\edmxResourcesToEmbed
And the property looks to be correctly set in my .csproj
<ItemGroup>
<EntityDeploy Include="ProjectModels.edmx">
<Generator>EntityModelCodeGenerator</Generator>
<LastGenOutput>ProjectModels.Designer.cs</LastGenOutput>
</EntityDeploy>
</ItemGroup>
I don't use .edmx database designs often, so I'm unfamiliar on how it is suppose to handle the resources, are they suppose to get compiled into the assembly, or are they just loaded at runtime? My build process both locally and in my Pipeline both show:
Skipping target "EntityDeployNonEmbeddedResources" because it has no outputs.
EntityDeployEmbeddedResources:
Processing 1 EDMX files.
Starting to process input file 'ProjectModels.edmx'.
Finished processing input file 'ProjectModels.edmx'.
Finished processing 1 EDMX files.
Not sure what that indicates, but since it occurs on both I'm assuming its not part of the issue. The only other difference that I can think of is my Azure Pipeline uses Visual Studio 2019 Enterprise, while my local build uses Visual Studio 2019 Community.
Any ideas on what I can do to get these resources to load on my Pipeline build?

Azure DevOps Pipeline not finding edmx resources
This issue should comes from the dotnet test.
AFAIK, dotnet cli does not support embed edmx resources. These types of files are supported in MSBuild props/targets that don't ship in the CLI and only ship in full framework VS.
You could check this ticket for some more details.
And the remaining work for this is already tracked at dotnet/ef6#231.
Hope this helps.

Related

How to exclude artifact files after web deployment?

We have a build/release pipeline that is finally working correctly, but the developer asked that we exclude the stage config files (Web.Dev.config, Web.Test.config, Web.Prod.config) as well as the artifact archive itself from the site/wwwroot.
As you can see, every time we deployed, these zip files have been getting stored in the site root as well. They aren't harmful but it doesn't look good:
This is the Release App Service Web Deploy YAML:
steps:
- task: AzureRmWebAppDeployment#4
displayName: 'Azure App Service Deploy: project-123'
inputs:
azureSubscription: 'Azure Dev Service Connection'
WebAppName: 'project-123'
packageForLinux: '$(System.DefaultWorkingDirectory)/Project123 Dev Build Artifact/Release'
enableCustomDeployment: true
enableXmlTransform: true
How do we exclude those files after successful deployment?
Kudu dir structure:
Building on #theWinterCoder answer, Unfortunately, there doesn’t appear to be a way to honor the MSDeploySkipRules defined in the csproj file. Instead, files and folders can be skipped by defining the AdditionalArguments parameter of the Azure App Service Deploy (AzureRmWebAppDeployment) task.
Since there doesn’t appear to be any official documentation for the -skip rules, and the MSDeploy.exe documentation that Azure Pipelines references is out-of-date, in 2012, richard-szalay wrote a useful article, “Demystifying MSDeploy skip rules”, which provides a lot of details for anyone requiring additional control.
Brief Explanation:
The dirPath argument represents the Web Deploy Provider to skip a directory whilst the filePath argument is used to skip an individual file.
The dirPath starts at wwwroot.
For ASP.NET Core applications, there’s another wwwroot under wwwroot; as such, the absolutePath in that case would look like this: absolutePath=wwwroot\\somefoldername which would map to D:\home\site\wwwroot\wwwroot\somefoldername
Solution:
Therefore, since I’m skipping files, i set the web deploy provider to filePath, and since we’re not using .NET Core, we set absolutePath to Web.Dev.config. That would map to D:\home\site\wwwroot\Web.Dev.config.
The same thing applies for the zip artifact, however, if we don’t prepend \\ before the wildcard it will fail with following error:
Error: Error: The regular expression '.zip’ is invalid. Error: parsing ".zip" - Quantifier {x,y} following nothing. Error count: 1.
-skip:objectName=filePath,absolutePath=Web.Dev.config
-skip:objectName=filePath,absolutePath=Web.Prod.config
-skip:objectName=filePath,absolutePath=Web.Test.config
-skip:objectName=filePath,absolutePath=\\*.zip
or with regular expression:
-skip:objectName=filePath,absolutePath="Web.Dev.config|Web.Prod.config|Web.Test.config|\\*.zip"
Thats it 😃
You can add an additional arguments line to the yml that will tell it to skip certain files. It will look something like this:
AdditionalArguments: '-skip:objectName=dirPath,absolutePath=wwwroot\\Uploads'
More details can be found in this thread

Is there a way to pipe the smoke test output outside the agent?

I have a release pipeline with a QA/Smoke Test stage, that generates XML files containing test results.
If I run this manually on my machine, obviously I have access to the XML files and I can see the details but on the agent I cannot since we don't have access to those Microsoft hosted agents to view the files.
Is there a way to pipe the files "out" in the task for viewing? maybe there's a third marketplace task that can achieve that?
Here's the deployment result:
2021-06-06T23:34:19.1260519Z Results File: D:\a\r1\a\qa-automation\TestResults\CurrentReport\Logs\junit.xml
2021-06-06T23:34:19.2448029Z Results File: D:\a\r1\a\qa-automation\TestResults\.\CurrentReport\Logs\detailedLogs.xml
2021-06-06T23:34:19.2533810Z
2021-06-06T23:34:19.2596243Z Failed! - Failed: 22, Passed: 2, Skipped: 0, Total: 24, Duration: 52 m 11 s - EED.dll (netcoreapp3.1)
Here's the stage YAML:
steps:
- script: |
git clone https://.../qa-automation.git -b master
cd qa-automation
testrun.bat --cat "EDSmoke" --env dev
displayName: 'Clone qa-automation repo'
Is there a way to pipe the files "out" in the task for viewing? maybe there's a 3rd marketplace task that can achieve that?
You can try with following task:
Write-host "##vso[task.uploadfile]<PathOfTheFiles>\<filename>"
Like:
Write-host "##vso[task.uploadfile]$(System.DefaultWorkingDirectory)\qa-automation\TestResults\CurrentReport\Logs\junit.xml"
View and download attachments associated with releases
Would you like to upload additional logs or diagnostics or images when
running tasks in a release? This feature enables users to upload
additional files during deployments. To upload a new file, use the
following agent command in your script:
Write-host "##vso[task.uploadfile]"
The file is then available as part of the release logs. When you
download all the logs associated with the release, you will be able to
retrieve this file as well.
You can also add a powershell script task in your release definition to read the smoke test output and output it to the console. Then you will be see the content of the log files from "Logs" tab powershell script step. And you can also click "Download all logs as zip" to download the smoke test result files.

Publish files with UniversalPackages#0 - "The path provided is invalid."

I'm trying to publish a file (Helloworld.txt) to my Universal Packages directory to pass on to a different stage within my Release Piepline (using UniversalPackages#0).
I cannot for the life of me figure out how to make it work.
I'm trying to follow this KB but it leaves me confused:
https://learn.microsoft.com/en-us/azure/devops/pipelines/artifacts/universal-packages?view=azure-devops&tabs=yaml
For example, I can't determine what vstsFeedPublish is and don't know if vstsFeedPackagePublish can be a new name I make up on the spot or if it is actually the name of an existing file/folder.
What "path" am I entering incorrectly that's making it fail?
Stage Deployment Steps:
Helloworld Output to TXT:
cd $(Build.ArtifactStagingDirectory)
"$(System.DefaultWorkingDirectory)/Helloworld Build/Helloworld EXE Folder/Helloworld.exe" >> WriteLineOutput.txt
I cd into $(Build.ArtifactStagingDirectory) first because that is the default folder UniversalPackages#0 looks to publish files, so I want WriteLineOutput.txt to be generated there.
Publish Output TXT to Universal Dir (YAML):
steps:
- task: UniversalPackages#0
displayName: 'Publish Output TXT to Universal Dir'
inputs:
command: publish
vstsFeed: '0a3a9abd-83fd-495f-967b-e986c523f2d2'
vstsPackageVersion: 1
vstsFeedPublish: '0a3a9abd-83fd-495f-967b-e986c523f2d2'
vstsFeedPackagePublish: 'writelineoutput-txt'
versionOption: minor
packagePublishDescription: 'TXT output from Helloworld.exe'
Working Directory Structure:
"Publish Output TXT to Universal Dir" Task Output:
2019-12-02T03:35:35.2264352Z ##[section]Starting: Publish Output TXT to Universal Dir
2019-12-02T03:35:35.2381247Z ==============================================================================
2019-12-02T03:35:35.2381328Z Task : Universal packages
2019-12-02T03:35:35.2381359Z Description : Download or publish Universal Packages
2019-12-02T03:35:35.2381412Z Version : 0.160.1
2019-12-02T03:35:35.2381461Z Author : Microsoft Corporation
2019-12-02T03:35:35.2381490Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks
2019-12-02T03:35:35.2381520Z ==============================================================================
2019-12-02T03:35:36.0651445Z SYSTEMVSSCONNECTION exists true
2019-12-02T03:35:36.6058302Z Downloading: https://0t3vsblobprodcus362.vsblob.vsassets.io/artifacttool/artifacttool-win10-x64-Release_0.2.128.zip?sv=2017-04-17&sr=b&sig=tHZQU3V2DuXcC0Y1xnmzB7Zw7kMdjJSijWVDiztc9UE%3D&spr=https&se=2019-12-02T04%3A35%3A37Z&sp=r&P1=1575261037&P2=11&P3=2&P4=LS6Ffab5P%2bb8Q9r3aGsGLlK9ELRD6bRxxlTkDc5aEc8%3d
2019-12-02T03:35:39.3862184Z Caching tool: ArtifactTool 0.2.128 x64
2019-12-02T03:35:40.4414172Z SYSTEMVSSCONNECTION exists true
2019-12-02T03:35:41.4415128Z Publishing package: writelineoutput-txt, version: 0.1.0 using feed id: 65dc653c-5c3b-771c-b308-34b199d8fcee, project: null
2019-12-02T03:35:41.4469092Z [command]C:\hostedtoolcache\windows\ArtifactTool\0.2.128\x64\ArtifactTool.exe universal publish --feed 65dc653c-5c3b-771c-b308-34b199d8fcee --service https://vsrm.dev.azure.com/sawtooth-capstone/ --package-name writelineoutput-txt --package-version 0.1.0 --path D:\a\r1\a\$(Build.ArtifactStagingDirectory) --patvar UNIVERSAL_PUBLISH_PAT --verbosity None --description "TXT output from Helloworld.exe"
2019-12-02T03:35:44.0492154Z {"#t":"2019-12-02T03:35:43.3152933Z","#m":"ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 23e076ea-5122-4c06-b92a-2aef5974defd","#i":"8778ba0f","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.315Z"}
2019-12-02T03:35:44.0493421Z {"#t":"2019-12-02T03:35:43.7170274Z","#m":"Ensuring that the package does not yet exist...","#i":"40e01e14","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.717Z"}
2019-12-02T03:35:44.0494251Z {"#t":"2019-12-02T03:35:43.8900269Z","#m":"Package does not yet exist","#i":"c781eca5","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.890Z"}
2019-12-02T03:35:44.0494769Z {"#t":"2019-12-02T03:35:43.8913512Z","#m":"Pushing content...","#i":"3aa40378","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.891Z"}
2019-12-02T03:35:44.0495859Z {"#t":"2019-12-02T03:35:43.9060246Z","#m":"DedupManifestArtifactClient will correlate http requests with X-TFS-Session 23e076ea-5122-4c06-b92a-2aef5974defd","#i":"09a6f3ce","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.906Z"}
2019-12-02T03:35:44.0496478Z {"#t":"2019-12-02T03:35:44.0301714Z","#m":"The path provided is invalid.","#i":"05178f7d","#l":"Error","SourceContext":"ArtifactTool.Program","UtcTimestamp":"2019-12-02 03:35:44.030Z"}
2019-12-02T03:35:44.0776082Z ##[error]Error: An unexpected error occurred while trying to push the package. Exit code(16) and error({"#t":"2019-12-02T03:35:43.3152933Z","#m":"ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 23e076ea-5122-4c06-b92a-2aef5974defd","#i":"8778ba0f","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.315Z"}
{"#t":"2019-12-02T03:35:43.7170274Z","#m":"Ensuring that the package does not yet exist...","#i":"40e01e14","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.717Z"}
{"#t":"2019-12-02T03:35:43.8900269Z","#m":"Package does not yet exist","#i":"c781eca5","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.890Z"}
{"#t":"2019-12-02T03:35:43.8913512Z","#m":"Pushing content...","#i":"3aa40378","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.891Z"}
{"#t":"2019-12-02T03:35:43.9060246Z","#m":"DedupManifestArtifactClient will correlate http requests with X-TFS-Session 23e076ea-5122-4c06-b92a-2aef5974defd","#i":"09a6f3ce","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.906Z"}
{"#t":"2019-12-02T03:35:44.0301714Z","#m":"The path provided is invalid.","#i":"05178f7d","#l":"Error","SourceContext":"ArtifactTool.Program","UtcTimestamp":"2019-12-02 03:35:44.030Z"})
2019-12-02T03:35:44.0789627Z ##[error]Packages failed to publish
2019-12-02T03:35:44.0898947Z ##[section]Finishing: Publish Output TXT to Universal Dir
Key Lines:
2019-12-02T03:35:44.0496478Z {"#t":"2019-12-02T03:35:44.0301714Z","#m":"The path provided is invalid.","#i":"05178f7d","#l":"Error","SourceContext":"ArtifactTool.Program","UtcTimestamp":"2019-12-02 03:35:44.030Z"}
2019-12-02T03:35:44.0776082Z ##[error]Error: An unexpected error occurred while trying to push the package. Exit code(16) and error({"#t":"2019-12-02T03:35:43.3152933Z","#m":"ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 23e076ea-5122-4c06-b92a-2aef5974defd","#i":"8778ba0f","SourceContext":"ArtifactTool.Commands.UPackPublishCommand","UtcTimestamp":"2019-12-02 03:35:43.315Z"}
EDIT:
Well, I read it's best practice to use the Build pipeline to publish artifacts and the Release pipeline to run/test them, so I decided to just publish the artifact from the Build pipeline instead of trying to make the same Helloworld.txt again during the Release pipeline. But I'm willing to take the inevitable downvotes if someone can help me figure out how to publish new files created in the Release pipeline into the Universal packages directory.
I'd better suggest you call exe to generate the txt file in Build. Then publish them as artifacts to release pipeline. Then you can easily pick the HelloWord.txt file in Release pipeline.
This task can be used both in Build and Release pipeline. The difference between apply this task in Build and release just be the Path to file(s) to publish value which you very confused.
For Release pipeline:
If what you want is publish the HelloWord.txt into package, just click , and then select the corresponding file HelloWord.txt from it.
So, what its value should be $(System.DefaultWorkingDirectory)/{Source alias}/{artifact name}/s/HelloWorld.txt.
Note: This way is applied when you put the txt generated in Build pipeline, and then publish them as artifact and used in release pipeline.
Similar in Build pipeline, you can also select the file from that button. For its YAML format, just input the file name like:
inputs:
command: publish
publishDirectory: 'azure-pipelines.txt'
vstsFeedPublish: '*****'
vstsFeedPackagePublish: merlin
versionOption: custom
versionPublish: 0.0.1
packagePublishDescription: published in 2019/12/2
If azure-pipelines.txt file is under a folder, just input it as a {folder name}/azure-pipelines.txt.
In your issue, if you want to put the file generate and the package publish both in Build pipeline. Since I do not very clear know how's your HellowWorld.exe script like. The location of HelloWorld.txt is decided by your script. If you did not define the generated file location in script, as default, this txt file path should same with exe. According to the pic you shared, seem its path is HelloWorld EXE Folder/HelloWorld.txt. I suggest you can use private agent firstly to run this Build to confirm its(.txt) path.
vstsFeedPublish should be the existing feed you have created in your project. Here it can not be the new name that dose not exist before the pipeline executed.
vstsFeedPackagePublish is the package name you want it stored in feed. For me as sample, here I name it as merlin. After the release finished:
You can specify a new name or use the exists one. Use a new name means create a new package in Feed, and use the exists one just means the version incremental.

VSTS Agent cannot overwrite nor remove the DLL when used by IIS

VSTS agent for AZure devOPS is not able to remove or overwrite the file when a URL has been invoked . However when the IIS worker process is killed and the Release is redeployed it is working fine .
I have enabled both overwrite and clean option . But it is still unable to deploy.
2018-11-17T13:05:23.2176452Z ##[section]Starting: Copy Files to: D:\IIS Hosting\CICD
2018-11-17T13:05:23.2179160Z ==============================================================================
2018-11-17T13:05:23.2179236Z Task : Copy Files
2018-11-17T13:05:23.2179314Z Description : Copy files from source folder to target folder using match patterns (The match patterns will only match file paths, not folder paths)
2018-11-17T13:05:23.2179373Z Version : 2.117.2
2018-11-17T13:05:23.2179438Z Author : Microsoft Corporation
2018-11-17T13:05:23.2179495Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkID=708389)
2018-11-17T13:05:23.2179554Z ==============================================================================
2018-11-17T13:05:23.4602460Z found 7 files
2018-11-17T13:05:23.4602904Z Cleaning target folder: D:\IIS Hosting\CICD
2018-11-17T13:05:23.4650798Z ##[error]Unhandled: Failed rmRF: EPERM: operation not permitted, unlink 'D:\IIS Hosting\CICD\AZSync.Web.dll'
2018-11-17T13:05:23.4659907Z ##[section]Finishing: Copy Files to: D:\IIS Hosting\CICD
Gives an error -
The solution was to have a task to copy app_offline.htm to the IIS hosting directory which would stop dotnet core from running . Copy/ overwrite all files and then remove app_offline.htmusing a delete file task.

What is the Build Drop Location environment variable name for PowerShell in TFS 2015/2017

In previous versions of TFS (before 2015), there was a build environment variable for PowerShell called: TF_BUILD_DROPLOCATION, which gave the The location of the drop:
https://msdn.microsoft.com/library/hh850448%28v=vs.120%29.aspx.
I can't find the equivalent variable in TFS 2017.
What is the best practice to get it?
With Build agent tasks taking over things are different. What I do to see the various build environment variables is to make a simple batch file containing this:
SET C:\temp\EnvVars.txt
That'll produce a quick list of what is available.
Here's what I see with the TFS 2017 build agent:
agent.jobstatus=Succeeded
AGENT_BUILDDIRECTORY=C:\Agent\_work\2
AGENT_HOMEDIRECTORY=C:\Agent
AGENT_ID=2 AGENT_JOBNAME=Build
AGENT_JOBSTATUS=Succeeded
AGENT_MACHINENAME=BUILDMACHINE
AGENT_NAME=BUILDMACHINE
AGENT_OS=Windows_NT
AGENT_ROOTDIRECTORY=C:\Agent\_work
AGENT_SERVEROMDIRECTORY=C:\Agent\externals\vstsom
AGENT_TEMPDIRECTORY=C:\Agent\_work\_temp
AGENT_TOOLSDIRECTORY=C:\Agent\_work\_tool
AGENT_VERSION=2.122.1
AGENT_WORKFOLDER=C:\Agent\_work
BUILD_ARTIFACTSTAGINGDIRECTORY=C:\Agent\_work\2\a
BUILD_BINARIESDIRECTORY=C:\Agent\_work\2\b
BUILD_BUILDID=2036
BUILD_BUILDNUMBER=Database Build_20190708.2
BUILD_BUILDURI=vstfs:///Build/Build/2036
BUILD_CONTAINERID=2281
BUILD_DEFINITIONNAME=Database Build
BUILD_DEFINITIONVERSION=17
BUILD_QUEUEDBY=Smith, John
BUILD_QUEUEDBYID=8c588342-b87a-40cb-9b8c-a0ed10b57a3f
BUILD_REASON=Manual
BUILD_REPOSITORY_CLEAN=false
BUILD_REPOSITORY_GIT_SUBMODULECHECKOUT=False
BUILD_REPOSITORY_ID=$/
BUILD_REPOSITORY_LOCALPATH=C:\Agent\_work\2\s
BUILD_REPOSITORY_NAME=Collection
BUILD_REPOSITORY_PROVIDER=TfsVersionControl
BUILD_REPOSITORY_TFVC_WORKSPACE=ws_2_2
BUILD_REPOSITORY_URI=http://TFSSERVER:8080/tfs/Project/
BUILD_REQUESTEDFOR=Smith, John
BUILD_REQUESTEDFOREMAIL=John.Smith#Mailinator.com
BUILD_REQUESTEDFORID=7a588222-b66a-40ee-9b2a-a0ba10b12a3f
BUILD_SOURCEBRANCH=$/Collection/Project/Code
BUILD_SOURCEBRANCHNAME=Code
BUILD_SOURCESDIRECTORY=C:\Agent\_work\2\s
BUILD_SOURCEVERSION=9811
BUILD_SOURCEVERSIONAUTHOR=Smith, John
BUILD_SOURCEVERSIONMESSAGE=Added missing permission
BUILD_STAGINGDIRECTORY=C:\Agent\_work\2\a
You can list all Environment Variables with the following command:
get-childitem ENV:\
I am assuming you could create a simple build job that executes this and then look at the console output to determine what the name is of the Environment Variable you need.