MSBuild with WebPublishMethod to package produces wrong file names in zip file - azure-devops

We have standard CI/CD on Azure DevOps. Recently we found, that files with diacritics (accent) in names are published wrong. Seems that instead of original chars all is in utf-8 (or something like that).
In build phase the is normal Visual Studio Build task with common msbuild command and arguments
/t:XXX /p:BuildProjectReferences=true /p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:DesktopBuildPackageLocation="$(build.artifactstagingdirectory)\XXX" /p:outputDir="$(build.artifactstagingdirectory)\XXX" /P:PackageTempRootDir=
After build, zip file contains encoded filenames. So encoding is part of build process.
Tried to simply call msbuild and publish to package from local machine on sample project and the same problem. Output zip package contains encoded filenames. And now it is impossible to extract zip file to get original naming.
My question is - how to force build and package task to not encode names, or use extract zip task to get filanames as they are in project? I found no info how to extract encoded files from artifacts.

Recently we found, that files with diacritics (accent) in names are published wrong.
Based on my test, I could reproduce this issue. When the Folder Name or File name contain the diacritics, their names will be encoded into other formats after Msbuild.
For example:
As you said, this problem also exists in the local msbuild, so this problem exists in msbuild itself. Since the SDK used by msbuild may be of a lower version, it cannot recognize these special characters
I am afraid that there is no such arguments in msbuild to force the original encoding of the file to be maintained.
Workaround1:
In Azure Devops, you could use Extract files task to unzip the zip file.
Then you could use Powershell task to run the script to change the file or folder name:
Folder:
Rename-Item 'C:\xxx\diakritik├│s' -newName 'C:\xxxm\diakritikós'
File:
Rename-Item -Path "c:\logfiles\daily_file.txt" -NewName "monday_file.txt"
Finally, you could use Archive files task to zip the files again.
Sample:
Workaround2:
Based on my test, you could use template for dotnet build.
In this case, dotnet build can keep the file name.
For the issue itself, I suggest you can report it here: Msbuild Issue.

Related

VSTS build does not seem to publish output files to proper directory

I am developing an ASP.NET core application.
I use VSTS online for my source control. I also have a dedicated machine to build the application.
To automate the process, I am now trying to define a build script from VSTS online.
The default VSTS template defines many tasks. I removed all the unneeded tasks for my purpose and am down to just three tasks - "Use NuGet," "NuGet Restore," and "Build Solution."
The default msbuild argument for "Build Solution" task is:
/p:DeployOnBuild=true /p:WebPublishMethod=Package
/p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true
/p:DesktopBuildPackageLocation=
"$(build.artifactstagingdirectory)\WebApp.zip"
/p:DeployIisAppPath="Default Web Site"
Although there is no error in running this script, I need to change it such that the output goes to a folder and is not packaged.
Here is part of my directory structure that is relevant for my question:
C:\Dev\RCWebsite\RCWebsite.sln
C:\Dev\RCWebsite\RCWeb\RCWeb.csproj
C:\Dev\RCWebsite\RCWeb\Properties\PublishProfiles\FolderProfile.pubxml
Here is the content of the publish profile:
<PropertyGroup>
<WebPublishMethod>FileSystem</WebPublishMethod>
<PublishProvider>FileSystem</PublishProvider>
<LastUsedBuildConfiguration>Release</LastUsedBuildConfiguration>
<LastUsedPlatform>Any CPU</LastUsedPlatform>
<SiteUrlToLaunchAfterPublish />
<LaunchSiteAfterPublish>True</LaunchSiteAfterPublish>
<ExcludeApp_Data>False</ExcludeApp_Data>
<ProjectGuid>d7d9f3b7-fd0e-49c1-b6d2-3af5dddb6699</ProjectGuid>
<publishUrl>C:\StagingSites\rcweb</publishUrl>
<DeleteExistingFiles>True</DeleteExistingFiles>
</PropertyGroup>
From the console window, I can run the following command:
msbuild .\RCWeb\RCWeb.csproj /p:DeployOnBuild=true
/p:PublishProfile=RCWeb\Properties\PublishProfiles\FolderProfile.pubxml
This works as expected. The final output is generated in C:\StagingSites\rcweb\ directory.
As this command works, I replaced the msbuild argument in VSTS "Build Solution" task as:
/p:DeployOnBuild=true
/p:PublishProfile=rcWeb\Properties\PublishProfiles\FolderProfile.pubxml
Note that I haven't specified .\RCWeb\RCWeb.csproj as an argument. Guess the build mechanism automatically takes care of this.
When I run this build and look at the log file, I see that the solution is built fine. However, it is never copied to C:\StagingSites\rcweb\ directory.
Can someone please tell me what is it that I am missing? Do I need another "deployment" task after "build solution" task? Regards.
Although there is no error in running this script, I need to change it such that the output goes to a folder and is not packaged.
No you do not need another deployment task. Since you do not need to package and deploy the ASP .NET project with msbuild, you can get rid of the /p:DeployOnBuild=true flag to msbuild. Also, after having a look at your publish profile I realized that you aren't really passing any information to msbuild (nothing that can't be passed from msbuild parameters atleast) and since you no longer plan to directly deploy from msbuild, it would be a good idea to keep the publish profile aside.
After trimming down the parameters and adding a few necessary ones (/T:Package tells msbuild that you want to package the binaries but not deploy them), this is what your msbuild command would look like:
/p:OutDir=$(Build.BinariesDirectory) /T:Package /p:PackageLocation=$(Build.BinariesDirectory)\WebApp.zip /p:PackageAsSingleFile=true
This is assuming that you wish to have a .zip file that is ready to be deployed, as the output. If that's not the case and you just require the binaries in a folder that can be directly deployed but not zipped, you can use the below command:
/p:OutDir=$(Build.BinariesDirectory) /T:Package /p:PackageLocation=$(Build.BinariesDirectory)\WebApp.zip /p:PackageAsSingleFile=true p:_PackageTempDir=$(Build.BinariesDirectory)\MyAspNetWebsite
This will generate the zip as well as the binaries folder ready to be deployed (sadly you need the zip for the temporary dir flag to work)
NOTE: If you plan on using the msbuild command within a powershell script, the VSO variables will need to accessed in a different way, like so: 'BUILD_BINARIESDIRECTORY'

How do I correct the output directory of a CI/CD NuGet package build using Visual Studio Team Services?

I've just started trying to configure a CI/CD process using Visual Studio Team Services, with a view to having a gated check-in process which then automatically packs and pushes the output to a private NuGet Feed. I'm successfully getting through the "Get Sources" task which exists natively in the build definition, however my package task is failing at this stage:
d:\a\_tool\NuGet\4.0.0\x64\nuget.exe pack
d:\a\1\s\Core\Core\Core.csproj -NonInteractive -
OutputDirectory d:\a\1\a -Properties Configuration=$Release;OutDir=$(OutDir)
-Symbols -version 1.0.0-CI-20170811-095629 -Verbosity Detailed
NuGet Version: 4.0.0.2283
Attempting to build package from 'Core.csproj'.
MSBuild auto-detection: using msbuild version '14.0' from 'C:\Program Files (x86)\MSBuild\14.0\bin\amd64'. Use option -MSBuildVersion to force nuget to use a specific version of MSBuild.
NuGet.CommandLine.CommandLineException: Unable to find 'd:\a\1\s\Core\Core\$(OutDir)\Core.dll'. Make sure the project has been built.
Essentially, the 2nd line of the log demonstrates that my project file is in d:\a\1\s, however the output is directed to d\a\1\a - not the same place. The instruction to NuGet to package my file then looks in the correct location (d:\a\1\s), but the file isn't here.
I understand that I can specify a Package Folder within the build definition, however I've not changed this from the default ($(Build.ArtifactStagingDirectory)) - I expect this to work natively, but for reasons I can't explain, it's not.
Is there a token or wildcard I can provide in an additional build property that will rectify this? I've taken the guidance of the first posted answer in here and changed $(Build.ArtifactStagingDirectory) to $(Build.Repository.LocalPath), which gets me much closer to the goal as the error now reads -OutputDirectory d:\a\1\s\... - I've tried manually putting a variety of sensible paths no the end of this, but the error persists.
Refer to this article. It has the build variables for the TFS/VSTS environment.
$(Build.ArtifactStagingDirectory) itself refers to the \1\a folder. You might want to try the $(Build.Repository.LocalPath) variable.
Not a 100% if it would work, but might as well try.
The issue is related to OutDir=$(OutDir) instead of OutputDirectory.
The OutputDirectory specifies the folder in which the created package is stored and the OutDir specifies the build output, but the project need to be built before NuGet pack task, so you need to add Visual Studio Build task or related to build project.
You can leave Additional build properties box blank to use default output per to configuration, because you just need to know the package stored path to push package to remote feed.
If you want to change build output, you can specify /p:OutDir msbuild argument to change build output location. For example:
Visual Studio Build task (MSBuild Arguments: /p:OutDir=$(Build.ArtifactStagingDirectory))
NuGet pack task (Additional build properties: OutDir=$(Build.ArtifactStagingDirectory))
BTW, to change package stored location, you need to specify OutputDirectory.

How do I get TFS 2013 Build definition to zip the build output?

I am using VS2013 and TFS2013. In TFS, I am using the new "default" TfvcTemplate12.xaml template.
In my build definition, in the Test-> Post-Build Script Arguments (or Path if I have to save the command in a file), what PowerShell script do I enter to get it to Zip the entire build output and drop that zip file onto a share drive? I want to avoid manipulating the XAML workflow.
I know the basic PS script to zip a folder is....
Get-Childitem C:\source -Recurse | Write-Zip -IncludeEmptyDirectories -OutputPath C:\stage\zomeZipFile.zip
But I don't know if this is the right approach, and if so, what to replace the source and stage parameters with.
Our code migration system requires us to have 1 ZIP file with everything in it for deployment.
You need to add the PowerShell to the post-test script location.
https://msdn.microsoft.com/en-us/library/hh850448.aspx
MSDN had a comprehensive list of the cracks that you have access to when it is executing.

Retrieve created NuGet package filename from 'nuget pack' command?

I'm writing a script that will automatically pack and publish Nuget files to my private repository (a fileshare) and a private symbol server (on localhost).
When I run nuget pack in Powershell I get a string output that contains the file location of both the nuget and symbol package, but as far as I can tell there's no way to programmatically get those paths without parsing the string. The package version is determined by the .csproj itself, so I don't necessarily know which version will be placed in the filename.
Here's the output from the pack command:
Attempting to build package from 'MyProject.csproj'.
Building project 'C:\Users\me\prg\MyProject.csproj' for target framework '.NETFramework,Version=v4.0'.
Packing files from 'C:\Users\me\prg\bin\Release'.
Using 'Myproject.nuspec' for metadata.
Successfully created package 'C:\nuget\MyProject.1.0.0.0.nupkg'.
Attempting to build symbols package for 'MyProject.csproj'.
Building project 'C:\Users\me\prg\MyProject.csproj' for target framework '.NETFramework,Version=v4.0'.
Packing files from 'C:\Users\me\prg\bin\Release'.
Using 'MyProject.nuspec' for metadata.
Successfully created package 'C:\nuget\Myproject.1.0.0.0.symbols.nupkg'.
Should I just bite the bullet and write some regex to parse out the "successfully created package" lines?
By default nuget pack will output the created packages to the current directory. You can override this by specifying the OutputDirectory option. It is not clear to me from where you run the nuget command (be it from within C:\Users\me\prg\, C:\nuget\, or somewhere else), so you might already know this, but it can offer you an alternative option:
You can create yourself a temporary directory (e.g. C:\nuget\temp\), which you can specify as your OutputDirectory option. If you then in your script make sure that this directory is empty before you run nuget pack, your can simply copy *.nupkg to your fileshare (and then afterwards move it to C:\nuget\, if that is required).
Parsing the output in powershell is not too hard.
$a = nuget pack .\Exceptional.csproj -Prop Configuration=Release | Select-String nupkg
$a -Replace "^[^']'", "" -Replace "'[^']$", ""
gives
C:\Users\phelan\workspace\Exceptional\Exceptional\Weingartner.Exceptional.1.2.6011.19372.nupkg
I made a slight change to the answer provided by #bradgonesurfing as his partly worked for me.
$pack = . $nugetPath pack rest of command........ | Select-String nupkg
$pack -match "'(?<nupkg>.*)'"
$packageFile = $Matches["nupkg"]
To get it in a post build event, I would use the powershell.exe to shell out to a powershell script. I have an example here that I used to automate running editbin that can be modified for nuget: https://gist.github.com/tcartwright/cc1bd4dc98349579630c43283c5f1fe9

Powershell in CruiseControl.net to backup existing folder before deploying new version of the code

I would like to zip a bunch of files (.exe and .dll) before I overwrite them with the new build. Is there a simple way to zip files without using some sort of dll?
Just creating a folder with the build number / date time stamp will also work great. How do I pass parameters from the cruise control build process into my Powershell script that will do the work then?
Is this a sustainable way to do things?
Thanks
You could either use CCNET's Package Publisher Task directly or zip the files via the PowerShell Task introduced in CCNET 1.5.
Configuration sample for PowerShell Task:
<powershell>
<description>Adding scheduled jobs</description>
<scriptsDirectory>ScheduledTasks</scriptsDirectory>
<script>CreateScheduledJobsFromListOfTasks.ps1</script>
<buildArgs>-zipDir="C:\foo"</buildArgs>
</powershell>