Analyzing non-Visual Studio, Unity C# code with MSBuild/SonarQube via Jenkins - unity3d

I inherited the responsibility for a SonarQube Linux server which was running version 5.1 and analyzing dozens of Unity C# projects. After upgrading it to SQ 6.1 I discovered that support for analyzing C# projects on non-Windows systems was dropped in version 5.5, so I downgraded it to version 5.4 for the present time.
I have subsequently set up a Windows system as a Jenkins node and SonarQube server. However, the projects I am hoping to evaluate with SonarQube are (mostly) developed on and (always) built on macOS in Unity and therefore lack MSbuild solution files. Is it at all possible for me to use SonarQube within these constraints, short of adding Visual Studio-specific code to the projects? Is there some kind of dummy solution file that I could add to the projects that would allow them to be analyzed without the expectation of a working binary being produced (along the lines of parsing/tokenizing without compiling/linking)? Thanks.

Here is the procedure I have developed for analyzing Unity projects in SonarQube via Jenkins.
A user-facing Jenkins job with the following settings and options, among others specific to my environment.
General—This project is parameterized
String parameter PROJECTKEY
String parameter PROJECTNAME
String parameter PROJECTVERSION
General—Restrict where this project can be run set to the Jenkins build node name you used for your SonarQube server.
Source Code Management
Repository URL set to the repository you want to analyze.
Branches to build set to the main branch, i.e. refs/heads/master.
Additional Behaviors—Recursively update submodules checked as appropriate for your repository.
Build Triggers
Poll SCM checked, but no schedule specified.
Build Environment
Run buildstep before SCM runs—Inject environment variables set to HOME=C:\Users\Jenkins (or as appropriate for your system). This environment variable makes it possible for git to find the correct ssh credentials to use with the Git server. You'll need to put your git private key in %HOME%\.ssh; that is documented elsewhere and left as an exercise to the reader.
Use build environment from another project—Template Project pointing to a second Jenkins job. Mine is called SonarQube_Scanner (see below).
Build
Use builders from another project—Template Project pointing to the same Jenkins job SonarQube_Scanner.
Post-build Actions
Archive the artifacts—Files to archive set to log.zip.
A non-user-facing Jenkins job (SonarQube_Scanner) with the following settings and options.
General—Disable this project checked.
General—Restrict where this project can be run checked and set to your SonarQube server.
Source Code Management set to None.
Build Environment—Inject environment variables to the build process checked.
Properties Content contains
`LANG=en_US.UTF-8`
Build
Execute Windows batch command contains
set FILENAME=%WORKSPACE%\client\Assets\Editor\SQMenuItems.cs
REM Yes, this syntax works and is sooo much more readable
> %FILENAME% ECHO using UnityEditor;
>> %FILENAME% ECHO public static class SQMenuItems {
>> %FILENAME% ECHO static void AssetsOpenCSharpProject() {
>> %FILENAME% ECHO #if UNITY_5
>> %FILENAME% ECHO EditorApplication.ExecuteMenuItem("Assets/Open C# Project");
>> %FILENAME% ECHO #else
>> %FILENAME% ECHO EditorApplication.ExecuteMenuItem("Assets/Sync MonoDevelop Project");
>> %FILENAME% ECHO #endif
>> %FILENAME% ECHO EditorApplication.Exit(0);
>> %FILENAME% ECHO }
>> %FILENAME% ECHO }
"C:\Program Files\Unity_5.5.2\Editor\Unity.exe" -quit -batchmode -nographics -logFile "%WORKSPACE%\log.txt" -buildTarget android -projectPath %WORKSPACE%\client -executeMethod SQMenuItems.AssetsOpenCSharpProject
"C:\Program Files\7-zip\7z.exe" -tzip a log.zip log.txt
SQMenuItems.cs is what triggers Unity to create the .sln and .*proj files that MSBuild needs. The .sln and .*proj files are typically in the .gitignore file and therefore not in the repository.
SonarQube Scanner for MSBuild - Begin Analysis
SonarQube Installation set to the Jenkins build node name you used for your SonarQube server.
Project key set to ${PROJECTKEY}.
Project name set to ${PROJECTNAME}.
Project version set to ${PROJECTVERSION}.
Additional arguments
/d:sonar.exclusions=**\\SQMenuItems.cs
Execute Windows batch command
"C:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe" /maxcpucount /nr:false /nologo /target:rebuild /verbosity:quiet client\client.sln
You may need to change client\client.sln above to match your environment. I haven't yet tried replacing that with an environment variable defined in the user-facing Jenkins job.
SonarQube Scanner for MSBuild - End Analysis
DONE
GOTCHAS
If the job fails because Unity fails because it can't find the projectPath, it may have forgotten its license key. Remote desktop to your SonarQube server and launch the Unity GUI. If it pops up asking for your license key, re-enter the key, then quit. Subsequent jobs should succeed.
Things that make it very difficult to pass information to SonarQube in a dynamic way.
Windows environment variables defined in 2.5.1 “Execute Windows batch command” are not preserved through to 2.5.2 “SonarQube Scanner for MSBuild - Begin Analysis”.
The SonarQube.Analysis.xml file specifically prohibits assigning the properties sonar.projectName, sonar.projectKey, and sonar.projectVersion.
I've probably overlooked something; please don't hesitate to inquire.

Related

How to name build steps in Buildbot

I have Buildbot set up on a Windows 7 machine that's set to run a series of tests when I check in python code to a git repo. When I check on the build details via the web interface it displays a set of build steps that have been run as defined in my master.cfg configuration file.
Currently I have several shell command steps that each run a windows .bat file. Each of these steps is sequentially titled "shell", "shell_1", "shell_2", etc. Is there a way to re-name these steps (i.e. "format_check", "unittest", etc.)? I don't see an option to name build steps in the "build steps" section of the buildbot documentation.
Generally you just add it as a parameter in the call to addStep: e.g. name="step_name"
Updating the referenced documentation's examples to include a name parameter:
f.addSteps([
steps.SVN(repourl="http://svn.example.org/Trunk/", name="svn"),
steps.ShellCommand(command=["make", "all"], name="make"),
steps.ShellCommand(command=["make", "test"], name="test")

Visual studio team services build .net core 1.1

I'm trying to build a .net core 1.1 project on vsts. The project is developed in vs2017 and it uses the csproj instead of project.json. I have tried multiple options to build id on vsts with the hosted agents (windows and linux).
i have tried the following build steps
Visual studio build
Set to use vs 2017 but i get a warning "Visual Studio version '15.0' not found. Looking for the latest version." And then i get errors because it cant include .net core packages.
.NET Core (PREVIEW)
Cant find project.json. When i set it to use csproj file it gives an error "The file type was not recognized"
Command build step
I tried to run the commands with command build steps. "dotnet build" gives the error that it cant find the project.json file.
Anyone building dotnet 1.1 with csproj on vsts that can help me how to do it?
In Visual Studio Team Services, go to Build & Release > Builds and click Edit for the build definition you want to update
Navigate to the Options tab, change Default agent queue to Hosted VS2017, and save.
You can download dotnet SDK manually and run dotnet build from command line.
So it could be something like this:
Inline PowerShell step (I've used Inline Powershell extension by Peter Groenwegen):
Invoke-WebRequest https://go.microsoft.com/fwlink/?linkid=837977 -OutFile $(System.DefaultWorkingDirectory)\dotnet.zip
Extract files step:
From: $(System.DefaultWorkingDirectory)\dotnet.zip
To: $(System.DefaultWorkingDirectory)\dotnet
Restore packages:
$(System.DefaultWorkingDirectory)\dotnet\dotnet.exe restore
... and so on
But there is some limitation — you still haven't had .Net Core 1.1 installed at build agent machine so some features may not work. At least dotnet test will fail because it requires appropriate .Net Core runtime. Maybe some other features as well.
extending on #Nikolay Balakin's answer, it's true the .NET Core projects using *.csproj are not supported yet.
You can work around this by installing the latest .NET core on the hosted build environment yourself.
This will allow running dotnet restore, dotnet build, dotnet publish, and dotnet test.
Use the Inline powershell extension to run a script. You can link to a script, or paste the text in inline. I am running a script which is checked in to the project.
It seems each powershell script will be run in it's own environment, so paths etc. will not persist between scripts, so the installation steps and the build steps need to be combined into one script.
You need to copy the dotnet installation script from github and add your own build commands to the end.
I know this is not a long term solution, but we justified it by assuming the VSTS will in the near future support the *.csproj files, and we will convert to use the official build task.
Here is an example powershell script, showing the last line of the installation script, and the custom build commands on the end.
...
...
Say "Installation finished"
# this is the end of the downloaded script, add your steps after here.
Say "Running dotnet restore AdminPortal\AdminPortal.csproj"
dotnet restore AdminPortal\AdminPortal.csproj
Say "dotnet publish AdminPortal\AdminPortal.csproj --configuration Release"
dotnet publish AdminPortal\AdminPortal.csproj --configuration Release
Say 'Zipping publish file'
$source = $env:BUILD_REPOSITORY_LOCALPATH
$source = $source + '\AdminPortal\bin\Release\net461\publish'
$destination = $env:BUILD_REPOSITORY_LOCALPATH
$destination = $destination + '\AdminPortal\bin\Release\net461\publish.zip'
Add-Type -assembly "system.io.compression.filesystem"
[io.compression.zipfile]::CreateFromDirectory($source, $destination)
Say "Publish finished"
dotnet test "AdminPortal.Tests\AdminPortal.Tests.csproj"
Say "Test finished"
exit 0
According to this issue .NET Core projects using *.csproj files are not supported yet:
https://github.com/Microsoft/vsts-tasks/issues/3311
"if you are using hosted agent - then the tooling there works only
with project.json files"
I've tried the tutorials here, but they also seem to be outdated (I couldn't even get tfx-cli installed on my machine):
http://mattvsts.blogspot.nl/2016/11/start-building-aspnet-core-11-with-tfs.html
In my case, I have a .NET Core web app and four library projects, all targeting the full framework since I'm using EF 6.
I tried all of the suggestions here and none of them worked. Building with Visual Studio Build on Hosted Agent 2017 does build the project, but doesn't output any binaries. And all the options above did build as well but didn't generate the output files.
Reading around I found the only way to get the output files was by running dotnet publishbut this generates a build error because nuget isn't restoring well the packages and msbuild can't find them. After being tired of trying to make it work during a whole day, casually I enabled the "Restore Nuget Packages" on the VS Buid task, and though it says it's deprecated, that seems to have solved my isse.
In VSTS you need to add netcore exists as a demand.
Go to your build definition
Click on the options tab
Add the demand netcore exists

How to Deploy a .Net web app to Azure using Powershell

I have a "Web app" in Azure to which I deploy/publish a .Net web application using Visual Studio. (Build --> Publish), and it works.
I want to be able to deploy/publish my application using a Powershell script. I got the following script to work for the build portion:
CMD> "c:\Program Files (x86)\MSBuild\14.0\bin\msbuild.exe" WebApplication1.sln
To make it also deploy, I need to add a few parameters:
CMD> "c:\Program Files (x86)\MSBuild\14.0\bin\msbuild.exe" WebApplication1.sln /p:DeployOnBuild=true /p:PublishProfile="C:\Users\jgodse\Documents\Visual Studio 2015\Projects\WebApplication1\WebApplication1\Properties\PublishProfiles\jg-7-web-app-1 - Web Deploy.pubxml" /p:Configuration=Release
I got an error:
Build FAILED.
"c:\Users\jgodse\Documents\Visual Studio 2015\Projects\WebApplication1\WebApplication1.sln" (default target) (1) ->
"c:\Users\jgodse\Documents\Visual Studio 2015\Projects\WebApplication1\WebApplication1\WebApplication1.csproj" (default target) (2) ->
(MSDeployPublish target) ->
C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Web\Microsoft.Web.Publishing.targets(4295,5): msdeploy error ERROR_USER_UNAUTHORIZED: Web deployment task failed. (Connected to the remote computer ("jg-7-web-app-1.scm.azurewebsites.net") using the Web Management Service, but could not authorize. Make sure that you are using the correct user name and password, that the site you are connecting to exists, and that the credentials represent a user who has permissions to access the site. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_USER_UNAUTHORIZED.) [c:\Users\jgodse\Documents\Visual Studio 2015\Projects\WebApplication1\WebApplication1\WebApplication1.csproj]
0 Warning(s)
1 Error(s)
I am obviously missing my Azure credentials (seeing as Visual Studio was able to capture them), and I am also not running in Powershell.
So I took the entire command and put it into a file called deploy.bat, opened up a Powershell window and did the following:
PS> Login-AzureRmAccount
(I typed in my user/password in the GUI popup).
PS> cmd /c .\deploy.bat
The build was fine, but I got the same error when trying to publish. I guess that the Azure credentials did not carry through when shelling out to the CMD program.
How do I use Powershell to call MSBuild on my .Net project to publish to an Azure web app?
You can use your existing .pubxml file but you need the password in order to be able to deploy. There are several ways to get it. The most obvious one is to get it from the portal by navigating to the blade of you Web app and then clicking on "More" and finally on "Get publish profile"
This file contains all sorts of data but the one that you need is called userPWD - this is the password that you need to use. Copy the password and add it to your MsBuild command:
CMD> "c:\Program Files (x86)\MSBuild\14.0\bin\msbuild.exe" WebApplication1.sln /p:DeployOnBuild=true /p:PublishProfile="C:\Users\jgodse\Documents\Visual Studio 2015\Projects\WebApplication1\WebApplication1\Properties\PublishProfiles\jg-7-web-app-1 - Web Deploy.pubxml" /p:Configuration=Release /p:Password="Value of userPWD"
Obviously storing this value in your build scripts is not recommended. What you could do is download the publish settings using Powershell (Get-AzurePublishSettingsFile), extract the userPWD value, pass it to MsBuild to publish your app, and finally clean everything up.
There are various ways to implement you build infrastructure and what I have proposed might not be the best solution for you but it is up to you to experiment and decide what works best for you.
Some resources:
Automate Everything (Building Real-World Cloud Apps with Azure) - This one uses ASM instead of Resource Manager but you can easily tweak it to use ARM
Using Windows PowerShell scripts to publish to dev and test environments
Hope this helps.

Invalid file names when trying to deploy SSDT project with TeamCity 8

I am trying to deploy Visual Studio 2012 SSDT project to Sql Server using TeamCity 8 and MSBuild Publish task but the deployment fails.
When I look at TeamCity logs and use /v:diag switch in my build configuration I see that for unknown reason MSBuild searches for MyProject.sqlproj.publish.sql and for MyProject.sqlproj.dacpac files.
The exact error:
[SqlPublishTask] C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\SSDT\Microsoft.Data.Tools.Schema.SqlTasks.targets(1233, 5): File "C:\Program Files\TeamCity\buildAgent\work\abf8bc05a2cfe7f\*MyProject*\bin\Debug\*MyProject*.sqlproj.dacpac" does not exist.
The correct .sql and .dacpac files get generated (without the .sqlproj in the middel) in buildAgent/work/identificator/*MySolution*/MyProject/bin/Debug folder.
My TeamCity build step is configured as follows:
Runner type: MSbuild
Build file path: MyProject/*MyProject*.sqlproj
MSBuild version: 4.5
MsBuild ToolsVersion: 4.0
Run platform: 4.0
Targets: Publish
Command line parameters: /p:SqlPublishProfilePath="Debug.publish.xml" /p:Configuration=Debug
If I execute this from commandline I get no errors.
Any ideas on how can I configure TeamCity to search for correct files or configure my project to generate the files that TeamCity is searching for.
Or is my plan to use MSBuild's Publish task futile and I should utilise sqlpackage.exe instead?
UPDATE
After spending almost three days trying to figure this out I gave up and used sqlpackage.exe which works like a charm.
But I would still be interested in an answer though, passing paths to executables in build servers seems a bit crude way to accomplish things.
I had a similar issue and came to the conclusion that the way TeamCity produces "pseudo-project" files with *.teamcity suffixes is confusing something in the MSBuild/SSDT target chain.
I simply replaced the MSBuild runner build step with a pure Command Line step and the problem went away.
We lose the user friendliness of the TeamCity MSBuild runner configuration, but if it works, it's a compromise I'm willing to make.
Note - we are running TeamCity 7 - I am not sure if this has been addressed in later versions.
I found out you can set a System Property named "system.SqlTargetName" on the build configuration to override the default value.
Setting this to your project name without the ".sqlproj" makes the error go away.

Automatization Build Server TFS 2008 (vs 2008)

I use VS 2008, Team Explorer and TFS. I'm looking to automate your builds by executing TFSBuild.exe command.
I follow those steps:
Open TeamExplorer, in VS2008, connected to TFS;
My Team Project has a Build (named MainBuild) in Builds. Then, I do Query New build option.
I fill the properties in window dialog opened:
build Definition: Mainbuild
Build Agent: Machine1BuildAgent
drop folder for this build: \Machine1\Build_drop
priority in queue: Normal
MsBuild command-line arguments:
/p:BuildAll=false /p:RunTest=false /p:SkipClean=true /p:SkipGet=true /p:SkipLabel=true /p:SkipGetChangesetsAndUpdateWorkItems=true
In MDSN I have seen TFSBuild command and this sample:
The following example builds the Nightlies build type which is in the AdventureWorks team project on server01. The resulting build is located on Machine1 in the BuildDrop directory.
TFSBuild start http://server01:8080 AdventureWorks Nightlies /m:Machine1 /d:"C:\BuildDrop"
edit: Now, for my issue, how can I do automation for build MainBuild for set the values for "MsBuild command-line arguments" using TFSBuild.exe command ? I'm looking for a way to automatically have the TFS Build Agent job run nightly with command-line arguments..
I need similar command line like this:
TFSBuild start http://machine01:8080 MyteamProject MainBuild /m:Machine1 /d:"\Machine1\Build_drop" /p:BuildAll=false /p:RunTest=false /p:SkipClean=true /p:SkipGet=true /p:SkipLabel=true /p:SkipGetChangesetsAndUpdateWorkItems=true
I need pass the arguments (MsBuild command-line arguments) to TFSBuild.exe command and automatize the TFS build.
any sample scripting code ?
You can pass MSBuild properties to TFSBuild using the /msBuildArguments switch
TFSBuild start http://machine01:8080 MyteamProject MainBuild /m:Machine1 /d:"\Machine\Build_drop"
/msBuildArguments:"/p:BuildAll=false;/p:RunTest=false;/p:SkipClean=true;
/p:SkipGet=true;/p:SkipLabel=true;/p:SkipGetChangesetsAndUpdateWorkItems=true"
Alternatively, you could also specify these properties in the response file TFSBuild.rsp.
I'm not sure I understand your question clearly, but if you're looking to automate your builds by executing MSBuild from the command-line, you can create a batch file to execute the necessary command using the command-line arguments you want. Then add a scheduled task to 'Scheduled Tasks' in Windows so that this batch file is executed on a regular (e.g. nightly) basis.