I have a web api (REST) project that is written in .NET and I have written a few webtests (.webtest) that test those apis.
While those tests run fine locally from visual studio, I want to integrate them into my VSTS (Azure Devops) build pipeline, so as to identify and breaking changes that could break any of those APIs.
I am not able to find any task in build pipeline which can run the webtests as part of build. I see option for running unit-tests though.
So, wanted to check what am I missing here.
You might want to find an alternative approach as this link implies it has been deprecated.
Visual Studio web performance test (.webtest file) is tied to the load
test functionality and is deprecated. Some customers have used
.webtest for other purposes such as running API tests, even though it
was not designed for that purpose. Many API testing alternatives are
available in the market. SOAP UI is a free, open source alternative to
consider, and is also available as a commercial option with additional
capabilities.
You could try to use cmd task command line to run MSTest with arguments.
Add Run Command Line step/task to execute MSTest command
Add Publish Test Results step/task
On the other hand, you can do test in Unit Test too, just send the request and check the response, related thread.
Also as Matt mentioned, since Visual Studio web performance tests (.webtest files) are tied to the load test functionality and is also deprecated. You could take a look at this blog here: Cloud-based load testing service end of life
Related
I'm not sure if I'm searching correctly, but I'm hoping I can get some guidance here.
Current Setup
I have one solution with two projects:
Web API project
Node.js project
I'm using Azure DevOps with 2 Builds, each with their own Releases, one for each project. Each build definition only triggers when their respective project is updated/changed.
This works great!
I've noticed that each build actually builds the entire solution. In order to not waste processing power, I'd prefer to have each project only build their own project, with a few caveats.
The Web Api project does not depend on changes to the Node.js project, however, the Node.js can depend on changes to the Web Api. Because of this, if the Web Api fails, I don't want the Node.js to build/release.
Goals
What I'm trying to do is setup my build definitions so that the Web Api build, only builds the Web Api project. Whether it completes or errors out, let the build proceed as it normally does.
However, I want the Node.js project build both the Web Api and the Node.js project, so that if the Web Api build fails, then the whole build fails even if the Node.js would not have failed.
I've tried adding a new Visual Studio Build task and select the project only, but I got the following error:
[warning]C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(781,5): Warning : The OutputPath property is not set for project 'MyProject.csproj'. Please check to make sure that you have specified a valid combination of Configuration and Platform for this project. Configuration='release' Platform='any cpu'. You may be seeing this message because you are trying to build a project without a solution file, and have specified a non-default Configuration or Platform that doesn't exist for this project.
I'm currently looking on how to fix this, but I want to pose the following questions in case I'm headed in the wrong direction.
Questions
How can I setup the build definition to only build 1 project?
Is there a different configuration I should be creating instead of the one I mentioned?
Build Definition
What I'm trying to do is setup my build definitions so that the Web Api build, only builds the Web Api project. Whether it completes or errors out, let the build proceed as it normally does.
This is the case where you want a build definition that is targeting the project file instead of the .sln. Your error is that building the .csproj requires the OutputPath property to have a value, so just add it to the MSBuild Arguments box: /p:OutputPath="$(build.binariesDirectory)\MyProject". Build.BinariesDirectory is a predefined variable, but is otherwise not a required directory value. You can use what makes sense for you.
I want the Node.js project build both the Web Api and the Node.js project, so that if the Web Api build fails, then the whole build fails even if the Node.js would not have failed.
The simplest and least sophisticated way
From what I understand about your situation, this case doesn't require any additional changes. If you build the solution, then the pipeline will fail if either of the projects are broken. The downsides to this are:
The Node.js project doesn't "get" the newest changes to the Web API "dependency" until the Node.js project is changed and CI triggers a build
If you use a build completion trigger to mitigate downside 1 above, the Web API project gets built twice even though we know it should be successful both times
The more complex and sophisticated (but elegant?) way
Set a Build completion trigger on the Node.js pipeline that will trigger a build when the Web API pipeline is successful. This is similar to what you have now with some differences. With build completion AND a CI triggers on your Node.js pipeline, the Web API build can succeed regardless of the result of the downstream Node.js, but you will be building the Node.js project even when changes are not made to that project explicitly. (This may not be what you want if you're trying to save on agent activity)
Your Node.js pipeline can then have 2 separate build steps, each targeting one of the project files. However, the step for the Web API project build can have a condition to NOT perform if the Build.Reason is BuildCompletion. This allows the Node.js to be a downstream project of the Web API, but doesn't build the Web API if we already know it's successful.
Note: depending on how your references work between these projects in this solution, you may need to add other tasks for downloading the build artifacts and what-not to make sure everything is where it should be for building.
I'm trying to create a release pipeline in VSTS that runs my xUnit-tests as specified in a Test Plan.
Long story short: I can't get it to work.
What I'm using:
Azure DevOps (formerly VSTS)
Visual Studio Test task (v2.*)
Test project targeting .NET Core 2.1
xunit 2.4 with xunit.runner.visualstudio 2.4
In Azure DevOps I defined a Test Plan that contains a Test Suite which contains a Test that has an Associated Automation which points to my xUnit test.
I had to use the REST API to link the test code to the Test as described here.
I can select that Test in the visual designer for the VSTest task.
When I run the release pipeline the VSTest task fails with the following error message:
DiscoveryMessage : System.IO.FileNotFoundException: Unable to find tests for D:\a\r1\a\Foo.Tests.dll. Make sure test project has a nuget reference of package "Microsoft.NET.Test.Sdk" and framework version settings are appropriate. Rerun with /diag option to diagnose further.
The path to the Foo.Tests.dll is correct, all required files are copied as well.
I explicitly specified the framework version in a .runsettings file (as the option Other console options doesn't work when using the Test plan option).
Specified the path to custom test adapters
used Visual Studio 2017 and Installed by Tools Installer options
Added a .NET Core Tool installer to install the correct .NET Core SDK
...and any other combination of settings I could think of.
The error message is still the same.
Any ideas what I might be missing? Your help would be greatly appreciated at this point!
After several more hours we stumbled across a web page that stated that you don't have to copy the binaries of your test project as input for the VSTest task but PUBLISH it instead. That never came to mind as vstest.console.exe runs smoothly when you point it at the binaries on a local machine.
UPDATE: We had to add a Publish Artifact task at the end of our Build Pipeline and make the Release Pipeline pick up the published artifact.
I've switched to using Visual Studio Code and and am missing the ability to right-click and publish files.
I read here that a task runner can be used to build/deploy files...
Publish Web Deploy using VS Code
That answer only mentions Grunt and I cannot find any articles on how to do this with Gulp which I am already using. I've got as far as finding an article on how to hook up Gulp and MSbuild which I assume must be the first step.
However, I'm unsure about the deployment step as I don't know what the "publish" button in standard VS does. It seems to be using something called MSDeploy or "web deploy" and requires a bunch of publish settings, so presumably this can be done from gulp?
I know I must sound a bit clueless but I'm not really a Microsoft guy I'm a front-ender, just happen to be working with other people's .net projects occasionally. Any help would really be appreciated.
WebDeploy is the technology that VisualStudio uses for web deployments. WebDeploy uses MSDeploy behind the scenes. MSDeploy is capable of far more than just web app deployments. We use it for deploying Windows Services, Scheduled Tasks and SQL databases as well.
Here are details on you to use the MSDeploy commandline. The sync command is used most commonly for deployments.
You can use the gulp-mswebdeploy-package plugin to create a web deploy package as part of you gulp build task. you can the use the msdeploy command line to deploy the site.
I have spent a couple of days going through a lot of sites and reading about Nant , Rake etc.
please forgive my Noob question but I still cannot find what is the difference between Nant and CruiseControl.
As far as I can see Nant can do automated builds , run tests .
so what extra does cruisecontrol do ?
Also there was mention of Teamcity . there too from the documents I can see it can do builds but it also can use Nant but I fail to understand why it needs to use Nant when it can do the builds itself
I am basically trying to follow proper software practices by introducing automated builds at my workplace
Appreciate all help
Nant by itself can do builds and test, but it needs to be launched by some other mechanism such as a windows scheduled job. There is not a capability of launching the build only when source code changes, at least without
an amount of additional scripting.
Nant itself is just a script runner, not a scheduler - it requires some other software (or a manual user action) to launch it.
Continuous integration (CI) tools such as CruiseControl or TeamCity provide monitoring of source control to launch a build process in addition to other things. The build itself could the be a single nant script which runs the build and tests as you suggest, or the build could be done using a series of tasks which are built in to the CI server. The difference is not how the builds are done, but how they are initiated and reported.
CI servers additionally usually provide web-based reporting of the details of the build runs and unit tests.
In summary, Ci tools provide monitoring, scheduling, and reportingin addition to scripting of the build process.
Nant is a scripting language
CruiseControl is a free continuse integration tools
Teamcity in another contiuse integration tools,
regards,
What is the best way to integrate nunit tests into TFS 2010? Is it via generic tests or is there a better approach to running them?
Ideally I'd like to have the granularity of one generic test per test assembly and have a way to surface the results in the TFS build report.
As of now (Oct 2011), the easiest way is probably via the NUnit activity that can be found in the Community TFS Build Extensions.
You can run nunit tests from command line and therefore you can automate these tests via your (Workflow) build template.
Since there aren't a lot of custom build activities available for TFS 2010 yet, you could write your own to make sure that it integrates better with the TFS Build Report.
Here you can find some information about writing a custom build activity.
Another approach would be to create unit tests via Visual Studio. These tests will be automatically run when performing a build.
Ian Battersby wrote a great tutorial how to set up NUnit tests for Visual Studio 2010.
Basically, you need to:
download his build workflow template and set to your build definition
install NUnit and NUnitTFS in your tfs machine
add paths to installed directories in build process parameters
set Configurations to Build setting in build process parameters
change your NUnitTfs.exe.config with your collection name