We've built a number of tests using SpecFlow and are using SpecRun to run the tests locally and on the build agent. Wed like to take the tests out of the build and stick them in the first release phase of the VSTS Release we have linked to the build.
The team followed the guidance from Tarun and Utkarsh to setup SpecRun in the build, but when we use the Publish Artefact task to package up all the test assemblies, they aren't found when running on the Release Agent.
When building the project on the build agent, the NuGet references of the solution are restored to the packages folder, this includes the SpecRun.Runner package which contains the Visual Studio Test Adapter to discover and run the tests.
This test runner package isn't copied to the output directory of the Test Project. The VsTest task in the build will automatically detect the test runner from the packages directory, which is why it works like magic in the build phase.
To make it work in Release we've found a few options:
ugly use the publish artefact option and publish the solution packages folder. In the release stage point the VsTest step to the other artefact location to find the Test Adapter:
better use the nuget to install the SpecRun.Runner nuget as part of the release stage. This runs the risk of the build downloading a newer or unexpected version of the SpecRun.Runner package.
best check in a separate packages.config with the reference to the SpecRun.Runner project and its dependencies, restore it during the build and attach it as a separate artefact to the build. This ensures the Release stages will be able to use the same version of the SpecRun.Runner and that the version is stored in source control.
Related
I have a staged release pipeline that deploys to Development and then to Staging.
I want my Integration Test project to run during release pipeline only before deployment to staging. How do I do that? is there a specific test task that I will be able to hook into before the staging deployment? like a pre-deploy task or something?
In order to run Integration Tests in your release pipeline. You should include your Test projects or test assembly dll files in the artifacts published in your build pipeline. So that your Integration test projects are accessible to the test tasks in your release pipeline.
To Include your test files in the artifacts. You can add a second publish build artifacts task in your Build Pipeline. Specify Path to publish to the location of your test files. Fore below example.
I add a another publish build artifacts task to publish all the test files that needed to run the tests.
Then I specify the Path to publish field to folder that contains all the test files.
After the build pipeline completed. You will see the Test folder being added to the build artifacts and published to the azure devops pipeline server.
Now you can run your tests in your release pipeline by adding the VsTest task or other test tasks in your release pipeline. The release pipeline will download your artifacts to
folder $(System.DefaultWorkingDirectory)(eg. C:\agent_work\r1\a).
For above example you will find the test files in folder $(System.DefaultWorkingDirectory)\artifact_alias\Test and the artifacts to be deployed in $(System.DefaultWorkingDirectory)\artifact_alias\drop.
Then you can either create a new stage with vstest task before deployment stage as #Kontekst mentioned, or just add the vstest task on the top of your deployment stage before your deployment task.
In above screenshot I add the Vstest task before the Azure deployment task and point the Search folder to the Test folder where the test files reside.
Update:
You can find the artifacts-alias in the field shown in below screenshot.
The main idea is to publish the test files in the artifacts in build pipeline and then add the test tasks in the release pipeline to run the tests.
I suppose you are using C# unit tests.
Add new stage "Integration tests" right before stage "Deployment to Staging".
Add task installing VSTest Platform. It will be cached in agent folder.
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/tool/vstest-platform-tool-installer?view=azure-devops
Add to your tests projects proper unit tests adapter NuGet package, so that VSTest Platform will be able to find unit tests in dll assemblies, for example for NUnit tests it will be "NUnit 3 Test Adapter".
Add VSTest task specyfing:
a) Test files -> for example "***IntegrationTests.dll"
b) Test platform version -> "Installed by Tools Installer"
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/test/vstest?view=azure-devops
All test results returned by VSTest task will be integrated with Azure DevOps services.
I Love both answers by Levi Lu-MSFT and Kontekst!
My team used to do it like that and it got the job done.
However if you'd like more "fancier" reporting and if like me, you were looking for something out of the box that Microsoft was offering specifically for integration tests or automated tests then Azure DevOps' Test Plan feature might be what you're looking for.
Test Plans really seem to be catering for actual web automated tests but you can basically run any tests that you can fit in a release pipeline.
It integrates well with Visual Studio Test Manager.
Here's the link to the documentation: https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
And here's some screenshots in case the link fails in the future:
I am trying to automate the generation of the following NuGet package using VSO:
NuGet Package
My problem is that all three "runtimes" libraries are built on different platforms. Hence, I have the following build configuration in VSO:
VSO Build
When build is completed, each agent job publish a native artifact for its platform:
Native Artifacts
My question is: Is it possible to add a 4th agent job to the build that would wait for all the other jobs and then aggregate the artifacts and generate the NuGet package with everything?
Currently, I need to manually download the artifacts and run a script to pack and sign the NuGet package. It's annoying because it require to install a certificate on my local machine and install the signing tools etc.
Thanks!
Your looking for the "Run this Job" setting on the 4th agent job. "Only When all previous jobs are complete" sounds like the value you want. You could then use powershell and/or the NuGet task to automate your process.
I have a build definition in VSTS and I want the unit test projects to be run as part of the build, do I then need to include those dlls in the deployment to the Azure web app? surely they are no longer needed after the build.
No matter you are going to use Release or not, you could only copy the files that you need and publish them:
If you are going to use TFS Release to deploy, then in Release definition, you can just select the build artifact. If you want to deploy in build definition, you can add deploy task after Copy files and deploy from the target folder in Copy files task.
I have successfully created a Build definition in VSTS for some SharePoint client side projects that I'm working on. I tried creating a Release definition but I can't seem to find any way for me to copy/download the artifacts created from my build definition locally. I may be missing something since I'm still quite new with VSTS but I can't seem to figure it out.
To downbload build artifacts from release when build successful, you can specify the release definition as below:
Add the build artifacts with latest version in release definition.
Enable Continuous deployment trigger for the artifacts.
Select the private agent which you want to download on the local machine.
Now when a build succeed, a new release will be triggered to download the latest build artifacts.
We are building packages for multiple deployment environments using TeamCity server and OctoPack. The problem is that tentacle agent chooses the latest by number version of the package, so it's the same (latest) package that is deployed on all environments. Here's the summary of our setup:
Environments DEV and STAGE;
Deployment to DEV is triggered from Git "dev" branch;
Deployment to STAGE is triggered from Git "stage" branch;
OctoPack is configured to generate packages MyProduct.1.0.0.dev-%build_counter% for DEV build configuration;
OctoPack is configured to generated packages MyProduct.1.0.0.%build_counter% for STAGE build configuration;
TeamCity is configured to expose OctoPack artefacts (NuGet packages) via its NuGet feed;
Octopus project is configured to deploy packages with NuGet Id MyProduct from TeamCity NuGet feed.
So what happens is that since DEV builds are run more frequently, they have larger %build_counter%, and STAGE doesn't get a chance to get a deployment of its own packages - Octopus tentacle preferes packages with 1.0.0.dev-* suffix.
This must be fairly common scenario, but I haven't found a simple way to solve it.
There are some parts that are not documented here: https://github.com/OctopusDeploy/Octopus-Tools. But if you look at https://github.com/OctopusDeploy/Octopus-Tools/blob/master/source/OctopusTools/Commands/CreateReleaseCommand.cs it is possible to figure out what you can do.
I think the tools is backward compatible, but not 100 % sure about that.
When you are using the octo tools, which I expect that you use, you can set the version (also called releasenumber now) option to specify the release number. If you doesn't specify anything else it will take the latest package so what you want to do is set the packageversion (also called defaultpackageversion now) that should be used for the release.
I think that should do it. If it doesn't, what are you using to create the release?
Example of what we are using from our TeamCity when using octo tools which we have added to the environment path on the build agents:
create-release --server=%conf.OctoServerApi% --project=%conf.OctoProject% --version=%env.OctopusPackageVersion% --deployto=%conf.OctoDeployEnv% --packageversion=%env.OctoPackPackageVersion% --apiKey=%conf.OctoApiKey% --waitfordeployment %conf.OctoExtraParams%
UPDATE:
The documentation for 2.0 is much better: http://docs.octopusdeploy.com/pages/viewpage.action?pageId=360596
Inspired by Tomas Jansson's answer, simply adding the following to Additional command line arguments in the OctopusDeploy: Create release build step (TeamCity v9) worked for me:
--packageversion=%build.number%