I have a web api (REST) project that is written in .NET and I have written a few webtests (.webtest) that test those apis.
While those tests run fine locally from visual studio, I want to integrate them into my VSTS (Azure Devops) build pipeline, so as to identify and breaking changes that could break any of those APIs.
I am not able to find any task in build pipeline which can run the webtests as part of build. I see option for running unit-tests though.
So, wanted to check what am I missing here.
You might want to find an alternative approach as this link implies it has been deprecated.
Visual Studio web performance test (.webtest file) is tied to the load
test functionality and is deprecated. Some customers have used
.webtest for other purposes such as running API tests, even though it
was not designed for that purpose. Many API testing alternatives are
available in the market. SOAP UI is a free, open source alternative to
consider, and is also available as a commercial option with additional
capabilities.
You could try to use cmd task command line to run MSTest with arguments.
Add Run Command Line step/task to execute MSTest command
Add Publish Test Results step/task
On the other hand, you can do test in Unit Test too, just send the request and check the response, related thread.
Also as Matt mentioned, since Visual Studio web performance tests (.webtest files) are tied to the load test functionality and is also deprecated. You could take a look at this blog here: Cloud-based load testing service end of life
The software I worked on has both unit tests and system tests. System tests can take minutes to run, they take input values and we validate the results against expected output. There are hundreds of system tests. The software must be built (done this) and tested on both windows and Linux.
How can I automate testing with VSTS? I'd like to avoid doing this at build stage, because it would slow the builds down. I can't see how to automate this in the Test stage. Do I need additional extensions to do this? Everything seems so geared up for web development, e.g. selenium tests, how do we run automated tests for good old binary programs?
I would suggest using Release Management to deploy your application to a test environment and then run your tests as a part of your Release Definition. You can then choose to run tests in parallel to make sure that your system tests don't take days to run.
On a side note, having so many system tests is a code smell. I would suggest looking into building as many fast running unit tests as possible and only using system tests when absolutely necessary.
I recently set up a CI server in TeamCity and now want to take it to the next step, continuous deployment. Basically, we host a suite of restful services and about 3 web applications for each one of our customers. All customers get 3 environments QA, UAT and Prod. We want to be able to automatically deploy our builds once our tests pass. I'm not looking for custom scripting options to do this. I've seen plenty of those of SO. What we're looking for is a solutions like UDeploy but at a lower price point. Is anyone aware of alternatives to UDeploy? Or other Continuous Deployment plugins that work with TeamCity?
Thanks,
I agree with #Niklas Ringdahl -- I think you're thinking about it wrong.
You can deploy directly from TeamCity using MS WebDeploy.
See Troy Hunt's excellent blog series about this:
Part 1: Config transforms
Part 2: MS Build and deployable packages
Part 3: Publishing with WebDeploy
Part 4: Continuous builds with TeamCity
Part 5: WebDeploy with TeamCity
If this is .net you should look at Octopus, it now works nicely with TeamCity.
http://www.octopusdeploy.com/
http://www.paulstovell.com/octopus/octopus-octopack-and-teamcity
I don't know if there's any part of this I don't understand, but I would use TeamCity for that also.
If you have a build configuration to manage your tests, you can easily create configurations for deploy, which are triggered by the success of the test configuration. We handle lots of our deploys that way.
Again, sorry if I'm missing something here...
EDIT:
A build configuration in TeamCity can be triggered when another configuration is successfully built:
Build triggering in JetBrains documentation (Look at "Finish Build trigger")
Or you could use one build configuration with many "Build steps":
Build steps in JetBrains documentation
Each step is run in order, and only if the previous step is successful.
Another alternative: RedGate has recently updated pricing on their Deployment manager tool: http://www.red-gate.com/delivery/deployment-manager/
If you have 5 projects or less, the tool appears to be free.
I'm familiar with Nolio ASAP but have no knowledge about its pricing:
http://www.noliosoft.com/product/nolio-automation-center
As a developer for the product, I'll add BuildMaster to the list of options here. Basically, you can run a basic tutorial to get started deploying from TeamCity (http://inedo.com/buildmaster/extensions/teamcity), then customize your deployment plans to any further environments as needed.
Is NAnt still being actively developed, or are there other projects that are gaining more momentum in the Automated Build space?
Edit:
Obviously MSBuild will continue to advance but given all the extra tasks that can be accomplished using NAnt why has development suddenly stopped given that many open source and I assume commercial applications depend on nant.
There is NAnt Contrib, but to a reasonable degree, NAnt is one of the rare projects that can be considered "complete" :P
You may also be interested in Hudson; from the looks of it it can do builds without requiring NAnt (looks like it can do some MSBuild stuff directly, but I haven't used it)
You should also look at Fabric. It is an excellent tool and I use it well with Jenkins
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
What tools do you use for Automated Builds / Automated Deployments? Why?
What tools do you recommend?
Hudson for automated builds. I chose it because it was the easiest to setup and demo. A system that's too complex and isn't slick-looking won't impress management enough to get them on-board for automated builds. Especially in a project that has a lot of inertia.
NAnt for builds (but MSBuild, Rake, almost anything would be fine) and CruiseControl.NET for deployments. I'm currently working with the new Cruise from ThoughtWorks studios as it provides a better way to stage the various pipelines and let's me deploy any version I want to a target environment.
We use TeamCity, from JetBrains. They also make Resharper And IntelliJ.
We use it for building our .Net applications, and it has been quite easy to set up, connect to TFS, and run additional tools from. It is very polished, and actually kinda reminds me of this site. Found it much nicer than CruiseControl, and for our team size it is free. If you need lots of different builds, more per-user builds, and so on then it costs a bit (but still quite reasonable).
Funnily enough I just spent two weeks overhauling (read implementing from scratch) our nightly build process. Great fun (no, really). I toyed with the idea of installing Team Foundation Server, but we use Perforce for source control and I didn't think it was worth the hassle.
Our process is now a set of Powershell scripts that run on a dedicated build/test server that do the following on a scheduled task:
Wipe out the entire source tree (check that you didn't have anything checked out first!)
Bring down the entire source tree from Perforce (from the last labelled build)
Generate a change report (by syncing to HEAD and watching what comes down)
Build the App
Index the PDB files to the Perforce sources
Store the binaries and symbols in a dedicated symbol server
Run the test projects
Build the installer
Label
Send out emails to the group with status reports on all of the above
Works well.
make and bash on linux
make and cmd on windows
Visual Build Pro
We use a combination of build tools and continuous integration server:
Build tools:
Maven
SBT
Gradle
Rake
Continuous Integration Servers:
Jenkins
Hudson
Travis CI
Automated Build Studio.
Instead of letting you mes with scripts or xml files, it comes with predefined graphical macro operations that allows you to create tasks easily.
For our Windows-compilable stuff, we use FinalBuilder.
CruiseControl for automated builds. Works great.
For automated builds, I think the best tool going right now is JetBrain's Team City. The free version has all the features you'll need for most 5-10 person teams. Set up is easy, configuring new projects is painless (relatively), and most importantly, it's reliable.
For automated migrations, nothing beats PowerShell.
UppercuT uses NAnt to build and it is the insanely easy to use Build Framework.
Automated Builds as easy as (1) solution name, (2) source control path, (3) company name for most projects!
http://code.google.com/p/uppercut/
Some good explanations here: UppercuT
More information
UppercuT is a conventional automated build, which means you set up a config file and then you get a bunch of features for free. Arguably the most powerful feature is the ability to specify environment settings in ONE place and have them applied everywhere, including documentation when it builds the source.
Documentation available: https://github.com/chucknorris/uppercut/wiki
Features :
Simple setup
Simple upgrades
Custom extension points (pre, post, and replace) for each step of the build process http://uppercut.pbworks.com/CustomizeUsingExtensionPoints
Has documentation for integration w/Team City, CruiseControl.NET, and Jenkins (formerly Hudson) https://github.com/chucknorris/uppercut/tree/master/docs
Works on Linux w/Mono
Versioning DLLs based on build number and source control revisions (SVN, TFS, Git, HG)
Compile activities - F5 or Ctrl + Shift + B
Strong naming made as easy as true/false
Code Testing and Analysis
Testing
NUnit
MbUnit v2
Gallio
xUnit
NCover
NDepend
Nitriq
Mono Migration Analyzer
Obfuscation
ILMerge
Environment Templating and Building (ConfigBuilder, DocBuilder, SQLBuilder, DeploymentBuilder) https://github.com/chucknorris/uppercut/blob/master/docs/ConfigBuilder.doc?raw=true
Packaging output to prepare for deployment
Zips up output
At work we use good ol' Ant to build our Java servlets.
We used to use Visual Build from Kinook software, but recently with our new application we switched to MSBuild since it had better integration with TFS and the ability to create custom tasks.
The GNU Autotools definitely. The autoconf and automake are de-facto standard for unix systems.
I've had success using buildbot, triggered by a post-commit script on a subversion repository. This has been used for both automated builds and automated testing.
ANT for both build and deployment/installs.
Makes a great cross-platform installer.
We use Hericus Zed Builds And Bugs Management for our automated builds.
We have 4 branches of code, each with java, c++, C#, cross platform compiles and installers for 5 OS's.
Make for the builds.
Debian packages for deployments (since our production servers runs it).
TeamCity running NAnt scripts for building/packaging and PowerShell for deployment.
I've found that using NAnt, powered by TeamCity, instead of the native TeamCity runners allows us to have a much richer build process (eg. css minimiser, etc). It also means the full build/package process can be run on any developers PC instead of just the TeamCity servers making it much easier to customise and debug problems in the build process.