I'm using TeamCity for my CI builds, and I'd like to set up a second build for running automated UI tests on Windows XP and Windows 7 virtual machines.
I imagine the build working as follows:
Compile, run unit tests, etc.
Prepare MSI using WiX
Copy MSI to target test machines
Remotely execute MSI's
Copy test harness code to remote machine
Run tests
Build finishes
The automated UI tests are written using NUnit and would need to be run directly on the test virtual machine (they can't run remotely). It's important that if the tests fail, it appears in the TeamCity build log and the build fails. I'd rather not install VS or the TeamCity build agents on either of the test virtual machines.
It seems that most of this should be possible using psexec.exe. Are there any alternative (preferably open source) tools that I should look at?
takes a deep breath
We were looking into something to help us out with our automated UI tests. We use ranorex to test the UI and TeamCity/Msbuild to execute the tests.
We never found any tools to help us out (I’m constantly keeping an eye out for some so will monitor this thread) but here is what we did instead.
The CI server copies the setup files and test scripts to the Testing Host Server.
The CI server then launches a custom app on the Testing Host Server providing the name of the VM to launch.
The Test Host Server then launches the VM software, using Virtual PC.exe -singlepc -pc vhdname.vhd -launch , and waits for it to shutdown (after it’s run its tests).
The VM grabs the setup files and scripts from the network location and executes.
After the tests are run it then returns the results to a networked location and shuts itself down.
Control is returned to the custom app.
Control is returned to the CI server which determines from the results if it has passed or failed (and updates the UI so developers are made aware of the result).
Results are collection as artifacts in TeamCity and tagged in Svn.
I think that's everything. Its convoluted, however, it works. Hope someone of that helps you.
Jeff Brown of the Gallio team has been talking about a tool called Archimedes that he's planning to write to support this kind of requirement. It sounds promising, but I don't think there has been much progress on it so far.
In the mean time though, there is something in the Gallio project called VM Tool that may do what you want. It provides commands to stop, start and snapshot virtual machines, and more importantly, to copy files back and forth and execute commands.
I presume you have also considered Powershell Remoting?
Related
The software I worked on has both unit tests and system tests. System tests can take minutes to run, they take input values and we validate the results against expected output. There are hundreds of system tests. The software must be built (done this) and tested on both windows and Linux.
How can I automate testing with VSTS? I'd like to avoid doing this at build stage, because it would slow the builds down. I can't see how to automate this in the Test stage. Do I need additional extensions to do this? Everything seems so geared up for web development, e.g. selenium tests, how do we run automated tests for good old binary programs?
I would suggest using Release Management to deploy your application to a test environment and then run your tests as a part of your Release Definition. You can then choose to run tests in parallel to make sure that your system tests don't take days to run.
On a side note, having so many system tests is a code smell. I would suggest looking into building as many fast running unit tests as possible and only using system tests when absolutely necessary.
I've setup Continuous Integration in my project hosted at Visual Studio Online.
Among other things, I deploy my web application to a test slot and test database and I run integration tests.
However, I don't need the test slot and database to be running all the time. I would like to start them both for the integration tests to run and stop them after my integration tests are done, all automated.
I played around with the Azure PowerShell vNext task, but I couldn't figure out how to accomplish this. Any help would be appreciated. More than the actual answer, I would appreciate a link with more information, if possible.
For the web app slot, you can add AzurePowerShell step in your build defition can create two power-shell script: one start the website via Start-AzureWebsite command before the build start and another one stop the website via Stop-AzureWebsite after the testing is finished.
For the database, you cannot stop it. Refer to this question for details: How do I stop and start a SQL Azure database?
I have spent a couple of days going through a lot of sites and reading about Nant , Rake etc.
please forgive my Noob question but I still cannot find what is the difference between Nant and CruiseControl.
As far as I can see Nant can do automated builds , run tests .
so what extra does cruisecontrol do ?
Also there was mention of Teamcity . there too from the documents I can see it can do builds but it also can use Nant but I fail to understand why it needs to use Nant when it can do the builds itself
I am basically trying to follow proper software practices by introducing automated builds at my workplace
Appreciate all help
Nant by itself can do builds and test, but it needs to be launched by some other mechanism such as a windows scheduled job. There is not a capability of launching the build only when source code changes, at least without
an amount of additional scripting.
Nant itself is just a script runner, not a scheduler - it requires some other software (or a manual user action) to launch it.
Continuous integration (CI) tools such as CruiseControl or TeamCity provide monitoring of source control to launch a build process in addition to other things. The build itself could the be a single nant script which runs the build and tests as you suggest, or the build could be done using a series of tasks which are built in to the CI server. The difference is not how the builds are done, but how they are initiated and reported.
CI servers additionally usually provide web-based reporting of the details of the build runs and unit tests.
In summary, Ci tools provide monitoring, scheduling, and reportingin addition to scripting of the build process.
Nant is a scripting language
CruiseControl is a free continuse integration tools
Teamcity in another contiuse integration tools,
regards,
I know, I know. There are a million threads everywhere talking about problems with mapped drives. I've ready many of them but I still can's seem to wrap my head around this problem or come to a solution.
I have a build server/continuous integration server (Win 2003 running CruiseControl.NET) that listens to our source control server. When a change is detected, the build server gets the new code, compiles it, tests it and if successful, copies the files to one of our web servers. There are 6 web servers - 3 Windows 2003 boxes, 3 LAMP boxes. Each OS has a separate development, staging and production box. All 6 web servers are mapped to a different drive on the build server. I have a Windows start-up script that calls a few "NET USE" commands that set the stage for the deployments.
CCNET is the service that listens to SVN. However, CCNET calls NAnt to perform all of the actual processing and tasks (compilation, testing, copying).
When I map the drives manually and run the NAnt scripts manually, everything works beautifully. When the startup script maps the drives and CCNET triggers the NAnt, the drives are nowhere to be found. I think the problem has something to do with user accounts. CCNET runs under the LOCAL SYSTEM account. I don't know what account the startup script runs under. Obviously manual execution runs under my account.
The weirdest part is that at certain points in the past, everything was working great. I am not sure what changed. How can I get the mapped drives to be visible to all users and services? (Also, any other critique of any part of this setup/process is welcome)
The problem definitely was with user accounts. The drives were mapped under different accounts than the account CCNET was running on. Once I finally straightened everything out and got it running on the LOCAL SYSTEM account everything worked fine.
I am running nightly builds using cc.net 1.4.2. I am also using nunit2.4.8. If I force the build manually it works fine but most of my nightly schduled builds fails saying testfixture setup failed. Is this some bug in nunit2.4.8 or something othert thing as I cant find out the reason
Is it the same cc.net project being forced that is scheduled to run nightly? If not, there may be a difference in how they are configured. If they are the same, I would suggest adding some logging to the test fixture setup code to see if you can track down the problem. Perhaps there is a nightly task running on the server (backup or virus scan perhaps) that is causing the issue. Another possibility is a task running on another server (taking a database offline for example.) I don't believe this is a bug in NUnit.