I am running nightly builds using cc.net 1.4.2. I am also using nunit2.4.8. If I force the build manually it works fine but most of my nightly schduled builds fails saying testfixture setup failed. Is this some bug in nunit2.4.8 or something othert thing as I cant find out the reason
Is it the same cc.net project being forced that is scheduled to run nightly? If not, there may be a difference in how they are configured. If they are the same, I would suggest adding some logging to the test fixture setup code to see if you can track down the problem. Perhaps there is a nightly task running on the server (backup or virus scan perhaps) that is causing the issue. Another possibility is a task running on another server (taking a database offline for example.) I don't believe this is a bug in NUnit.
Related
Puzzling out how to implement VSTS for testing team.
Regarding the scenario where a manual test is run and a bug is created...
The bug gets prioritized and fixed at some later point.
How does the application indicate to the tester when they can run the test again because the bug has been fixed?
When you run the tests each time it has its own test run ID. That means they will have their own test results even if you run the same test case multiple times (They have different test run IDs).
Generally, once the bug is fixed and the fixed sources are integrated into the next release of the application, then you can run the test again to check if the bug is really fixed or not.
More information about the manual tests please see Run manual tests and FAQs for manual testing
The software I worked on has both unit tests and system tests. System tests can take minutes to run, they take input values and we validate the results against expected output. There are hundreds of system tests. The software must be built (done this) and tested on both windows and Linux.
How can I automate testing with VSTS? I'd like to avoid doing this at build stage, because it would slow the builds down. I can't see how to automate this in the Test stage. Do I need additional extensions to do this? Everything seems so geared up for web development, e.g. selenium tests, how do we run automated tests for good old binary programs?
I would suggest using Release Management to deploy your application to a test environment and then run your tests as a part of your Release Definition. You can then choose to run tests in parallel to make sure that your system tests don't take days to run.
On a side note, having so many system tests is a code smell. I would suggest looking into building as many fast running unit tests as possible and only using system tests when absolutely necessary.
I have a VSO release management definition in which I'm deploying a cloud service and then running some tests. The deployment executes without issues, but then the tests don't run, I recieve the following message in the logs:
Warning: No test is available in My DLL Path. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.
Now, the strange thing is that this release is triggered by a build, which runs exactly the same set of tests and they all run happily.
I've included a runsettings file specifying the framework version (based on some SO posts I found from a year ago with a similar issue) but its made no difference. Been messing with this for nearly 2 days now with no progress. Any suggestions happily accepted!
Arrrrgh! So it turns out, if I deleted the whole project, created it again and added my tests again, it just works. Gremlins apparently!
Admins, if this needs to be deleted, go ahead
My project at work used Urban Code Deploy (UCD) for its continuous deployment process. My code runs locally and passes all unit tests, but the build group says that my code is failing the FVT test being run by UCD. Is there any way to run this FVT test locally, or at least attempt to run it, so I can hopefully figure out what is failing?
Mike
UC Deploy isn't a testing tool. So the team that has set it up, has it running some other testing tool at the end of hte deployment (which is pretty normal).
So you'll need to ask them what testing tool they're using and go from there.
If you can see how the build group is deploying your code, you should be able to see what testing they are doing, and then be able to replicate that in your own environment. Often the code changes and changes in requirements will not be reflected in the FVT tests, and you need to deliver updated FVT test scripts in conjunction with your code changes.
I'm using TeamCity for my CI builds, and I'd like to set up a second build for running automated UI tests on Windows XP and Windows 7 virtual machines.
I imagine the build working as follows:
Compile, run unit tests, etc.
Prepare MSI using WiX
Copy MSI to target test machines
Remotely execute MSI's
Copy test harness code to remote machine
Run tests
Build finishes
The automated UI tests are written using NUnit and would need to be run directly on the test virtual machine (they can't run remotely). It's important that if the tests fail, it appears in the TeamCity build log and the build fails. I'd rather not install VS or the TeamCity build agents on either of the test virtual machines.
It seems that most of this should be possible using psexec.exe. Are there any alternative (preferably open source) tools that I should look at?
takes a deep breath
We were looking into something to help us out with our automated UI tests. We use ranorex to test the UI and TeamCity/Msbuild to execute the tests.
We never found any tools to help us out (I’m constantly keeping an eye out for some so will monitor this thread) but here is what we did instead.
The CI server copies the setup files and test scripts to the Testing Host Server.
The CI server then launches a custom app on the Testing Host Server providing the name of the VM to launch.
The Test Host Server then launches the VM software, using Virtual PC.exe -singlepc -pc vhdname.vhd -launch , and waits for it to shutdown (after it’s run its tests).
The VM grabs the setup files and scripts from the network location and executes.
After the tests are run it then returns the results to a networked location and shuts itself down.
Control is returned to the custom app.
Control is returned to the CI server which determines from the results if it has passed or failed (and updates the UI so developers are made aware of the result).
Results are collection as artifacts in TeamCity and tagged in Svn.
I think that's everything. Its convoluted, however, it works. Hope someone of that helps you.
Jeff Brown of the Gallio team has been talking about a tool called Archimedes that he's planning to write to support this kind of requirement. It sounds promising, but I don't think there has been much progress on it so far.
In the mean time though, there is something in the Gallio project called VM Tool that may do what you want. It provides commands to stop, start and snapshot virtual machines, and more importantly, to copy files back and forth and execute commands.
I presume you have also considered Powershell Remoting?