I have 13 separate but related architecture assemblies, and 13 separate NUnit test assemblies, each one containing all the test fixtures for its matching architecture assembly. I am using NUnit 2.5.2 (latest version currently).
I can run each test assembly separately in the NUnit GUI and all the tests pass. However, when I come to combine them into a single NUnit project file, NUnit insists on applying a single config file to the whole test run. This won't work because each test assembly requires different config. I can't merge them into one "uber-config" file because some of the sections are mutually exclusive. I have tried running each assembly in the project in separate AppDomains, and also separate processes, but in both cases it fails to use the DLL-specific config file, so all the tests crash and burn.
I have done a Google search but so far I have not found any indication that NUnit supports this scenario. Am I right, or have I missed something?
I have tried my hardest to re-architecture the tests so that they could share the same config file, but I've had to admit defeat on that front.
NUnit 2.5 has as setting where you can enable each assembly to run in a separate AppDomain. By doing this, NUnit will load the config for the assembly and not the one for the .nunit project.
For more info, see here:
http://nunit.org/index.php?p=settingsDialog&r=2.5
In the past I've done this with a batch file running each assembly through the nunit console independently. At one point I had something that merged the xml output together. It might be in the CruiseControl.Net code.
I haven't worked on the NUnit project for a while. I only have the older code in my head. But the issue is that you get one config per AppDomain and NUnit loads all the test assemblies into one AppDomain.
You might want to try alternate runners such as Resharper or TestDriven.net
Related
I did some research and I always found that the tests are located in some dll. However in my project there are no .dlls created and the tests are located in the .exe file.
Do I have to set some specific parameters?
This is the log from vsts after I pointed the path for test assemblies to the exe. Before I pointed it to **\*.test*.dll but it could never find any tests. When running the tests or build locally there is also never a test dll created.
2017-08-15T08:55:15.1386003Z Starting test execution, please wait...
2017-08-15T08:55:15.4276084Z Information: NUnit Adapter 3.8.0.0: Test execution started
2017-08-15T08:55:15.4276084Z
2017-08-15T08:55:15.4276084Z Information: Running all tests in d:\a\1\s\Workshop\Workshop\bin\Release\Workshop.exe
2017-08-15T08:55:15.4286085Z
2017-08-15T08:55:16.8199687Z Information: NUnit3TestExecutor converted 1 of 1 NUnit test cases
2017-08-15T08:55:16.8199687Z
2017-08-15T08:55:16.9559755Z Information: NUnit Adapter 3.8.0.0: Test execution complete
2017-08-15T08:55:16.9559755Z
2017-08-15T08:55:17.2181003Z Passed TwoIntegers_Add_Sum
How do I make the tests show up as .dlls instead of being inside the .exe?
As your example shows, NUnit as well as VS Test Explorer are very happy to have tests in an .exe along with the production code. Most people prefer to have the tests separate from the production code, which is distributed to users, so they create a separate project to contain them.
So, your answer is that there is no way to make the tests "show up" as a .dll if they are actually in the .exe. OTOH, assuming you are the developer of the application, you can create a separate project, which creates a class library, and move the tests into that project.
So my nunit 2.6 tests run in team city, the results appear in a 'tests' tab. I just select the built in nunit build step, type the name of the file with the tests in "Runs Tests from:" and the name of the category(s) in "NUnit categories include:"
However, If I upgrade my tests and select NUnit3 from the "NUnit runner" drop down things start to go wrong.
After much googling and mucking round for command lines and console runners I can't get this working with the full 'results in a tab' level of compatibility.
There is a lot of advice out there on how to make this work, but the best articles are at least a year old and I'm not at all clear that they apply to 3.5.0
Has anyone got this fully working or is it no longer supported? Can you explain how you managed it?
NUnit 3.5.0 does work with TeamCity, but it requires that you use the NUnit TeamCity Event Listener Extension. You can install it along with the NUnit Console Runner package or use the NUnit Console Runner with Extensions which includes the TeamCity extension along with other commonly used NUnit extensions.
Once you have the extension, TeamCity should automatically add the --teamcity command line option to the nunit3-console.exe.
Check here more information on the various NUnit Runner NuGet packages and what is included in each.
You add these packages to one of your test projects. That will cause them to be installed in the package directory of your solution root. From there, the built in NUnit 3 step will work, just update the executable location to point to 3.5.0. See the Getting Started With NUnit and TeamCity document. I would use the Case 4, NUnit Build Step.
If you only want to test certain categories, you will need to add your --where clause as an additional command line parameter.
I'm using VS 2013 and NUnit.
I'm running my tests in Resharpers's TestRunner GUI - this thing:
I have multiple projects (Solution.Foo, Solution.Bar, etc), and tests for each project in Solution.Foo.Tests, Solution.Bar.Tests, etc.
One of my tests in Foo.Tests is accessing a file, located in the Solution.Foo.Tests Folder.
I use TestContext.CurrentContext.TestDirectory to locate it and when I run that test it all works just fine.
If I run the whole suite of tests in Foo.Tests, (i.e. click on Solution.Foo.Tests in the Runner GUI and run) then it's fine.
But if I select one test from Bar.Tests and my file-reading test from Foo.Tests, then it fails, because suddenly TestDirectory is set to a path in Bar.Tests.
It's as if NUnit set TestDirectory once when it starts running a collection of tests, determined by where the first test is and then never updates it again.
Any ideas what's going on and how to fix it?
ReSharper has an optimisation for speed, by sharing an AppDomain across multiple assemblies. The downside to this is that it will pick an arbitrary assembly to be the main one, which means the directory can be wrong (and it might not pick up the correct app.config, too).
You can disable the optimisation in ReSharper → Options → Unit Testing, by checking the "Use separate AppDomain for each assembly with tests".
Well we are facing a strange problem with JetBrains TeamCity induced unit tests on our main project where tests from few library projects are failing regularly. Apparently, it's not reading the config file (coming from app.config and nicely stored in project -> bin -> debug -> projectName.dll.config).
Hints or tips on what could be the real issue would be highly appreciated.
I've got the same problem and wasted a couple of hours to figure out what the problem is.
In our case, the NUnit plugin was configured to run the tests from:
**\*Tests.dll
Though this sounds to be OK, it has turned out that this pattern will not only match to the MyTests.dll in the bin\Debug folder but also to the obj\Debug\MyTests.dll. The obj folder is used internally for the compilation and does not contain the config file.
Finally the solution was to change the plugin configuration to
**\bin\Debug\*Tests.dll
Actually we use a system variable for the build configuration so we did not have the "Debug" hard-coded. Using bin* might be also dangerous when the workspace is also used for Debug/Release builds and you don't have a full cleanup specified.
You might wonder why I did not realize the test count mismatch (actually it was doubled, because they were running once from bin and once from obj), but this is typical: while everything is green, you don't care about the count. When we have introduced the first test depending on the config, we had only one failure (because the one from bin was passing), so the duplication was not outstanding.
In addition to Gaspar Nagy's accepted answer, check to see if your project has multiple test dlls and one of them is referencing another.
This causes the referenced dll to be run twice, and the copy that was in the other dll's folder to not have the proper app.config entries. The proper fix is to remove any and all references from the other test project.
TeamCity (v6.5.4) has its own NUnit Test Runner and there seems to be an inconsistency between it and the NUnit GUI test runner (2.5.10). The NUnit GUI Test Runner follows the long standing convention of expecting the configuration file name to be of the format .config. You can see this in NUnit by looking at Project -> Edit...
TeamCity on the other hand is looking for an app.config.
Your options are to either:
Set the NUnit GUI to point to app.config and include the resultant
nunit project in your source control.
Have both an app.config and a .config - syncing both
manualy.
Add a step to your build process to copy .config to
app.config (or vice versa).
I had similar woes
This may help; additionally we had issues where this still would not work - we ended up copying the relevant config sections into the highest level config file. (i.e. if it was a web app copy it into the Web.Config) - fairly kludgy but we had wasted a few days on the issue
I learned recently that app.config files are not read for a class library... Maybe this link could help :)
app.config for a class library
If you need a config file for your "unit" tests then you are doing it wrong. Proper unit testing never needs configuration or access to the database, file system etc. You should change your testing strategy.
Good point to start is mark your tests that need configuration with the[Category("Integration")] annotation and set the Teamcity test runner to ignore this category. Then you should focus on refactoring these test.
I've got a large acceptance test (~10 seconds per test) test suite written using NUnit. I would like to make use of the fact that my machines are all multiple core boxes. Ideally, I'd be able to have one test running per core, independently of other tests.
There is PNUnit, but it's designed for testing for threading synchronization issues and things like that, and I didn't see an obvious way to accomplish this.
Is there a switch/tool/option I can use to run the tests in parallel?
If you want to run NUnit tests in parallel, there are at least 2 options:
NCrunch offers it out of the box (without changing anything, but is a commercial product)
NUnit 3 offers a Parallelizable attribute, which can be used to denote which tests can be run in parallel
NUnit version 3 will support running tests in parallel:
Adding the attribute to a class: [Parallelizable(ParallelScope.Self)] will run your tests in parallel.
• ParallelScope.None indicates that the test may not be run in parallel
with other tests.
• ParallelScope.Self indicates that the test
itself may be run in parallel with other tests.
• ParallelScope.Children indicates that the descendants of the test may
be run in parallel with respect to one another.
• ParallelScope.Fixtures indicates that fixtures may be run in parallel
with one another.
NUnit Framework-Parallel-Test-Execution
If your project contains multiple test DLLs you can run them in parallel using this MSBuild script. Obviously you'll need to tweak the paths to suit your project layout.
To run with 8 cores run with: c:\proj> msbuild /m:8 RunTests.xml
RunTests.xml
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="RunTestsInParallel" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"/>
<PropertyGroup>
<Configuration Condition=" '$(Configuration)' == '' ">Release</Configuration>
<Nunit Condition=" '$(Nunit)' == '' ">$(MSBuildProjectDirectory)\..\tools\nunit-console-x86.exe</Nunit>
</PropertyGroup>
<!-- see http://mikefourie.wordpress.com/2010/12/04/running-targets-in-parallel-in-msbuild/ -->
<Target Name="RunTestsInParallel">
<ItemGroup>
<TestDlls Include="..\bin\Tests\$(Configuration)\*.Tests.dll" />
</ItemGroup>
<ItemGroup>
<TempProjects Include="$(MSBuildProjectFile)" >
<Properties>TestDllFile=%(TestDlls.FullPath)</Properties>
</TempProjects>
</ItemGroup>
<MSBuild Projects="#(TempProjects)" BuildInParallel="true" Targets="RunOneTestDll" />
</Target>
<Target Name="RunOneTestDll">
<Message Text="$(TestDllFile)" />
<Exec Command="$(Nunit) /exclude=Integration $(TestDllFile) /labels /xml:$(TestDllFile).results.xml"
WorkingDirectory="$(MSBuildProjectDirectory)\..\bin\Tests\$(Configuration)" />
</Target>
</Project>
Update
If I were answering this question now I would highly recommend NCrunch and its command line test running tool for maximum test run performance. There's nothing like it and it'll revolutionise your code-test-debug cycle at the same time.
As an alternative to adding the Parallelizable attribute to every test class:
Add this into the test project AssemblyInfo.cs class for nunit3 or greater:
// Make all tests in the test assembly run in parallel
[assembly: Parallelizable(ParallelScope.Fixtures)]
In this article it is mentioned that in order to speed up tests the poster runs multiple instances of NUnit with command parameters specifying which tests each instance should run.
FTA:
I ran into an odd problem.
We use nunit-console to run test on
our continuous integration server.
Recently we were moving from Nunit
2.4.8 to 2.5.5 and from .Net 3.5 to 4.0. To speed up test execution we run multiple instances of Nunit in
parallel with different command line
arguments
We have two copies of our test assemblies and the nunit binaries in
folder A and B.
In folder A we execute
nunit-console-x86.exe Model.dll
Test.dll /exclude:MyCategory
/xml=TestResults.xml
/framework=net-4.0 /noshadow
In folder B we execute
nunit-console-x86.exe Model.dll
Test.dll /include:MyCategory
/xml=TestResults.xml
/framework=net-4.0 /noshadow
If we execute the commands in sequence
both run successfully. But if we
execute them in parallel only one
succeeds. As far as I can tell it's
the one that first loads the test
fixtures. The other fails with the
message "Unable to locate fixture".
Is this problem already known? I could
not find anything related in the bug
list on launchpad. BTW Our server runs
Windows Server 2008 64-bit. I could
also reproduce the problem on Windows
7 64-bit.
Assuming this bug is fixed or you are not running the newer version(s) of the software mentioned you should be able to replicate their technique.
Update
TeamCity looks like a tool you can use to automatically run NUnit tests. They have an NUnit launcher discussed here that could be used to launch multiple NUnit instances. Here is a blog post discussing the mergind of multiple NUnit XML results into a single result file.
So theoretically you could have TeamCity automatically launch multiple NUnit tests based on however you want to split up the workload and then merge the results into a single file for post test processing.
Is that automated enough for your needs?
Just because PNUnit can do synchronization inside test code doesn't mean that you actually have to use that aspect. As far as I can see there's nothing to prevent you from just spawning a set and ignoring the rest till you need it.
BTW I don't have the time to read all of their source but was curious to check out the Barrier class and it's a very simple lock counter. It just waits till N threads enter and then sends the pulse for all of them to continue running at the same time. That's all there is to it - if you don't touch it, it won't bite you.
Might be a bit counter intuitive for a normal threaded development (locks are normally used to serialize access - 1 by 1) but it is quite a spirited diversion :-)
You can now use NCrunch to parallelize your unit tests and you can even configure how many cores should be used by NCrunch and how many should be used by Visual Studio.
plus you get continuous testing as a bonus :)
It would be a bit of a hack, but you could split the unit tests into a number of categories. Then, start up a new instance of NUnit for each category.
Edit: It looks like they have added a /process option to the console app. The command-line help states this is the "Process model for tests: Single, Separate, Multiple". The test runner also appears to have this feature.
Edit 2: Unfortunately, although it does create separate processes for each assembly, the process isolation option (/process from the command line) runs the agents one at a time.
Since the project hasn't been mentioned here, I would like to bring up NUnit.Multicore. I haven't tried the project myself, but it seems to have an interesting approach to the parallel test problem with NUnit.
You can try my small tool TBox or console parallel Runner or even plugin to do distributed calulations, which also can run unit tests on the set of PCs SkyNet
TBox is created to simplify work with big solutions, which contains many projects. It supports many plugins and one of them provide ability to run NUnit tests in parallel. This plugin does not require any changes to your existing tests.
Also it support:
Cloning of the folder with unit test (if your tests changes local data),
Synchronizations of the tests (for example if your tests on
testfixtureteardown kills all dev servers or chromerunner for qunit )
x86 mode and Admin privileges to run tests
Batch run - you can run tests for many assemblies in parallel
Even for single thread run, works faster than standart nunit runner, if you have much small tests.
Also this tool supports command line tests runner (for parallel run) and you can use it with continuous integration.
I have successfully used NUnit 3.0.0 beta-4 to run tests in parallel
Runs on build server
Runs Selenium tests
Has Visual Studio support
no Resharper support yet
Thanks for peers answer.
Gotchas:
Parallelizable attribute is not inherited, so it has to be specified on the test class.
You can use following PowerShell command (for NUnit3, for NUnit2 change runner name):
PS> nunit3-console (ls -r *\bin\Debug\*.Tests.dll | % FullName | sort-object -Unique)
Presented command runs all test assemblies in single nunit instance, which allows to leverage engine built-in parallel test run.
Remarks
Remember to tweak directory search pattern. Given example runs only assemblies ending with .Tests.dll and inside \bin\Debug directories.
Be aware of Unique filtering - you may not want to have it.
To achieve level of parallelism ensure to do these two:
1)Nunit Explorer - Settings - Run tests in parallel
2)LevelOfParallelism
This is an assembly-level attribute used to specify the level of parallelism, that is, the maximum number of worker threads executing tests in the assembly.
In Assemblyinfo.cs, set
[assembly:LevelOfParallelism(N)] => here N is number