How to stop NUnit ITestRunner? - nunit

Question

Using nunit.engine 3.10.0, I can't stop an asynchronously running ITestRunner. The TestPackage is set up to be executed locally, i.e. InProcess and in the current AppDomain. No more tests are started after the second test as expected, but the while loop never ends.
public static void Main(string[] args)
{
// 2 assemblies x 2 TestFixtures each x 2 Tests each = 8 test cases
string[] testAssemblyFileNames = { TestAssemblyFileName1, TestAssemblyFileName2 };
string assemblyDirectory = Path.GetDirectoryName(Uri.UnescapeDataString(
new UriBuilder(Assembly.GetExecutingAssembly().CodeBase).Path));
// Nunit 3.10.0
var minVersion = new Version("3.4");
ITestEngine testEngine = TestEngineActivator.CreateInstance(minVersion);
// configure a test package that executes
// in the current process and in the current domain
var testPackage = new TestPackage(testAssemblyFileNames);
testPackage.AddSetting(EnginePackageSettings.ProcessModel, "InProcess");
testPackage.AddSetting(EnginePackageSettings.DomainUsage, "None");
testPackage.AddSetting(EnginePackageSettings.DisposeRunners, "True");
testPackage.AddSetting(EnginePackageSettings.WorkDirectory, assemblyDirectory);
ITestRunner testRunner = testEngine.GetRunner(testPackage);
// prepare a listener that stops the test runner
// when the second test has been started
const bool StopAfterSecondTest = true;
int testStartedCount = 0;
var listener = new MyTestEventListener();
listener.TestStarted += (sender, eventArgs) =>
{
testStartedCount++;
if ( StopAfterSecondTest && testStartedCount == 2 )
{
testRunner.StopRun(force: true);
}
};
var testFilterBuilder = new TestFilterBuilder();
TestFilter testFilter = testFilterBuilder.GetFilter();
ITestRun testRun = testRunner.RunAsync(listener, testFilter);
bool keepRunning;
int loopCount = 0;
do
{
bool completed = testRun.Wait(500);
bool running = testRunner.IsTestRunning;
keepRunning = !completed && running;
loopCount++;
} while ( keepRunning );
Console.WriteLine($"Loop count: {loopCount}");
XmlNode resultNode = testRun.Result;
Console.WriteLine(resultNode.InnerText);
Console.ReadKey();
}
private class MyTestEventListener : ITestEventListener
{
private const string TestCaseStartPrefix = "<start-test";
private const string TestMethodTypeAttribute = " type=\"TestMethod\"";
public event EventHandler<EventArgs> TestStarted;
public void OnTestEvent(string report)
{
if ( report.StartsWith(TestCaseStartPrefix) &&
report.Contains(TestMethodTypeAttribute) )
{
TestStarted?.Invoke(this, new EventArgs());
}
}
}
If I skip waiting and try to get the test result, I get an InvalidOperationException: 'Cannot retrieve Result from an incomplete or cancelled TestRun.'
How can I stop the test runner and get the results of the tests that were completed before the stopping?

Answer 1

You can't do it from inside a test. Your listener is executed in the context of the test itself. For that reason, listeners are specifically forbidden from trying to change the outcome of a test. Additionally, the event is buffered and may not even be received in this case until after the test run is complete.
StopRun is intended to be called by the main runner itself, generally as triggered by some user input.
You should also take note of this issue: https://github.com/nunit/nunit/issues/3276 which prevents StopRun(true) from working under any circumstances. It was fixed in PR https://github.com/nunit/nunit/pull/3281 but is not yet in any release of the framework. You will have to either use a recent dev build of the framework or switch to StopRun(false).

Answer 2

Based on the answer by #Charlie, this is how to modify the code in order to stop all threads:
public static void Main(string[] args)
{
// 2 assemblies x 2 TestFixtures each x 2 Tests each = 8 test cases
// each test case includes a 200 ms delay
string[] testAssemblyFileNames = { TestAssemblyFileName1, TestAssemblyFileName2 };
string assemblyDirectory = Path.GetDirectoryName(Uri.UnescapeDataString(
new UriBuilder(Assembly.GetExecutingAssembly().CodeBase).Path));
// Nunit 3.10.0
var minVersion = new Version("3.4");
ITestEngine testEngine = TestEngineActivator.CreateInstance(minVersion);
// configure a test package that executes
// in the current process and in the current domain
var testPackage = new TestPackage(testAssemblyFileNames);
testPackage.AddSetting(EnginePackageSettings.ProcessModel, "InProcess");
testPackage.AddSetting(EnginePackageSettings.DomainUsage, "None");
testPackage.AddSetting(EnginePackageSettings.DisposeRunners, "True");
testPackage.AddSetting(EnginePackageSettings.WorkDirectory, assemblyDirectory);
ITestRunner testRunner = testEngine.GetRunner(testPackage);
var listener = new TestStartListener();
var testFilterBuilder = new TestFilterBuilder();
TestFilter testFilter = testFilterBuilder.GetFilter();
ITestRun testRun = testRunner.RunAsync(listener, testFilter);
// wait until the first test case has been started
while ( listener.Count < 1 )
{
Thread.Sleep(50);
}
bool keepRunning = true;
while ( keepRunning )
{
int testStartedCount = listener.Count;
testRunner.StopRun(force: false);
Writer.WriteLine($"{GetTimeStamp()}, Stop requested after {testStartedCount} test cases.");
// wait for less time than a single test needs to complete
bool completed = testRun.Wait(100);
bool running = testRunner.IsTestRunning;
Writer.WriteLine($"{GetTimeStamp()} Completed: {completed}, running: {running}");
keepRunning = !completed && running;
}
listener.WriteReportsTo(Writer);
XmlNode resultNode = testRun.Result;
Writer.WriteLine("Test result:");
resultNode.WriteContentTo(ResultWriter);
Console.ReadKey();
}
private class TestStartListener : List<string>, ITestEventListener
{
private const string TestCaseStartPrefix = "<start-test";
private const string TestMethodTypeAttribute = " type=\"TestMethod\"";
public event EventHandler<EventArgs> TestStarted;
public void OnTestEvent(string report)
{
if ( report.StartsWith(TestCaseStartPrefix) &&
report.Contains(TestMethodTypeAttribute) )
{
Add($"{GetTimeStamp()}, {report}");
TestStarted?.Invoke(this, new EventArgs());
}
}
public void WriteReportsTo(TextWriter writer)
{
Writer.WriteLine($"Listener was called {Count} times.");
foreach ( var report in this )
{
Writer.WriteLine(report);
}
}
}
The two test assemblies get executed in the runner's process, in a single domain and on two threads, one for each test assembly. In total, two test methods get executed and pass; one for each of the two test assemblies. Other test methods do not get executed and not reported. Other test fixtures (classes) do not get executed and get reported with result="Failed" label="Cancelled".
Note that testRunner.StopRun(force: false) is called repeatedly. If only called once, the other thread will run to completion.