I have 3 normal and 1 explicit test but when I run my test using the Test Explorer window I get this output under "Tests" in the Output window
========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running all tests in xyz.dll
NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Explicit run
ExplicitMethod(03/10/2022 08:00:00,03/10/2022 16:00:00): OneTimeSetUp:
NUnit Adapter 4.2.0.0: Test execution complete
========== Test run finished: 1 Tests (0 Passed, 0 Failed, 0 Skipped) run in 462 ms ==========
*Note I'm using NUnit Adapter 4.2.1 instead of 4.2.0 so that's already weird
And this is how the Test Explorer window looks
Test Explorer window
*Note the full blue test is the Explicit test that get's skipped like it should
This is a problem because it does seem to only discover tests which are explicit.
The tests I want to run are of course not explicit, here is an example
[Test]
[TestCaseSource(nameof(TestNameData))]
public async Task<float> TestName(DateTime start, DateTime end, List<CalculateHoursObj>? list = default)
{
if (list == null) list = new List<CalculateHoursObj>();
return await EmployeeService.CalculateOverTimeHours(start, end, list);
}
public static IEnumerable TestNameData
{
get
{
yield return new TestCaseData(TenthMarch8_2022, TenthMarch16_2022).Returns(8.0f);
}
}
It fails to discover and/or run this test.
But curiously if I break all my non explicit tests by making the data non-static like this
public IEnumerable TestNameData
{
get
{
yield return new TestCaseData(TenthMarch8_2022, TenthMarch16_2022).Returns(8.0f);
}
}
It of course breaks the test and when I run all tests it does actually discover all tests
========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running selected tests in xyz.dll
NUnit3TestExecutor discovered 2 of 4 NUnit test cases using Current Discovery mode, Non-Explicit run
NUnit Adapter 4.2.0.0: Test execution complete
========== Test run finished: 2 Tests (0 Passed, 2 Failed, 0 Skipped) run in 394 ms ==========
But even now it only runs 2 of the 3 broken non explicit tests and of course they all fail
I have looked up everything online for 1.5 hours and really can't find a solution.
Don't bother responding with "have you updated visual studio or the nugget packages"
This question was part of a bug the solution is to not use new DateTime() withing nunit tests or to have nunit test be discover with id's using the <UseParentFQNForParametrizedTests>true</UseParentFQNForParametrizedTests> <UseNUnitIdforTestCaseId>true</UseNUnitIdforTestCaseId> flags
This was a bug because it was not reported correctly in the output window and/or Error List window.
I've created an Issue on GitHub about this if anyone wants to read.
Now stop editing this post because it truly is answered and the Issue link which I included shows the valid, correct and readable answer by "OsirisTerje" whom explains it way better than I could.
Related
I have .net core 2.1 project build pipeline as shown below.
I have 2 simple unit test cases which executed successfully. “.NET Core test” command executes successfully and publishes thereport.
2018-09-06T22:05:41.8494077Z
2018-09-06T22:05:42.0138714Z Passed Login_Test
2018-09-06T22:05:42.0139277Z Passed Logoff_Test
2018-09-06T22:05:42.0495295Z Results File: P:\*******2018-09-06_17_05_41.trx
2018-09-06T22:05:42.0514262Z
2018-09-06T22:05:42.0516990Z Attachments:
2018-09-06T22:05:42.0517972Z P:\*****\TestResults\329a8629-7ec1-454b-97d8-2623d29bbd0e\XXXXX 2018-09-06 17_05_39.coverage
2018-09-06T22:05:42.0518130Z
2018-09-06T22:05:42.0518516Z Total tests: 2. Passed: 2. Failed: 0. Skipped: 0.
2018-09-06T22:05:42.0519312Z Test Run Successful.
2018-09-06T22:05:42.0528123Z Test execution time: 1.9003 Seconds
2018-09-06T22:05:42.1424722Z Publishing test results to test run '90'
2018-09-06T22:05:42.1424881Z Test results remaining: 2. Test run id: 90
2018-09-06T22:05:42.1592889Z ##[section]Async Command Start: Publish test results
2018-09-06T22:05:42.2384075Z Published Test Run : http://******/_TestManagement/Runs#runId=90&_a=runCharts
2018-09-06T22:05:42.2384433Z ##[section]Async Command End: Publish test results
2018-09-06T22:05:42.2385471Z ##[section]Finishing: VsTest - testAssemblies
But I don’t see any “Test Results” under dashboard.
I included "Visual Studio Test" too, but still dont see the result in the dashboard. What am I missing here?
UPDATE:
AutoFixture team released a fix for this in version 3.51.
Simply extend the AutoDataAttribute doing so:
public class AutoDataFixedNameAttribute : AutoDataAttribute
{
public AutoDataFixedNameAttribute()
{
this.TestMethodBuilder = new FixedNameTestMethodBuilder();
}
}
Then use this new attribute instead of the built-in AutoData in your NUnit tests.
Starting from v4, this behavior is the default one.
Previous post
I'm trying to use AutoFixture with NUnit and Moq, using the following AutoMoqDataAttribute :
public class AutoMoqDataAttribute : AutoDataAttribute
{
public AutoMoqDataAttribute()
: base(new Fixture().Customize(new AutoMoqCustomization()))
{
}
}
But when I run this test :
[Test, AutoMoqData]
public void Test(Mock<IUser> user)
{
// do stuff with user
}
The test never runs. AutomMoqData is hit correctly, but the code inside the test is never executed and everything ends without any warning with the following message :
Test adapter sent back a result for an unknown test case. Ignoring result for 'Test(Mock<Sandbox.IUser>)'
The test also doesn't appear in the test runner list.
But if I remove the parameter :
[Test, AutoMoqData]
public void Test()
{
// do stuff without user
}
Everything runs fine, but this is less useful without the parameters passed :)
Am I missing something here ?
Here is the list of the Nuget packages versions :
<package id="AutoFixture" version="3.50.2" targetFramework="net452" />
<package id="AutoFixture.AutoMoq" version="3.50.2" targetFramework="net452" />
<package id="AutoFixture.NUnit3" version="3.50.2" targetFramework="net452" />
<package id="Moq" version="4.5.3" targetFramework="net452" />
<package id="NUnit" version="3.7.1" targetFramework="net452" />
EDIT:
Following #MarkSeemann's advice, I filed an issue on Github.
Visual Studio Test Runner repro
This looks like an issue with the NUnit Visual Studio test adapter. I can reproduce the issue when I also add the NUnit3TestAdapter package to my repro solution.
I'm also assuming that the test class has the [TestFixture] attribute, so that the entire repro class looks like this:
[TestFixture]
public class Tests
{
[Test, AutoMoqData]
public void Test(Mock<IUser> user)
{
Assert.NotNull(user);
}
}
When I attempt to run all tests using Visual Studio 2015's test runner, the test never runs, and this is the output to test output window:
------ Run test started ------
NUnit Adapter 3.7.0.0: Test execution started
Running all tests in C:\Users\mark\Documents\Stack Overflow\44564377\44564377\bin\Debug\Ploeh.StackOverflow.Q44564377.dll
NUnit3TestExecutor converted 1 of 1 NUnit test cases
NUnit Adapter 3.7.0.0: Test execution complete
Test adapter sent back a result for an unknown test case. Ignoring result for 'Test(Mock<Ploeh.StackOverflow.Q44564377.IUser:8e33>)'.
========== Run test finished: 0 run (0:00:01,1763498) ==========
TestDriven.Net
If, on the other hand, I try to run it with TestDriven.Net, it runs just fine:
------ Test started: Assembly: Ploeh.StackOverflow.Q44564377.dll ------
1 passed, 0 failed, 0 skipped, took 0,79 seconds (NUnit 3.7.1).
TestDriven.Net is sometimes extraordinarily tolerant of small errors in the test code, so this may not be that telling in itself.
NUnit 3 console runner
Since TestDriven.Net may be too liberal in what it accepts, a better test would be to try with the official NUnit 3 console runner:
$ packages/NUnit.ConsoleRunner.3.6.1/tools/nunit3-console.exe 44564377/bin/Debug/Ploeh.StackOverflow.Q44564377.dll
NUnit Console Runner 3.6.1
Copyright (C) 2017 Charlie Poole
Runtime Environment
OS Version: Microsoft Windows NT 10.0.15063.0
CLR Version: 4.0.30319.42000
Test Files
44564377/bin/Debug/Ploeh.StackOverflow.Q44564377.dll
Run Settings
DisposeRunners: True
WorkDirectory: C:\Users\mark\Documents\Stack Overflow\44564377
ImageRuntimeVersion: 4.0.30319
ImageTargetFrameworkName: .NETFramework,Version=v4.6.1
ImageRequiresX86: False
ImageRequiresDefaultAppDomainAssemblyResolver: False
NumberOfTestWorkers: 4
Test Run Summary
Overall result: Passed
Test Count: 1, Passed: 1, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
Start time: 2017-06-15 11:09:21Z
End time: 2017-06-15 11:09:22Z
Duration: 0.933 seconds
Results (nunit3) saved as TestResult.xml
This, too, successfully executes the test.
Interim conclusion
Since both the official console runner and TestDriven.Net successfully executes the test, I'd tentatively conclude that this looks like a defect in the NUnit3TestAdapter package. May I suggest filing an issue for it?
So I am trying to run a powershell script that is triggered by TeamCity to run specific unit tests based on the names of the files that were changed on each github commit.
Here is how I am running it from the command line:
C:\MyFolder\bin\NUnit.ConsoleRunner.3.4.1\tools\nunit3-console.exe "C:\MyFolder\Bin\UnitTesting.dll" --test="MyFolder.QuickTests.DaoTests.ProductDaoTests.ProductBasicTest"
But I keep getting this, it runs it just never runs any tests:
NUnit Console Runner 3.4.1
Copyright (C) 2016 Charlie Poole
Runtime Environment
OS Version: Microsoft Windows NT 10.0.14393.0
CLR Version: 4.0.30319.42000
Test Files
MyFolder\Bin\UnitTesting.dll
Test Filters
Test: MyFolder.QuickTests.DaoTests.ProductDaoTests.ProductBasicTest
Run Settings
WorkDirectory: C:\Users\Me
ImageRuntimeVersion: 4.0.30319
ImageTargetFrameworkName: .NETFramework,Version=v4.0
ImageRequiresX86: False
ImageRequiresDefaultAppDomainAssemblyResolver: False
NumberOfTestWorkers: 2
Test Run Summary
Overall result: Passed
Test Count: 0, Passed: 0, Failed: 0, Inconclusive: 0, Skipped: 0
Start time: 2016-10-17 20:28:43Z
End time: 2016-10-17 20:28:43Z
Duration: 0.303 seconds
Results (nunit3) saved as TestResult.xml
Now when I run it without the --test command like this:
C:\MyFolder\bin\NUnit.ConsoleRunner.3.4.1\tools\nunit3-console.exe "C:\MyFolder\Bin\UnitTesting.dll"
It runs every unit-test that we have, but I don't want to run them all, I want to run specific quick ones, and only run the large ones when we go to staging/production servers so our developers don't have to wait 15 to 20 minutes every time they commit something.
Some additional info:
-My namespace that I am using for this is
MyFolder.QuickTests.DaoTests.ProductDaoTests
The Class I am calling is:
ProductBasicTest
Some of the names like the folder directories were changed because they are %teamcity% placeholders for file directories.
What am I doing wrong to not be able to run specific tests?
For some reason my nunit-console is not recognizing the /run command or /fixture or --test=.
EDIT:
I upgraded to 3.5.0 and am still getting the same issues, I am not able to use --test.
C:\MyFolder\bin\NUnit.ConsoleRunner.3.5.0\tools\nunit3-console.exe "C:\MyFolder\Bin\UnitTesting.dll" --test="MyFolder.QuickTests.DaoTests.ProductDaoTests.ProductBasicTest"
That is the new location, and getting the same issue.
When I do --where for MyFolder it crashes Powershell but doesn't actually run anything.
When I do --explore it does the same as --where for MyFolder and does nothing for MyFolder.QuickTests .
EDIT EDIT:
Thanks to Rob I found the docs here and looked at the --where function with --where "name=ProductBasicTest" which will run all the files in that Test-Suite!
So thanks to Rob one of the issues that I was running into, is it was not recognizing my namespace correctly with QuickTests. So whenever I ran the function it was not running correctly.
To fix this you can go to the Test xml file output and see what names it was running tests under.
To run these individually you can run it by the name with the command:
"nunit3-console.exe C:\PathToDll.dll --where "name = NameOfTest"
I am currently trying to setup monitoring of windows scheduled tasks in Zabbix. It seemed easy enough to just monitor the Microsoft-Windows-TaskScheduler/Operational event log filtered by 201 events and regexing on the return code, but when I started simulating errors to test the monitoring, nothing happened.
It turns out that all our windows 2012 servers always log "return code 0" in the event log, even though it actually, sort of, displays it correctly in the Task Scheduler list view. When I say "sort of", it's because the "Last Run Result" actually displays 0x80070001 if the exit code of the program run by the scheduled task is 1.
I have spend a lot of time tweaking the settings, like user account, Run only when user is logged on, Run whether user is logged on or not, setting path on the action, Run with highest privileges, Configure for Vista/7/2012, etc. Nothing helped.
Finally I did some testing on my local machine, Windows 7, and a 2008R2 server, both of which just worked as expected.
The specific task I was testing ran a PowerShell script, using -Command so that it properly propagates the exit, but to rule out any PS issues I also tested with a batch file containing "exit 1" and finally with a small C# console program, that just returns whatever you supply on the command line.
PS, batch and console program all work fine on 7 and 2008, but they all fail in the same manner on 2012.
I've google this to death, but keep coming up short. Apparently 0x80070005 and other similar error codes are have some meaning, but that's not what happens in my case. In my case it seems that my exit code is bitwise or'ed with 0x80070000.
I should note that in all the cases, even 2012, the program started by the task, actually executes and run to the end, it's just the exit code which is handled weirdly.
Following is the output from the test runs:
From Powershell (my shell writes :( if $LASTEXITCODE > 0 ):
54 :( .\ExitCodeTest.exe 1
55 :( $LASTEXITCODE
1
56 :) .\ExitCodeTest.exe 10
57 :( $LASTEXITCODE
10
Windows Server 2008 R2 Standard:
Last Run Result (from list view): 0xA
Event 201 from event log Microsoft-Windows-TaskScheduler/Operational:
Task Scheduler successfully completed task "\ErrorTest" ,
instance "{b67a26cf-7fd8-461a-93d9-a5e48e72e558}" ,
action "D:\Tasks\ExitCodeTest.exe" with return code 10.
Windows Server 2012 Datacenter (notice that the return code in the event log is 0):
Last Run Result (from list view): 0x8007000A
Event 201 from event log Microsoft-Windows-TaskScheduler/Operational:
Task Scheduler successfully completed task "\error test" ,
instance "{2bde46b8-2858-4772-a7ec-d66b29d893a6}" ,
action "D:\Tasks\ExitCodeTest.exe" with return code 0.
Source for ExitCodeTest.exe:
static void Main( string[] args )
{
int exitCode = 0;
if ( args.Length > 0 )
{
exitCode = Convert.ToInt32( args[0] );
}
Environment.Exit( exitCode );
}
Please help, I am at my wits end.
Thanks,
John
(this is NOT an answer, but StackOverflow is refusing to let me add comments - when I click 'add comment', browser scrolls to top of page :-/)
You may be misinterpreting the Last Run Result column. According to Wikipedia (http://en.wikipedia.org/wiki/Windows_Task_Scheduler), LRR values of 0, 1 and 10 are common. Ignore the 0x8007 prefix - this just indicates a WIN32 error code transformed into an HRESULT (http://msdn.microsoft.com/en-us/library/gg567305.aspx).
Try running the test and forcing an exit code of something other than 1 or 10 to see if this influences LRR.
This does not explain of course why action return code is 0 in 2012. Error code 10 is defined as 'environment is incorrect'. Could it be that 2012 server does not want to run 32bit executable?
One other suggestion (and I'm a little out of my depth); according to (http://msdn.microsoft.com/en-us/library/system.environment.exit(v=vs.110).aspx): "Exit requires the caller to have permission to call unmanaged code. The return statement does not.". Might be worth re-compiling ExitCodeTest as follows:
static int Main(string[] args)
{
int exitCode = 0;
if ( args.Length > 0 )
{
exitCode = Convert.ToInt32( args[0] );
}
return exitCode;
}
I'm seeing a similar issue on Server 2012 with a batch file that looks like it succeeds, shows a return value of 0 in event log, but a Last Run Result of 0x80070001.
I see MSFT has a hotfix available for Server 2012 which might address this issue:
http://support.microsoft.com/kb/3003689
I had this problem and fixed it this way.
Instead of calling a batch file move the commands into the actions section of the scheduled task.
I realize this may not work for you as some batch files are long.
I suspect it has to do with circumventing security on a scheduled task -- if you can change the batch file then you could get a scheduled task to run as the identity without windows being the wiser.
I a have a few JUnit-Tests that makes use of the current Display to instantiate a few controls (TreeViewer for instance). Locally that works fine, no problem. When I commit these tests and jenkins runs the test I get a failed test for each test that makes use of Display.
My unit test uses the display variable in this manner:
#Test
public void testUtils() {
Display display = Display.getCurrent();
Shell shell = new Shell(display, SWT.NONE);
// org.eclipse.swt.widgets.Composite composite = new
// org.eclipse.swt.widgets.Composite(
// shell, SWT.NONE);
TreeViewer viewer = new TreeViewer(shell, SWT.MULTI | SWT.H_SCROLL
| SWT.V_SCROLL);
The error log jenkins generates is:
Time elapsed: 0.13 sec <<< ERROR!
org.eclipse.swt.SWTError: No more handles [gtk_init_check() failed]
at org.eclipse.swt.SWT.error(SWT.java:4109)
at org.eclipse.swt.widgets.Display.createDisplay(Display.java:902)
at org.eclipse.swt.widgets.Display.create(Display.java:890)
at org.eclipse.swt.graphics.Device.<init>(Device.java:154)
at org.eclipse.swt.widgets.Display.<init>(Display.java:499)
at org.eclipse.swt.widgets.Display.<init>(Display.java:490)
at org.eclipse.swt.widgets.Display.getDefault(Display.java:1693)
at org.eclipse.swt.widgets.Shell.<init>(Shell.java:260)
at org.eclipse.swt.widgets.Shell.<init>(Shell.java:253)
at
Is there any thing wrong with the way I am using Display in my tests? It works when executed on my local machine
The way you use Display looks OK to me. The error is likely related to the fact that your server is not running Gnome, hence SWT can't create a Display when you ask it to.
UPDATE
I just found a recent blog post, which explains what you need to do to run SWT UI tests on a headless server in more detail. Although the steps provided are meant for Hudson, they should be applicable to Jenkins as well.
It should all boil down to these two steps:
Check Run Xvnc during build (and don’t bother to check take screenshot, it doesn’t work)
Add an Execute shell build action before launching your tests with metacity –replace –sm-disable &
See the linked blog post for screenshots and more details.
You can try following two things,
execute command "xhost" or "xhost+" from your terminnal.
execute command "xhost" or "xhost+" from, jenkins terminal.