Nunit-console runner not running any tests - nunit

So I am trying to run a powershell script that is triggered by TeamCity to run specific unit tests based on the names of the files that were changed on each github commit.
Here is how I am running it from the command line:
C:\MyFolder\bin\NUnit.ConsoleRunner.3.4.1\tools\nunit3-console.exe "C:\MyFolder\Bin\UnitTesting.dll" --test="MyFolder.QuickTests.DaoTests.ProductDaoTests.ProductBasicTest"
But I keep getting this, it runs it just never runs any tests:
NUnit Console Runner 3.4.1
Copyright (C) 2016 Charlie Poole
Runtime Environment
OS Version: Microsoft Windows NT 10.0.14393.0
CLR Version: 4.0.30319.42000
Test Files
MyFolder\Bin\UnitTesting.dll
Test Filters
Test: MyFolder.QuickTests.DaoTests.ProductDaoTests.ProductBasicTest
Run Settings
WorkDirectory: C:\Users\Me
ImageRuntimeVersion: 4.0.30319
ImageTargetFrameworkName: .NETFramework,Version=v4.0
ImageRequiresX86: False
ImageRequiresDefaultAppDomainAssemblyResolver: False
NumberOfTestWorkers: 2
Test Run Summary
Overall result: Passed
Test Count: 0, Passed: 0, Failed: 0, Inconclusive: 0, Skipped: 0
Start time: 2016-10-17 20:28:43Z
End time: 2016-10-17 20:28:43Z
Duration: 0.303 seconds
Results (nunit3) saved as TestResult.xml
Now when I run it without the --test command like this:
C:\MyFolder\bin\NUnit.ConsoleRunner.3.4.1\tools\nunit3-console.exe "C:\MyFolder\Bin\UnitTesting.dll"
It runs every unit-test that we have, but I don't want to run them all, I want to run specific quick ones, and only run the large ones when we go to staging/production servers so our developers don't have to wait 15 to 20 minutes every time they commit something.
Some additional info:
-My namespace that I am using for this is
MyFolder.QuickTests.DaoTests.ProductDaoTests
The Class I am calling is:
ProductBasicTest
Some of the names like the folder directories were changed because they are %teamcity% placeholders for file directories.
What am I doing wrong to not be able to run specific tests?
For some reason my nunit-console is not recognizing the /run command or /fixture or --test=.
EDIT:
I upgraded to 3.5.0 and am still getting the same issues, I am not able to use --test.
C:\MyFolder\bin\NUnit.ConsoleRunner.3.5.0\tools\nunit3-console.exe "C:\MyFolder\Bin\UnitTesting.dll" --test="MyFolder.QuickTests.DaoTests.ProductDaoTests.ProductBasicTest"
That is the new location, and getting the same issue.
When I do --where for MyFolder it crashes Powershell but doesn't actually run anything.
When I do --explore it does the same as --where for MyFolder and does nothing for MyFolder.QuickTests .
EDIT EDIT:
Thanks to Rob I found the docs here and looked at the --where function with --where "name=ProductBasicTest" which will run all the files in that Test-Suite!

So thanks to Rob one of the issues that I was running into, is it was not recognizing my namespace correctly with QuickTests. So whenever I ran the function it was not running correctly.
To fix this you can go to the Test xml file output and see what names it was running tests under.
To run these individually you can run it by the name with the command:
"nunit3-console.exe C:\PathToDll.dll --where "name = NameOfTest"

Related

Nunit won't discover working non explicit tests Visual studio 2022

I have 3 normal and 1 explicit test but when I run my test using the Test Explorer window I get this output under "Tests" in the Output window
========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running all tests in xyz.dll
NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Explicit run
ExplicitMethod(03/10/2022 08:00:00,03/10/2022 16:00:00): OneTimeSetUp:
NUnit Adapter 4.2.0.0: Test execution complete
========== Test run finished: 1 Tests (0 Passed, 0 Failed, 0 Skipped) run in 462 ms ==========
*Note I'm using NUnit Adapter 4.2.1 instead of 4.2.0 so that's already weird
And this is how the Test Explorer window looks
Test Explorer window
*Note the full blue test is the Explicit test that get's skipped like it should
This is a problem because it does seem to only discover tests which are explicit.
The tests I want to run are of course not explicit, here is an example
[Test]
[TestCaseSource(nameof(TestNameData))]
public async Task<float> TestName(DateTime start, DateTime end, List<CalculateHoursObj>? list = default)
{
if (list == null) list = new List<CalculateHoursObj>();
return await EmployeeService.CalculateOverTimeHours(start, end, list);
}
public static IEnumerable TestNameData
{
get
{
yield return new TestCaseData(TenthMarch8_2022, TenthMarch16_2022).Returns(8.0f);
}
}
It fails to discover and/or run this test.
But curiously if I break all my non explicit tests by making the data non-static like this
public IEnumerable TestNameData
{
get
{
yield return new TestCaseData(TenthMarch8_2022, TenthMarch16_2022).Returns(8.0f);
}
}
It of course breaks the test and when I run all tests it does actually discover all tests
========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running selected tests in xyz.dll
NUnit3TestExecutor discovered 2 of 4 NUnit test cases using Current Discovery mode, Non-Explicit run
NUnit Adapter 4.2.0.0: Test execution complete
========== Test run finished: 2 Tests (0 Passed, 2 Failed, 0 Skipped) run in 394 ms ==========
But even now it only runs 2 of the 3 broken non explicit tests and of course they all fail
I have looked up everything online for 1.5 hours and really can't find a solution.
Don't bother responding with "have you updated visual studio or the nugget packages"
This question was part of a bug the solution is to not use new DateTime() withing nunit tests or to have nunit test be discover with id's using the <UseParentFQNForParametrizedTests>true</UseParentFQNForParametrizedTests> <UseNUnitIdforTestCaseId>true</UseNUnitIdforTestCaseId> flags
This was a bug because it was not reported correctly in the output window and/or Error List window.
I've created an Issue on GitHub about this if anyone wants to read.
Now stop editing this post because it truly is answered and the Issue link which I included shows the valid, correct and readable answer by "OsirisTerje" whom explains it way better than I could.

How to aggregate test results to publish to testrail after executing pytest tests with xdist?

I'm running into a problem like this. I'm currently using pytest to run test cases, and reducing execution time using xdist to run tests in parallel and publishing tests results to TestRail. The issue is when using xdist, pytest-testrail plugin creates Test-Run for each xdist workers and then publishes test cases like Untested.
I tried this hook pytest_terminal_summary to prevent pytest_sessionfinish plugin hook from being call multiple times.
I expect only one test run is created, but still multiple test runs are created.
I ran in to the same problem, but found a kind of workaround with duct tape.
I found that all results are collecting properly in test run, if we run the tests with --tr-run-id key.
If you are using jenkins jobs to automate processes, you can do following:
1) create testrun using testrail API
2) get ID of this test run
3) run the tests with --tr-run-id=$TEST_RUN_ID
I used these docs:
http://docs.gurock.com/testrail-api2/bindings-python
http://docs.gurock.com/testrail-api2/reference-runs
from testrail import *
import sys
client = APIClient('URL')
client.user = 'login'
client.password = 'password'
result = client.send_post('add_run/1', {"name": sys.argv[1], "assignedto_id": 1}).get("id")
print(result)
then in jenkins shell
RUN_ID=`python3 testrail_run.py $BUILD_TAG`
and then
python3 -m pytest -n 3 --testrail --tr-run-id=$RUN_ID --tr-config=testrail.cfg ...

.net core 2.1 Test Result not displaying in TFS 2018

I have .net core 2.1 project build pipeline as shown below.
I have 2 simple unit test cases which executed successfully. “.NET Core test” command executes successfully and publishes thereport.
2018-09-06T22:05:41.8494077Z
2018-09-06T22:05:42.0138714Z Passed Login_Test
2018-09-06T22:05:42.0139277Z Passed Logoff_Test
2018-09-06T22:05:42.0495295Z Results File: P:\*******2018-09-06_17_05_41.trx
2018-09-06T22:05:42.0514262Z
2018-09-06T22:05:42.0516990Z Attachments:
2018-09-06T22:05:42.0517972Z P:\*****\TestResults\329a8629-7ec1-454b-97d8-2623d29bbd0e\XXXXX 2018-09-06 17_05_39.coverage
2018-09-06T22:05:42.0518130Z
2018-09-06T22:05:42.0518516Z Total tests: 2. Passed: 2. Failed: 0. Skipped: 0.
2018-09-06T22:05:42.0519312Z Test Run Successful.
2018-09-06T22:05:42.0528123Z Test execution time: 1.9003 Seconds
2018-09-06T22:05:42.1424722Z Publishing test results to test run '90'
2018-09-06T22:05:42.1424881Z Test results remaining: 2. Test run id: 90
2018-09-06T22:05:42.1592889Z ##[section]Async Command Start: Publish test results
2018-09-06T22:05:42.2384075Z Published Test Run : http://******/_TestManagement/Runs#runId=90&_a=runCharts
2018-09-06T22:05:42.2384433Z ##[section]Async Command End: Publish test results
2018-09-06T22:05:42.2385471Z ##[section]Finishing: VsTest - testAssemblies
But I don’t see any “Test Results” under dashboard.
I included "Visual Studio Test" too, but still dont see the result in the dashboard. What am I missing here?

How to get Resharpers InspectCode to recognize Plugins?

I am trying to run ReSharpers command line tool InspectCode.exe. It's running fine doing it's job with the expected output.
However after my earlier attempt to get plugins to work, this time with the new version it is supposed to be supported. There is a switch in the command line interface that allows to specify the extension you want to use.
/extensions (/x) – allows using ReSharper extensions that affect code analysis. To use an extension, specify its ID, which you can find by opening the extension package page in the ReSharper Gallery, and then the Package Statistics page. Multiple values are separated with the semicolon.
But I cannot get it to work properly. I cannot even provoke any reaction to the /x switch at all. No matter how or what I pass, I get no feedback from the executable and the output is identical. I don't even get an error message when passing obvious garbage.
I tried the following commandlines for the exact same result:
inspectcode.exe /o="rcli.xml" /swea /x="ReSharper.StyleCop" "my.sln"
inspectcode.exe /o="rcli.xml" /swea /x=ReSharper.StyleCop "my.sln"
inspectcode.exe /o="rcli.xml" /swea "my.sln"
inspectcode.exe /o="rcli.xml" /swea /x=ABCDEFG "my.sln"
Result
JetBrains Inspect Code 9.1.1
Running in 64-bit mode, .NET runtime 4.0.30319.18444 under Microsoft Windows NT
6.1.7601 Service Pack 1
Enabled solution-wide analysis according to Inspect Code command line Setting.
Analyzing files
[files]
Inspection report was written to rcli.xml
What am I doing wrong? How to get extensions to work?
I already tried the R# forums, but it took them more then 24h to approve my post and so far I'm not sure someone else even read it.
Unfortunately, the support for extensions was dropped in 9.0 due to the refactorings in the "ReSharper platform". I hope that JetBrains will bring it back soon.
See RSRP-436208.
This is a late answer that might help future readers (like myself). Currently inspectcode.exe will automatically look for and use any NuGet packages that are in the same folder as the executable (source).
Example for CleanCode extension:
if you have a R# instance on some machine and install the extension, it will be placed in C:\Users\{user}\AppData\Local\JetBrains\plugins\MO.CleanCode.5.6.15
copy MO.CleanCode.5.6.15.nupkg and paste it next to inspectcode.exe
when running inspectcode with verbosity = VERBOSE, the extension should appear in the Zones list:
$cmd = "..\JetBrains.ReSharper.CommandLineTools.2019.3.4\inspectcode.exe"
$outputFile = "..\Output\$($outputName).xml"
& $cmd -o="$outputFile" $sln --verbosity=VERBOSE
Zones: (52pcs)[CodeInspectionPageImplZone, DaemonEngineZone,
DaemonZone, IAmd64CpuArchitectureHostZone, IAspMvcZone,
IBatchToolEnvironmentZone, IClrImplementationHost Zone,
IClrPsiLanguageZone, ICodeEditingOptionsPageImplZone,
IConsoleEnvironmentZone, ICppProductZone, ICpuArchitectureHostZone,
IDocumentModelZone, IEnvironmentZone, IHostSolutionZone,
IInspectCodeConsoleEnvironmentZone, IInspectCodeEnvironmentZone,
IInspectCodeZone, ILanguageAspZone, ILanguageBuildScriptsZone,
ILanguageCppZone, I LanguageCSharpZone, ILanguageCssZone,
ILanguageHtmlZone, ILanguageIlZone, ILanguageJavaScriptZone,
ILanguageMsBuildZone, ILanguageNAntZone, ILanguageProtobufZone, ILa
nguageRazorZone, ILanguageRegExpZone, ILanguageResxZone,
ILanguageVBZone, ILanguageXamlZone, INetFrameworkHostZone, INuGetZone,
IOperatingSystemHostZone, IProjectMode lZone,
IPsiAssemblyFileLoaderImplZone, IPsiLanguageZone,
IPublicVisibilityZone, IRdFrameworkZone, IRiderModelZone,
ISinceClr2HostZone, ISinceClr4HostZone, ITextContro lsZone,
IToolsOptionsPageImplZone, IWebPsiLanguageZone, IWindowsNtHostZone,
PsiFeaturesImplZone, ReplaceableByIntelliJPlatformZone, SweaZone]
Packages: (23pcs)[JetBrains.ExternalAnnotations,
JetBrains.Platform.Core.Ide, JetBrains.Platform.Core.IisExpress,
JetBrains.Platform.Core.MsBuild, JetBrains.Platform. Core.Shell,
JetBrains.Platform.Core.Text, JetBrains.Platform.Interop.CommandLine,
JetBrains.Platform.Interop.dotMemoryUnit.Framework,
JetBrains.Platform.Interop.dotMe moryUnit.Interop.Console,
JetBrains.Platform.Interop.dotMemoryUnit.Interop.Ide,
JetBrains.Platform.RdProtocol, JetBrains.Psi.Features.Core,
JetBrains.Psi.Features.Cpp .Src.Core, JetBrains.Psi.Features.src,
JetBrains.Psi.Features.Tasks, JetBrains.Psi.Features.UnitTesting,
JetBrains.Psi.Features.Web.Core, JetBrains.ReSharperAutomatio
nTools.src.CleanupCode,
JetBrains.ReSharperAutomationTools.src.CommandLineCore,
JetBrains.ReSharperAutomationTools.src.CommandLineProducts,
JetBrains.ReSharperAutomat ionTools.src.DuplicatesFinder,
JetBrains.ReSharperAutomationTools.src.InspectCode, MO.CleanCode]

run rake ts:index from within a rake task

I am running Rails 2.3.5.
In my project I have in lib/tasks the following rake task (test_task.rake):
desc 'test_synchro_task'
task :test_synchro_task => :environment do
# A bunch of things that are properly executed (basically I am inserting
# in the database)...
# ...
# and once the above is done, I want the following to be executed,
# the sphinx index to be rebuild, but it is NOT :
system("cd /sites/project/app")
system("RAILS_ENV='staging' rake ts:index")
end
I trigger the execution of the task via a crontab containing the following entry:
13 14 * * * cd /sites/project/app && /sites/ruby/bin/rake RAILS_ENV=staging test_task
which id correctly called and executed except for 2 system lines in the task.
Please note that when I place those 2 system lines in a ruby test.rb file in my project script directory, and run it manually using the ruby command:
ruby test.rb
those 2 system commands are properly executed and the index is rebuilt correctly.
In my rake task I tried replacing those 2 system lines by:
%x["cd /sites/project/app"]
%x["RAILS_ENV='staging' rake ts:index"]
or by
#cmd="cd /sites/project/app; RAILS_ENV='staging' rake ts:index"
`#{#cmd}`
but the rake ts:index is still not executed.
Any idea why?
Many thanks.
Yves
Problem resolved:
1- In the ruby script, I found that the $?.exitstatus was 127, which is "command not found".
2- This hinted me to a PATH problem occurring in the context of cron.
3- Found that post: http://dewful.com/?p=157 titled "Ruby - Cron Not Working For Ruby Script".
4- Added the PATH in the crontab and everything works fine.
Yves