How can I execute just a single test case through the NUnit ConsoleRunner using the --test=NAMES argument.
If I right click on a test case in Visual Studio and 'Copy' this is the information regarding the test.
Test Name: MakePaymentToAccountViaPayPal
Test FullName: POMAuctivaTest.TestSuite.ExistingUserTestSuite("firefox").MakePaymentToAccountViaPayPal
Test Source: c:\git\POMAuctivaTest\POMAuctivaTest.TestSuite\ExistingUserTestSuite.cs : line 263
Test Outcome: Not Run
Test Duration: 0:00:00
Here are some of the different ways I tried to run the tests.
I have tried with both the Test Name and the Test FullName and both attempts don't actually run the test cases. I have a feeling the issue is with the string being passed as a parameter but I am not sure.
Have just tried this, looks like you just need to escape the quotation marks. So use:
--test=POMAuctivaTest.TestSuite.ExistingUserTestSuite(\"firefox\").MakePaymentToAccountViaPayPal
Related
I am trying to run performance test on scenario tagged as perf from the below feature file-
#tag1 #tag2 #tag3
**background:**
user login
#tag4 #perf
**scenario1:**
#tag4
**scenario2:**
Below is my .scala file setup-
class PerfTest extends Simulation {
val protocol = karateProtocol()
val getTags = scenario("Name goes here").exec(karateFeature("classpath:filepath"))
setUp(
getTags.inject(
atOnceUsers(1)
).protocols(protocol)
)
I have tried passing the tags from command line and as well as passing the tag as argument in exec method in scala setup.
Terminal command-
mvn clean test-compile gatling:test "-Dkarate.env={env}" "-Dkarate.options= --tags #perf"
.scala update:- I have also tried passing the tag as an argument in the karate execute.
val getTags = scenario("Name goes here").exec(karateFeature("classpath:filepath", "#perf"))
Both scenarios are being executed with either approach. Any pointers how i can force only the test with tag perf to run?
I wanted to share the finding here. I realized it is working fine when i am passing the tag info in .scala file.
My scenario with perf tag was a combination of GET and POST call as i needed some data from GET call to pass in POST call. That's why i was seeing both calls when running performance test.
I did not find any reference in karate gatling documentation for passing tags in terminal execution command. So i am assuming that might not be a valid case.
I am attempting to run my SpecFlow/xUnit tests on the command line, as described here:
http://gasparnagy.com/2016/02/running-specflow-scenarios-in-parallel-with-xunit-v2/
If I type this:
.\packages\xunit.runner.console.2.3.1\tools\net452\xunit.console.exe --help
One of the flags described is this:
-trait "name=value" : only run tests with matching name/value traits
: if specified more than once, acts as an OR operation
I have a SpecFlow scenario with the trait #justthisone which I would like to run on its own. The Visual Studio test explorer lists this as having the trait Category [justthisone] I have tried this:
.\packages\xunit.runner.console.2.3.1\tools\net452\xunit.console.exe .\MyProj.Tests\bin\Debug\MyProj.Tests.dll -trait "name=justthisone"
But I get this output:
=== TEST EXECUTION SUMMARY ===
Order.UserInterface.Tests.dll Total: 0
How should I write the -trait flag/option to tell xUnit which tests I want to run?
Turns out I just had to specify the correct file name:
.\packages\xunit.runner.console.2.3.1\tools\net452\xunit.console.exe .\MyProj.Tests\bin\Debug\MyProj.Tests.dll -trait "**Category**=justthisone"
As shown here:
https://github.com/techtalk/SpecFlow/issues/938
So I am trying to run a powershell script that is triggered by TeamCity to run specific unit tests based on the names of the files that were changed on each github commit.
Here is how I am running it from the command line:
C:\MyFolder\bin\NUnit.ConsoleRunner.3.4.1\tools\nunit3-console.exe "C:\MyFolder\Bin\UnitTesting.dll" --test="MyFolder.QuickTests.DaoTests.ProductDaoTests.ProductBasicTest"
But I keep getting this, it runs it just never runs any tests:
NUnit Console Runner 3.4.1
Copyright (C) 2016 Charlie Poole
Runtime Environment
OS Version: Microsoft Windows NT 10.0.14393.0
CLR Version: 4.0.30319.42000
Test Files
MyFolder\Bin\UnitTesting.dll
Test Filters
Test: MyFolder.QuickTests.DaoTests.ProductDaoTests.ProductBasicTest
Run Settings
WorkDirectory: C:\Users\Me
ImageRuntimeVersion: 4.0.30319
ImageTargetFrameworkName: .NETFramework,Version=v4.0
ImageRequiresX86: False
ImageRequiresDefaultAppDomainAssemblyResolver: False
NumberOfTestWorkers: 2
Test Run Summary
Overall result: Passed
Test Count: 0, Passed: 0, Failed: 0, Inconclusive: 0, Skipped: 0
Start time: 2016-10-17 20:28:43Z
End time: 2016-10-17 20:28:43Z
Duration: 0.303 seconds
Results (nunit3) saved as TestResult.xml
Now when I run it without the --test command like this:
C:\MyFolder\bin\NUnit.ConsoleRunner.3.4.1\tools\nunit3-console.exe "C:\MyFolder\Bin\UnitTesting.dll"
It runs every unit-test that we have, but I don't want to run them all, I want to run specific quick ones, and only run the large ones when we go to staging/production servers so our developers don't have to wait 15 to 20 minutes every time they commit something.
Some additional info:
-My namespace that I am using for this is
MyFolder.QuickTests.DaoTests.ProductDaoTests
The Class I am calling is:
ProductBasicTest
Some of the names like the folder directories were changed because they are %teamcity% placeholders for file directories.
What am I doing wrong to not be able to run specific tests?
For some reason my nunit-console is not recognizing the /run command or /fixture or --test=.
EDIT:
I upgraded to 3.5.0 and am still getting the same issues, I am not able to use --test.
C:\MyFolder\bin\NUnit.ConsoleRunner.3.5.0\tools\nunit3-console.exe "C:\MyFolder\Bin\UnitTesting.dll" --test="MyFolder.QuickTests.DaoTests.ProductDaoTests.ProductBasicTest"
That is the new location, and getting the same issue.
When I do --where for MyFolder it crashes Powershell but doesn't actually run anything.
When I do --explore it does the same as --where for MyFolder and does nothing for MyFolder.QuickTests .
EDIT EDIT:
Thanks to Rob I found the docs here and looked at the --where function with --where "name=ProductBasicTest" which will run all the files in that Test-Suite!
So thanks to Rob one of the issues that I was running into, is it was not recognizing my namespace correctly with QuickTests. So whenever I ran the function it was not running correctly.
To fix this you can go to the Test xml file output and see what names it was running tests under.
To run these individually you can run it by the name with the command:
"nunit3-console.exe C:\PathToDll.dll --where "name = NameOfTest"
I have a smoke suite having xx test Cases. I have applied the Meta filter on only smoke level test cases. But when I tried with *.story as argument for eclipse, its hangup the execution while excluding the test cases as per the Meta Filter.
ENVIRONMENT: variable:- STORY_META_FILTER
Value:- +smoke
Story file Structure:
Scenario : test_xyz
Meta : #smoke
Given TEST1
When TEST2
Then TEST3
Can Anyone know the correct way to implement it in Eclipse? What will be the argument for Eclipse?
Set environment Variable with Meta_Filter and having value as +smoke
and run the story file with *.story
I'm using CTest and want to pass command-line arguments to the underlying tests at runtime. I know there are ways to hard code command-line arguments into the CMake/CTest script, but I want to specify the command-line arguments at runtime and have those arguments passed through CTest to the underlying test.
Is this even possible?
I've figured out a way to do it (using the Fundamental theorem of software engineering). It's not as simple as I'd like, but here it is.
First, create a file ${CMAKE_SOURCE_DIR}/cmake/RunTests.cmake with the content
if(NOT DEFINED ENV{TESTS_ARGUMENTS})
set(ENV{TESTS_ARGUMENTS} "--default-arguments")
endif()
execute_process(COMMAND ${TEST_EXECUTABLE} $ENV{TESTS_ARGUMENTS} RESULT_VARIABLE result)
if(NOT "${result}" STREQUAL "0")
message(FATAL_ERROR "Test failed with return value '${result}'")
endif()
Then, when you add the test, use
add_test(
NAME MyTest
COMMAND ${CMAKE_COMMAND} -DTEST_EXECUTABLE=$<TARGET_FILE:MyTest> -P ${CMAKE_SOURCE_DIR}/cmake/RunTests.cmake
)
Finally, you can run the test with custom arguments using
cmake -E env TESTS_ARGUMENTS="--custom-arguments" ctest
Note that if you use bash, you can simplify this to
TESTS_ARGUMENTS="--custom-arguments" ctest
There are some problems with this approach, e.g. it ignores the WILL_FAIL property of the tests. Of course I wish it could be as simple as calling ctest -- --custom-arguments, but, as the Stones said, You can't always get what you want.
I'm not sure I fully understand what you want, but I still can give you a way to pass arguments to tests in CTest, at runtime.
I'll give you an example, with CTK (the Common Toolkit, https://github.com/commontk/CTK):
In the build dir (ex: CTK-build/CTK-build, it's a superbuild), if I run: ('-V' for Verbose, and '-N' for View Mode only)
ctest -R ctkVTKDataSetArrayComboBoxTest1 -V -N
I get:
UpdateCTestConfiguration from : /CTK-build/CTK-build/DartConfiguration.tcl
Parse Config file:/CTK-build/CTK-build/DartConfiguration.tcl
Add coverage exclude regular expressions.
Add coverage exclude: /CMakeFiles/CMakeTmp/
Add coverage exclude: .*/moc_.*
Add coverage exclude: .*/ui_.*
Add coverage exclude: .*/Testing/.*
Add coverage exclude: .*/CMakeExternals/.*
Add coverage exclude: ./ctkPixmapIconEngine.*
Add coverage exclude: ./ctkIconEngine.*
UpdateCTestConfiguration from :/CTK-build/CTK-build/DartConfiguration.tcl
Parse Config file:/CTK-build/CTK-build/DartConfiguration.tcl
Test project /CTK-build/CTK-build
Constructing a list of tests
Done constructing a list of tests
178: Test command: /CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1"
Labels: CTKVisualizationVTKWidgets
Test #178: ctkVTKDataSetArrayComboBoxTest1
Total Tests: 1
You can copy-paste the "Test command" in your terminal:
/CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1"
And add the arguments, for example "-I" for interactive testing:
/CTK-build/CTK-build/bin/CTKVisualizationVTKWidgetsCppTests "ctkVTKDataSetArrayComboBoxTest1" "-I"
Tell me if it helps.
matthieu's answer gave me the clue to get it to work for me.
For my code I did the following:
Type the command ctest -V -R TestMembraneCellCrypt -N to get the output:
...
488: Test command: path/to/ctest/executable/TestMembraneCellCrypt
Labels: Continuous_project_ChasteMembrane
Test #488: TestMembraneCellCrypt
...
Then I copied the Test command and provided the arguments there:
path/to/ctest/executable/TestMembraneCellCrypt -e 2 -em 5 -ct 10
I'll note that the package I'm using (Chaste), is pretty large so there might be things going on that I don't know about.