Karate 1.0.1 Upgrade [duplicate] - upgrade

I have recently upgraded to version 1.0.0 from 0.9.6 and noticed that the generated karate-summary.html file, it doesn't display all the tested feature files in the JUnit 5 Runner unlike in 0.9.6.
What it displays instead was the last tested feature file only.
The below screenshots are from the provided SampleTest.java sample code (excluding other Tests for simplicity).
package karate;
import com.intuit.karate.junit5.Karate;
class SampleTest {
#Karate.Test
Karate testSample() {
return Karate.run("sample").relativeTo(getClass());
}
#Karate.Test
Karate testTags() {
return Karate.run("tags").relativeTo(getClass());
}
}
This is from Version 0.9.6.
And this one is from Version 1.0.0
However, when running the test below in 1.0.0, all the features are displayed in the summary correctly.
#Karate.Test
Karate testAll() {
return Karate.run().relativeTo(getClass());
}
Would anyone be kind to confirm if they are getting the similar result? It would be very much appreciated.

What it displays instead was the last tested feature file only.
This is because for each time you run a JUnit method, the reports directory is backed up by default. Look for other directories called target/karate-reports-<timestamp> and you may find your reports there. So maybe what is happening is that you have multiple JUnit tests that are all running, so you see this behavior. You may be able to over-ride this behavior by calling the method: .backupReportDir(false) on the builder. But I think it may not still work - because the JUnit runner has changed a little bit. It is designed to run one method at a time, when you are in local / dev-mode.
So the JUnit runner is just a convenience. You should use the Runner class / builder for CI execution, and when you want to run multiple tests and see them in one report: https://stackoverflow.com/a/65578167/143475
Here is an example: ExamplesTest.java
But in case there is a bug in the JUnit runner (which is quite possible) please follow the process and help the project developers replicate and then fix the issue to release as soon as possible.

Related

Flutter: Running integration tests breaks unit tests

Recently, I ran into an issue at work. To summarize it, I am currently working on writing test cases for an internal Flutter project. This had been working fine for a while, but after my summer vacation, it started breaking. It took me a while to figure out what roughly was going on, but I hen discovered that running integration tests generated a file named generated_main.dart under .dart_tool/flutter_build/. This was working just fine when using integration tests, however, it caused unit tests to fail due to not even loading.
I managed to identify the relevant line of code in the generated file as line eight: import 'file:///tmp/flutter_tools.WJSDQT/flutter_test_listener.YHXZLS/listener.dart' as entrypoint;. The capitalized strings of random characters are indeed random, apparently referring to a temporary path that only exists for the unit test's duration. Unit tests throw (even though not always reliably): .dart_tool/flutter_build/generated_main.dart:8:8: Error: Error when reading '/tmp/flutter_tools.WJSDQT/flutter_test_listener.YHXZLS/listener.dart': No such file or directory.
Surprisingly, there seem to be no issues when deleting the generated file. I have now added a print statement to the integration tests that tells whoever ran them that they need to delete the generated file. But this is obviously not a good solution long-term. Is there a way to disable this behavior so that testing does not become unnecessarily more complicated?
My environment is Ubuntu 20.04 LTS with Flutter 2.5.0-5.1.pre on channel beta, using Android Studio 2020.3.1. I am not currently able to test for this behavior on other platforms. The program needs to run on Linux. The test package version is 1.17.10, the newest one compatible with other dependencies.
Thank you in advance for your time.

Running Specflow scenarios in parallel

I am working with the following
Specflow - 2.2.1
nUnit - 3.9.0
And i was hoping its possible to allow 2 scenarios within the same feature to run in parallel. The reason for this is to speed up the test suite.
The aim would be to get a few tests (2-3) running in parallel within the same feature, rather than running sequentially.
Does anyone know how this is possible?
I have added this to my assembly -
[assembly: Parallelizable(ParallelScope.Children)]
but i am now seeing the following errors
An item with the same key has already been added.
Object reference not set to an instance of an object.
Scenario level parallelism is not supported by SpecFlow yet.
You can read in their docs they ONLY allow you to use:
[assembly: Parallelizable(ParallelScope.Fixtures)]
There is an open ticket that describes all the issues why this is not supported yet and ways to solve it - https://github.com/SpecFlowOSS/SpecFlow/issues/1535
Please note that even though that NUnit added support for Instance-per-test-case feature in the new version(3.13) it doesn't solve the problem (I tried it with LifeCycle.InstancePerTestCase and still got some errors).
There is an open issue in SpecFlow repo for this problem: https://github.com/SpecFlowOSS/SpecFlow/issues/894

NUnit Unit tests not showing in Test Explorer with Test Adapter installed

Want to improve this post? Provide detailed answers to this question, including citations and an explanation of why your answer is correct. Answers without enough detail may be edited or deleted.
I've installed NUnit Test Adapter for VS2012 + 2013. When I first installed the Adapter tests were showing up, but they stopped showing up for some reason today. After building, rebuilding, cleaning, restarting, nothing shows up in Test Explorer. Why would this be happening? I'm using VS2013 Ultimate.
If you're using a NUnit3+ version, there is a new Test Adapter available.
Go to "Tools -> Extensions and Updates -> Online" and search for "NUnit3 Test Adapter" and then install.
If your test project is set to target a 64bit platform, the tests won't show up in the NUnit Test Adapter.
My test assembly is 64-bit. From the menu bar at the top of visual studio 2012, I was able to select 'Test' -> 'Test Settings' -> 'Default Processor Architecture' -> 'X64'. After a 'Rebuild Solution' from the 'Build' menu, I was able to see all of my tests in test explorer. Hopefully this helps someone else in the future =D.
Check for NUnit versions mismatch. The currently available NUnit Test Adapter only works for NUnit version 2.6.4 and below.
To downgrade NUnit from version 3.x go to
Package Manager Console > update-package NUnit -version 2.6.4
http://jeremybytes.blogspot.co.ke/2015/11/review-of-unit-testing-makes-me-faster.html
In my situation the 'NUnit3 Test Adapter' has been disabled.
To re-enable it go to menu
Tools->Extensions and Updates...
On the left side select 'Installed'->'All'.
On the upper right corner search for 'nunit'.
If you have 'NUnit3 Test Adapter' installed, with the found item you can enable/disable it.
This answer seems pretty basic but wasn't completely obvious to me at first. If you (re)build the solution it only builds the projects that are configured to build in the Build -> Configuration Manager
This was my issue, I must have inadvertently changed a build configuration settings or something that caused my test projects not to build (when they previously were). So the Test Explorer window was looking at dlls that were out of date. It became clear to me this was the case after doing a Clean and seeing most of my tests disappear and not come back after a rebuild...further inspection of bin folder showed that these projects weren't being built at all.
I had a working setup (for NUnit2 and NUnit3 depending on the solution, and multiple versions of Visual Studio between 2012 and 2017), and it suddenly stopped working one day: no tests detected in any solution or version of VS.
In my case, it helped to delete %localappdata%\Temp\VisualStudioTestExplorerExtensions. After a restart of VS, everything worked as before.
If you are using VS 2017 and .net core ,as said here, you should add references to the test framework NUnit 3.6.1, to the test runner NUnit3TestAdapter 3.8.0-alpha1 and to the test SDK Microsoft.NET.Test.Sdk 15.0.0.
Tools
NuGet Package Manager
Manage NuGet Packages For Solution
Browse
NUnitTestAdapter.WithFramework
Ctrl+R,A to build/run tests
Using NUnitTestAdapter.WithFramework makes sure there are little/no inconsistencies across versions of NUnit and NUnit Adapter (i.e. "it just works")
Check whether you have stated
[TestFixureSetUp]
and
[Test]
in the test class
sample:
namespace ClassLibrary1
{
public class SimpleCalculator
{
public Calculator _calculator;
[TestFixtureSetUp]
public void initialize()
{
_calculator = new Calculator();
}
[Test]
public void DivideTest()
{
int a = 10;
int b = 2;
int expectedValue = a/b;
int actualValue = _calculator.Divide(a, b);
Assert.AreEqual(expectedValue, actualValue, "Functionality not working properly!");
}
}
}
I had to uninstall then re-install the xunit.runner.visualstudio nuget package. I tried this after trying all the above suggestions, so may be it was a mixture of things.
One other cause to this problem is if you open a project from a mapped drive - Visual Studio handles such projects properly, but apparently Nunit doesn't support them.
Copying the project to a physical fixed the issue.
I also found that when I uninstalled nunit v3.2.1, the nunit framework reference for v3.2.1 was still in my project in solution explorer.
Solution Explorer > ProjectName > References
If you right click it will show the version. Remove this
Then Right click on References > Add Reference.
Search for the version 2.x version and add then rebuild solution.
That worked for me!
I experienced the problem mentioned by op
My case was that I was handed an old project and the tests were actually part of the system under test. I assume they were using the external test runner.
this task chain resolved the issue for me
created a test project,
moved the test files there
added references so the test project would compile
added the Nunit and Nunit adapter NuGet packages to the test project
recompiled
i was able to successfully run the Nunit tests.
If you are using the TestCaseSource attribute, ensure the source exists and respects the documentation, otherwise your tests will not be discovered.
I had a similar issue where the tests where not being discovered. I had the correct version of NUnit, versions matched up between NUnit and adapter, and the tests where tagged correctly. I was running VS 2017 Enterprise not as an administrator. After starting VS as administrator the tests appeared.
I started a new solution with a test project in it, and compared it against my original, problem project. The original, for some reason, had an app.config in it. I excluded that file from the project and saw my tests reappear in the test explorer.
I had this problem too but the cause was different. I'm using VS2017 with F# 4.0.
Firstly, the console in Visual Studio does not give you enough details why the tests could not be found; it will just fail to the load the DLL with the tests. So use NUnit3console.exe on the command line as this gives you more details.
In my case, it was because the test adapter was looking for a newer version of the F# Core DLL (4.4.1.0) (F# 4.1) whereas I'm still using 4.4.0.0 (F# 4.0). So I just added this to the app.config of the test project:-
<dependentAssembly>
<assemblyIdentity name="FSharp.Core" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
<bindingRedirect oldVersion="0.0.0.0-65535.65535.65535.65535" newVersion="4.4.0.0" />
</dependentAssembly>
i.e. redirect to the earlier F# core.
I had the same problem, when suddenly any test didn't appeared on Test Explorer window.
I has the updated version of "NUnit3TestAdapter"
and after lots of searches and efforts,
I found that I need set the following values in project properties:
[In Solution Explorer window: right click on Project > Properties]
Under Build tab, set Platform=x64, and set Platform target=x86 or Any CPU
Build the project and all tests will be appear on Test Explorer window.
Important note:
I came to a solution after seeing the next msg in the output window:
"Test run will use DLL(s) built for framework Framework45 and platform X86. Following DLL(s) will not be part of run: AutomationTests.dll is built for Framework Framework45 and Platform X64."
I had some msbuild.exe processes that were hung. I don't know if that was my problem or not, but it took me a lot of trail and error with reinstalling various NUnit adaptors before I found the hung processes.
Just to add my $.02 here, I ran into a similar issue just yesterday, where 168 of my tests were missing. I tried most everything in this post - most especially making sure my version(s) of NUnit were the same - all to no avail. I then remembered that I had my tests divided into playlists; and these do not update automatically as you add new tests. So, when I deleted the playlists, BAM!, all of my tests were back once more.
I use NUnit 3 and tried most of the other solutions I saw here. None of those worked for me.
Even though already selected, reselecting the "Playlist: All Tests" option in test explorer showed all hidden tests in my case. I need to do this after every rebuild.
Make sure your test class is public. I often make this mistake then I just look at the code for 5 minutes what is going on.
[Test]
public void YourTest()
{
...
}
Tools
NuGet Package Manager
Manage NuGet Packages For Solution
Browse
NUnit3TestAdapter (NUnit 3 in my case ..)
After installation it should be possible to use the default TestExplorer included with the installation of Visual Studio.
If your test project is not x86, the tests will not be found until you tell the test runner the correct target platform.
Prior to Visual Studio 2022, this was the "Test/Test Settings/Default Processor Architecture" option.
In VS2022 you need to create a 'runsettings' file and set the test project to use it. In that file you can specify the Target Platform, like this:
<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
<!-- Configurations that affect the Test Framework -->
<RunConfiguration>
<TargetPlatform>x64</TargetPlatform>
</RunConfiguration>
</RunSettings>
You still also need to include the Nunit3TestAdapter and Microsoft.NET.Test.Sdk nugets (and make sure the project actually built, and check the various other more obscure mistakes mentioned in other answers)
Make sure your tests are properly marked with the Test attribute. If all of the tests are marked with only the Explicit attribute, the TestAdapter doesn't recognize the fixture.

Testing a Monotouch app with NUnit in MonoDevelop 2.4

I'm new to both Monotouch and Monodevelop. Trying to get started with NUnit and I'm having a lot of trouble -- none of the sketchy references I can find on line seem to match what I'm seeing in the UI (MonoDevelop 2.4 on Mac OS 10.6). I've tried:
Adding an "NUnit assembly test collection" project to my solution.
Adding an "Empty MonoTouch Project" project to my solution, and then adding the NUnit assemblies to it, and adding my main project as a reference.
Adding a C# "Empty Project" to my solution, and adding NUnit, MonoTouch, and my own project as references. This produces a build error complaining that "'[test project name].exe' does not contain a static 'Main' method suitable for an entry point."
(1) produces a strange sort of project to which I can only add assemblies -- no references and certainly no tests.
(2) and (3) behave pretty much identically:
First, a build error complaining that there's no static 'Main' method. I can fix this by changing the compile target to "Library" in the Build -> General project options.
Next, when I try to run the tests (from the Unit Testing tab), it says it's running them using the "Debug|iPhoneSimulator" configuration.
The Test Results panel shows this "running tests" message, and never any other output.
The counts stay at "Tests: 0 Failed: 0 Ignored: 0".
Clearly I'm doing something wrong here, but what am I supposed to be doing?
Just for grins, here's my test.
using System;
using NUnit.Framework;
namespace mynamespace
{
[TestFixture]
public class NavItemTest
{
[Test]
public void TestAll()
{
Assert.AreEqual(4, NavItem.all().Count);
}
}
}
In case you missed it, there is now a NUnitLite runner available for MonoTouch which is designed to work for UI agnostic code and executed on devices (or simulator).
See: .NET Unit test runner for iOS
Have written up a few details on what we have found to be best practice so far. You can find it here: http://ben.phegan.name/index.php/2011/02/28/monotouch-and-unit-testing. Would be happy to hear other ways of doing this.
Short answer:
Add NUnit project to normal solution
Add MonoTouch assemblies as references
Write tests, avoiding testing anything that uses monotouch.dll (design patterns to abstract this stuff).
Win!
I had the same problem:
This produces a build error complaining that "'[test project name].exe' does not contain a static 'Main' method suitable for an entry point."
Fixed by going into Project -> [Project name] Options -> Build General. Changed Compile Target to Library. I hadn't made a Main-method class yet, but will probably later; so then I'll change back.

java.lang.exception no runnable methods junit

I have a suite, inside which I have added the test class.
I am using surefire to run my JUnits.
My test class ends with test and the methods has #test annotations to it.
How can this problem be resolved?
Here are various suggestions for this incomplete question (for those unfortunate enough to be brought here by google looking for an answer to this common issue):
if using Junit4.x, just use annotations (#Test); don't create a test suite: see this question for details.
original question said "#Test" annotation is being used, which should prevent the error. But it can still happen if there are other errors that happen earlier, and junit can hide the original problem with this message. E.g., see if there are problems with Spring configuration (unset #Required attributes), misconfigured mock objects, etc.
to avoid other frequent issues that also may generate this error (such as running classes suffixed by "*Test" that do not have any #Test methods), try updating to the surefire plugin 2.7 (currently #2.8.1) and junit 4.7+ (currently #4.8.1) to avoid this issue (i'm using maven3, btw; perhaps do a "mvn clean" before "mvn test" to be safe)
long shot: upgrade to at least ant 1.7+ (currently 1.8+) to avoid junit 4 test suite issues
You are using the correct version of JUnit at least 4.X to be able to use annotations for that? (Maven?)