I'd like to limit the default number of iterations for Property Based Tests in Kotest, preferably in the code (instead of using the also existing system property kotest.proptest.default.iteration.count inside my gradle/maven project).
The Global Configuration | Kotest Property Testing page states that I can achieve this by setting PropertyTesting.defaultIterationCount to some value. What I don't get is where to place this assignment. The documentation for Project Wide Configuration in the Framework says I have to override AbstractProjectConfig, which sadly does not include any properties regarding PBT.
The following code works by invoking the assignemnt as a side effect:
import io.kotest.core.config.AbstractProjectConfig
import io.kotest.property.PropertyTesting
class TestConfig : AbstractProjectConfig() {
init {
PropertyTesting.defaultIterationCount = 123
}
}
is there a better way configuring PBT?
The global configuration for Property Testing in Kotest is not tied to Kotest framework itself, because you can use the prop testing modules from JUnit or anything.
So the way to make this work is to set PropertyTesting.defaultIterationCount in some kind of "before project" listener. Now if you're using Kotest framework, it's easy and you're almost there. Just put the assignment inside a beforeProject listener, and it'll be guaranteed to run before any tests.
class TestConfig : AbstractProjectConfig() {
override suspend fun beforeProject() {
PropertyTesting.defaultIterationCount = 123
}
}
Related
I use Autofac extensively. Recently I've gotten interested in tweaking the lifetime scopes when registering items for XUnit testing. Basically I want to register a number of standard components I use as "instance per test" rather than what I normally do for runtime (I've found a useful library on github that defines an instance-per-test lifetime).
One way to do this is to define two separate container builds, one for runtime and one for xunit testing. That would work but it gets increasingly expensive to maintain.
What I'd like to do (I think) is modify the registration pipeline dynamically depending upon the context -- runtime or xunit test -- in which it is being built. In pseudocode:
builder.RegisterType<SomeType>().AsImplementedInterfaces().SingleInstance();
...
void TweakPipeline(...)
{
if( Testing )
{
TypeBeingRegistered.InstancePerTest();
}
else
{
TypeBeingRegistered.SingleInstance();
}
}
Is this something Autofac middleware can do? If not is there another capability in the Autofac API which could address it? As always, links to examples would be appreciated.
This is an interesting question. I like that you started thinking about some of the new features in Autofac, very few do. So, kudos for the good question.
If you think about the middleware, yes, you can probably use it to muck with lifetime scope, but we didn't really make "change the lifetime scope on the fly" something easy to do and... I'll be honest, I'm not sure how you'd do it.
However, I think there are a couple of different options you have to make life easier. In the order in which I'd do them if it was me...
Option 1: Container Per Test
This is actually what I do for my tests. I don't share a container across multiple tests, I actually make building the container part of the test setup. For Xunit, that means I put it in the constructor of the test class.
Why? A couple reasons:
State is a problem. I don't want test ordering or state on singletons in the container to make my tests fragile.
I want to test what I deploy. I don't want something to test out OK only to find that it worked because of something I set up in the container special for testing. Obvious exceptions for mocks and things to make the tests actually unit tests.
If the problem is that the container takes too long to set up and is slowing the tests down, I'd probably troubleshoot that. I usually find the cause of this to be either that I'm assembly scanning and registering way, way too much (oops, forgot the Where statement to filter things down) or I've started trying to "multi-purpose" the container to start orchestrating my app startup logic by registering code to auto-execute on container build (which is easy to do... but don't forget the container isn't your app startup logic, so maybe separate that out).
Container per test really is the easiest, most isolated way to go and requires no special effort.
Option 2: Modules
Modules are a nice way to encapsulate sets of registrations and can be a good way to take parameters like this. In this case, I might do something like this for the module:
public class MyModule : Module
{
public bool Testing { get; set; }
protected override void Load(ContainerBuilder builder)
{
var toUpdate = new List<IRegistrationBuilder<object, ConcreteReflectionActivatorData, SingleRegistrationStyle>>();
toUpdate.Add(builder.RegisterType<SomeType>());
toUpdate.Add(builder.RegisterType<OtherType>());
foreach(var reg in toUpdate)
{
if(this.Testing)
{
reg.InstancePerTest();
}
else
{
reg.SingleInstance();
}
}
}
}
Then you could register it:
var module = new MyModule { Testing = true };
builder.RegisterModule(module);
That makes the list of registrations easier to tweak (foreach loop) and also keeps the "things that need changing based on testing" isolated to a module.
Granted, it could get a little complex in there if you have lambdas and all sorts of other registrations in there, but that's the gist.
Option 3: Builder Properties
The ContainerBuilder has a set of properties you can use while building stuff to help avoid having to deal with environment variables but also cart around arbitrary info you can use while setting up the container. You could write an extension method like this:
public static IRegistrationBuilder<TLimit, TActivatorData, TRegistrationStyle>
EnableTesting<TLimit, TActivatorData, TRegistrationStyle>(
this IRegistrationBuilder<TLimit, TActivatorData, TRegistrationStyle> registration,
ContainerBuilder builder)
{
if(builder.Properties.TryGetValue("testing", out var testing) && Convert.ToBoolean(testing))
{
registration.InstancePerTest();
}
return registration;
}
Then when you register things that need to be tweaked, you could do it like this:
var builder = new ContainerBuilder();
// Set this in your tests, not in production
// builder.Properties["testing"] = true;
builder.RegisterType<Handler>()
.SingleInstance()
.EnableTesting(builder);
var container = builder.Build();
You might be able to clean that up a bit, but again, that's the general idea.
You might ask why use the builder as the mechanism to transport properties if you have to pass it in anyway.
Fluent syntax: Due to the way registrations work, they're all extension methods on the registration, not on the builder. The registration is a self-contained thing that doesn't have a reference to the builder (you can create a registration object entirely without a builder).
Internal callbacks: The internals on how registration works basically boil down to having a list of Action executed where the registrations have all the variables set up in a closure. It's not a function where we can pass stuff in during build; it's self-contained. (That might be interesting to change, now I think of it, but that's another discussion!)
You can isolate it: You could put this into a module or anywhere else and you won't be adding any new dependencies or logic. The thing carting around the variable will be the builder itself, which is always present.
Like I said, you could potentially make this better based on your own needs.
Recommendation: Container Per Test
I'll wrap up by just again recommending container per test. It's so simple, it requires no extra work, there are no surprises, and it "just works."
I'm using NUnit 2.X library but want to use NUnit 3.X now. I have some problems about migration from 2.X to 3.X. First i have a setup fixture class. Here is the 2.X version;
using System;
using System.IO;
using System.Reflection;
using HalisEnerji.QuantSignal.Logging;
using NUnit.Framework;
namespace HalisEnerji.QuantSignal.Tests
{
[SetUpFixture]
public class Initialize
{
[SetUp]
public void SetLogHandler()
{
Log.LogHandler = new ConsoleLogHandler();
}
}
}
First problem is fixed via change "Setup" attribute with "OneTimeSetUp" attribute. Second problem fixed via add some codes for set test directory. Because i'm using Re-Sharper test engine. Here is final shape of setup fixture;
using System;
using System.IO;
using System.Reflection;
using HalisEnerji.QuantSignal.Logging;
using NUnit.Framework;
namespace HalisEnerji.QuantSignal.Tests
{
[SetUpFixture]
public class Initialize
{
[OneTimeSetUp]
public void SetLogHandler()
{
Log.LogHandler = new ConsoleLogHandler();
var assembly = Assembly.GetExecutingAssembly();
var localPath = new Uri(assembly.CodeBase).LocalPath;
var direcotyName = Path.GetDirectoryName(localPath);
if (direcotyName != null)
{
Environment.CurrentDirectory = direcotyName;
}
}
}
}
Well after solve setup fixture problem, my real problems begins with use TestCaseSource/TestCaseData. Here is sample 2.X version;
[Theory]
[TestCaseSource("CreateSymbolTestCaseData")]
public void CreateSymbol(string ticker, Symbol expected)
{
Assert.AreEqual(Symbol.Create(ticker), expected);
}
private TestCaseData[] CreateSymbolTestCaseData()
{
return new []
{
new TestCaseData("SPY", new Symbol(Security.GenerateEquity("SPY"), "SPY")),
new TestCaseData("EURUSD", new Symbol(Security.GenerateForex("EURUSD"), "EURUSD"))
};
}
2.X version creating exception and my tests are fail. Shortly exception telling that TestCaseData provider method must be static. Well, after mark method with static identifier test working correctly but this time my other test failing (before use static identifier it's not failing). Why my other test failing? Because it's reading a file from test directory and somehow test working before setup up fixture codes run and change test directory!
Before use static identifier first SetUpFixture codes run and then tests code run. After use static identifier order changing my test that read file from test directory (which is Re-Sharper's temporary directory and not contain necessary file) run first after that SetUpFixture codes run. Any idea how all my tests to be successful?
UPDATE:
Explain some units;
I have Initialize.cs (part of my test assembly) which is responsible setup CurrentDirectory.
I have Config.cs (part of my project infrastructure assembly) which is my project configuration file and it has public static readonly Setttings property which is reading configuration file from CurrentDirectory.
I have ConfigTests.cs (part of my test assembly) which is contain some test methods for read/write Settings property.
When i debug tests;
Before use any static TestCaseSource, they are working below order;
A. Initialize.cs => Setup method
B. Config.cs => static Settings property getter method
C. ConfigTests.cs => first test method
So initialize working first others working later all tests successfully passing from test.
After use static TestCaseSource for inside other test file lets say OrdersTests.cs (excluded from project for first scenario after that included again), somehow working order is changing like below;
A. Config.cs => static Settings property getter method
B. OrdersTests.cs => static TestCaseSource method (not test method)
C. Initialize.cs => Setup method
D. ConfigTests.cs => first test method
E. OrdersTests.cs => first test method
So, my ConfigTests.cs tests failing because Initialize.cs working after Config.cs. I hope with this update my problem is more clear.
Is this problem related NUnit or Resharper or V.Studio? I don't know and all i know is my successfully passing tests are failing now!
UPDATE 2:
Chris,
Yes you are right. I explore project in detail and i saw the problem is related that my project's some classes accessing to static Config class and it's static Settings property (before run test setup fixture method and even before static test case source method!). You talk about order of process of test methods; NUnit doing tests like your said, not like i said. But when i try to use your solution (set current directory before test case source) it's not working. Because of that i solve my problem in another way. I'm not happy but at least my test methods working now. Could you please tell me what are the technical reasons that run static test case methods before initialize/setup method? Is this because of NUnit or because of infrastructure of .Net Framework? I'm not fanatic about NUnit and/or TDD. I don't have deep knowledge about these concepts but it does not make sense to me: run any method before setup method.
Thanks for your interest.
Because it's reading a file from test directory and somehow test working before setup up fixture codes run and change test directory!
How are you reading this file? You should use TestContext.CurrentContext.TestDirectory to get the test directory in NUnit 3, rather than relying on the location of the current directory. See the Breaking Changes page for details.
Edit: I also see you've tagged this ReSharper 7.1. You should be aware that this version of resharper does not support NUnit 3 - the first version that did is ReSharper 10. Your tests will appear to run correctly - however you may experience weird side effects, and this may break in any future version of NUnit.
Response to update:
Take a look at NUnit 3's Breaking Changes page. There are two relevant breaking changes between NUnit 2 and 3.
TestCaseSource's must now be static.
The CurrentDirectory is no longer set to Environment.CurrentDirectory by default.
The first you've solved easily enough. The second, is what's now causing you issues.
NUnit 3 runs it's methods in this order:
Evalute TestCaseSource methods (OrdersTests.cs)
Run SetUpFixture (Initialize.cs)
Run Test (ConfigTests/OrdersTests)
I'm not sure what Config.cs is being called before your TestCaseSource method - are you sure of that order? Does anything in CreateSymbolTestCaseData() call anything in Config.cs You could try rewriting your TestCaseSource as such:
private TestCaseData[] CreateSymbolTestCaseData()
{
Environment.CurrentDirectory = "c:\RequiredDirectory";
return new []
{
new TestCaseData("SPY", new Symbol(Security.GenerateEquity("SPY"), "SPY")),
new TestCaseData("EURUSD", new Symbol(Security.GenerateForex("EURUSD"), "EURUSD"))
};
}
I am new to Scala, the PlayFramework, and Dependency Injection. I downloaded the sample scala play framework code. Can someone explain to me why we need to inject the Clock and the appLifeCycle? It is referenced above, so there is no need to inject it right? What is going on here? Why do we need to do this for Web Frameworks in general?
package services
import java.time.{Clock, Instant}
import javax.inject._
import play.api.Logger
import play.api.inject.ApplicationLifecycle
import scala.concurrent.Future
/**
* This class demonstrates how to run code when the
* application starts and stops. It starts a timer when the
* application starts. When the application stops it prints out how
* long the application was running for.
*
* This class is registered for Guice dependency injection in the
* [[Module]] class. We want the class to start when the application
* starts, so it is registered as an "eager singleton". See the code
* in the [[Module]] class to see how this happens.
*
* This class needs to run code when the server stops. It uses the
* application's [[ApplicationLifecycle]] to register a stop hook.
*/
#Singleton
class ApplicationTimer #Inject() (clock: Clock, appLifecycle: ApplicationLifecycle) {
// This code is called when the application starts.
private val start: Instant = clock.instant
Logger.info(s"ApplicationTimer demo: Starting application at $start.")
// When the application starts, register a stop hook with the
// ApplicationLifecycle object. The code inside the stop hook will
// be run when the application stops.
appLifecycle.addStopHook { () =>
val stop: Instant = clock.instant
val runningTime: Long = stop.getEpochSecond - start.getEpochSecond
Logger.info(s"ApplicationTimer demo: Stopping application at ${clock.instant} after ${runningTime}s.")
Future.successful(())
}
}
I assume you're using Lightbend's Play Scala Seed, which contains the code sample that you posted.
If you look at the documentation for java.time.Clock, you'll notice it says (emphasis mine):
Best practice for applications is to pass a Clock into any method that requires the current instant. A dependency injection framework is one way to achieve this. {.. code sample omitted .. } This approach allows an alternate clock, such as fixed or offset to be used during testing.
Ultimately the purpose of dependency injection is to allow you to define the interface that you want to inject into a class or an object, and configure the implementation of that interface in just one spot. The alternative is having to update a hardcoded dependency in multiple files, which can be messy and error prone. In the Play Scala Seed project, you'll notice a file called app/Module.scala. This file is one place where you can configure the bindings and they'll automatically be bound on application start. Notice the line where we bind the Clock implementation:
class Module extends AbstractModule {
override def configure() = {
// Use the system clock as the default implementation of Clock
bind(classOf[Clock]).toInstance(Clock.systemDefaultZone)
// Ask Guice to create an instance of ApplicationTimer when the
// application starts.
bind(classOf[ApplicationTimer]).asEagerSingleton()
// Set AtomicCounter as the implementation for Counter.
bind(classOf[Counter]).to(classOf[AtomicCounter])
}
}
This configuration is saying "when my application starts up, wherever I've injected a Clock should use Clock.systemDefaultZone." If you want ApplicationTimer to use a different clock during tests, you might do something like this:
import play.api.{Environment, Mode}
// Notice that we include the environment
class Module(environment: Environment) extends AbstractModule {
override def configure() = {
// Use the system clock as the default implementation of Clock
environment.mode match {
case Mode.Prod | Mode.Dev => {
bind(classOf[Clock]).toInstance(Clock.systemDefaultZone)
}
case Mode.Test => {
// Specifically use UTC clock in tests, because maybe it's easier to write assertions that way
// You could inject anything here and the ApplicationTimer would use it during tests
bind(classOf[Clock]).toInstance(Clock.systemUTC())
}
}
bind(classOf[ApplicationTimer]).asEagerSingleton()
bind(classOf[Counter]).to(classOf[AtomicCounter])
}
}
You can define modules in other places in the root package (i.e. no package com.example.whatever declaration at the top of the file) and they'll be automatically loaded, too. Otherwise, you'll need to add a binding in conf/application.conf that adds your module's name to the play.modules.enabled key. You can see a commented-out example of that in the Play Scala Seed. I also go into more depth in another answer that I've written here.
As for ApplicationLifecycle, this is a special module provided by Play that you could override with your own binding, but I'm not sure why you'd want to. It's there to give you access to hooks that execute before the application shuts down. Again, this component is injected because it would be simple to swap it out. Imagine having 100 modules that all depend on the application lifecycle. By default, it's bound to DefaultApplicationLifecycle (you can see it being bound here). If you had hardcoded DefaultApplicationLifecycle in all 100 modules, you'd have to update each one if you wanted to switch to a different lifecycle. With dependency injection, you just need to configure the binding to use a different lifecycle and the 100 modules will automatically use it.
The #RibbonClients annotation allows us to customise the Ribbon configuration per client. This process is described in the documentation at http://projects.spring.io/spring-cloud/spring-cloud.html#_customizing_the_ribbon_client
This is all fine. I tried to use the same approach to override the default configuration that should be applied to all my clients. So I defined the following configuration class and made sure it is considered by the component scan:
#Configuration
#RibbonClients(defaultConfiguration = MyDefaultRibbonConfig.class)
public class MyRibbonAutoConfiguration {
}
Unfortunately, it turns out that MyDefaultRibbonConfig is not taken into account when building the ribbon client's application context. A quick look and trace at RibbonClientConfigurationRegistrar let me think my #RibbonClients(default=...) annotation is unconditionally overridden by the one provided by org.springframework.cloud.netflix.ribbon.eureka.RibbonEurekaAutoConfiguration.
However, it works if the #RibbonClients annotation is set on a inner class instead of a top-level class:
#Configuration
public class MyRibbonAutoConfiguration {
#Configuration
#RibbonClients(defaultConfiguration = MyDefaultRibbonConfig.class)
static class SubConfig {
}
}
This is a side-effect the strategy followed by RibbonClientConfigurationRegistrar to give a name to the discovered configuration beans:
registerClientConfiguration(registry,
"default." + metadata.getEnclosingClassName(),
attrs.get("defaultConfiguration"));
The configuration for annotations declared on a top-level class are then registered with a bean name set to default.null.defaultConfiguration - so the next one overrides the previous (not sure the order is predictable though).
This behaviour looks strange to me. Did I miss something? Should I proceed differently?
This was an issue in SpringCloud-Netflix 1.0.1. See https://github.com/spring-cloud/spring-cloud-netflix/issues/374 for more information.
Is there some global state somewhere that I can access the currently-running test name?
I have tests which output files into a directory and read them back in. I'd like each test to create a directory to play in and then clean up after itself, and I don't want to push that name in (I'd have to make it unique, and then make sure each test keeps it unique; ew). I could use a GUID, but I'd like helper methods to be able to assume "this is the place where test files should be stored" without having to push that GUID around to them. Again, this augers for a global state somewhere.
Basically, I want a call like TestRunner.Current.CurrentTest.Name. Does such a thing exist?
(Assuming c#)
NUnit.Framework.TestContext.CurrentContext.Test.Name
or
NUnit.Framework.TestContext.CurrentContext.Test.FullName
or if you are really lazy and aren't driving your tests with TestCaseSource (thanks #aolszowka):
this.GetType().ToString()
I haven't upgraded to 2.5.7 yet myself, but it includes a TestContext class that seems to provide just what you're looking for: http://www.nunit.org/index.php?p=releaseNotes&r=2.5.7
Assuming one method per Test, in your NUnit code, you can use reflection to get the method name from the stacktrace.
If you write a helper method in your NUnit code called by other methods to do this file logging, you can use this syntax to check for the previous method:
string MethodName = new StackFrame(1).GetMethod().Name;
See the answers to question 44153, "Can you use reflection to find the name of the currently executing method?" for more details.
If we are using TestCaseSource tag then above solutions might not give correct answer
Try using TestContext.CurrentContext.Test.MethodName
Follow the below example
namespace NunitTests
{
public class Class1
{
static List<TestData> Data = new List<TestData>()
{
new TestData()
{
...
}
};
[Test]
[TestCaseSource(nameof(TenMBInstance))]
public void TestCase(TestData value)
{
TestContext.CurrentContext.Test.Name; //TestCase(NunitTests..TestData)
TestContext.CurrentContext.Test.MethodName; //TestCase
}
}
}