I have a set of input cases stored in a file.
I would like each case to be a specific scalatest "test", i.e., reported in the console as an individual test and failed individually.
Unfortunately, experimentation and Google suggest that this capability might not be present?
E.g., this seems to be the common case (eliding for simplicity)
class MyTestingGoop extends FunSuite {
val input : Seq[SpecificTestCase] = ...
test("input data test") {
forAll(input) { case => ... }
}
//...
}
Ideally, each case presents as a separate test. How can this be done with ScalaTest?
You can do this:
class MyTestingGoop extends FunSuite {
val input : Seq[SpecificTestCase] = ...
forAll(input) {
test("testing input" + input) {
// do something with the test
}
}
}
The only limit is that input has a unique toString.
Basically calling test in Funsuite registers the test and later runs it so as long as your test creation is done as part of the class construction and each test has a unique string, you should be fine.
Related
I have more of a philosophical confusion in-regards to the usefulness of methods like 'beforeAll' in scalaTest.
I have been looking for an answer why the need to even have constructs like beforeAll? I do understand that there is a reason why this design decision was taken but not able to think it through. Can anyone help?
e.g.
Suggested way as per tutorials online,
class TestExample extends FunSuite with BeforeAndAfterAll {
private var _tempDir: File = _
protected def tempDir: File = _tempDir
override def beforeAll(): Unit = {
super.beforeAll()
_tempDir = Utils.createTempDir(namePrefix = this.getClass.getName)
}
test("...") {
// using the variable in the function
}
}
vs
class TestExample extends FunSuite with BeforeAndAfterAll {
private val tempDir: File = Utils.createTempDir(namePrefix =
this.getClass.getName)
}
test("...") {
// Use the initialized variable here.
}
If you have cleanup to do in afterAll, I think it is symmetric to do setup in beforeAll. Also if you need to do some side effect that doesn't involve initializing instance variables, that can go in beforeAll. In the example you gave, though, where you don't have any cleanup to do in afterAll and all you're doing before all tests is initializing instance variables, I'd do with plain old initialization.
One other difference between val initializers and beforeAll is val initializers happen when the class is instantiated, whereas beforeAll happens later, when the instance is executed. If you want to delay the initialization until the class is run, you can use lazy vals.
One point worth noting is that some runners (such as the ScalaTest ant task, and Intellij IDEA), will instantiate all tests instances before running any tests. If your setup code happens to interact with any global variables or external state, then you probably want to defer those interactions until the test is run.
As a simple (contrived) example, suppose your code under test includes
Object Singleton {
var foo = ""
}
and you have two test classes:
class Test1 extends FunSuite {
Singleton.foo = "test1"
test("...") {
Singleton.foo should be("test1")
}
}
class Test1 extends FunSuite {
Singleton.foo = "test2"
test("...") {
Singleton.foo should be("test2")
}
}
If both classes are instantiated before any tests are run, then at least one of your two tests will fail. Conversely, if you defer your initialize work until beforeAll, you'll not see the same interference between tests.
I have a set of tests all of which require same code before each test, for this purpose I happily use BeforeAndAfterEach. I need to perform the set of tests a few times with different settings of the fixtures. To perform the tests several times I use test sharing with "should behave like" from FlatSpecLike. However, I don't see an elegant way to combine "should behave like" with BeforeAndAfterEach.
Currently I use the Tag-based method presented below. Quite ugly.
Question: is there a more elegant way to perform a set of tests under different settings that are used before each test in the set.
class TestSpec() extends FlatSpecLike with Matchers with BeforeAndAfterEachTestData {
// have to invent a tag with a name for each object of Settings
val TagNamePrefix = "MySettingsTag"
case object NormalTag extends Tag(TagNamePrefix+"Normal")
case object SpecialTag extends Tag(TagNamePrefix+"Special")
// and put them into a map by name
var settingsMap: String => Settings = Map (
NormalTag.name -> Settings(0, 0, someGlobalNormalObject),
SpecialTag.name -> Settings(10, 10, someGlobalSpecialObject)
)
override def beforeEach(testData: TestData) {
val s: Settings = retrieveSettings(testData)
// some code here depending on the settings
super.beforeEach(testData)
}
/** the ugly way to retrieve settings in beforeEach */
private def retrieveSettings(testData: TestData): Settings = {
val tags = testData.tags.filter(_.startsWith(TagNamePrefix))
assert(tags.size == 1)
settingsMap(tags.last)
}
"A thing" should behave like testSet(NormalTag)
"A thing with special settings" should behave like testSet(SpecialTag)
def testSet(t: Tag) {
it should "test something" taggedAs t in {
// ...
}
it should "test something else" taggedAs t in {
// ...
}
}
}
I have a number of test data sets that run through the same ScalaTest unit tests. I'd love if each test data set was it's own set of named tests so if one data set fails one of the tests i know exactly which one it was rather than going to a single test and looking on what file it failed. I just can't seem to find a way for the test name to be generated at runtime. I've looked at property and table based testing and currently am using should behave like to share fixtures, but none of these seem to do what I want.
Have I not uncovered the right testing approach in ScalaTest or is this not possible?
You can write dynamic test cases with ScalaTest like Jonathan Chow wrote in his blog here: http://blog.echo.sh/2013/05/12/dynamically-creating-tests-with-scalatest.html
However, I always prefer the WordSpec testing definitions and this also works with dynamic test cases just like Jonathan mentions.
class MyTest extends WordSpec with Matchers {
"My test" should {
Seq(1,2,3) foreach { count =>
s"run test $count" in {
count should be(count)
}
}
}
}
When running this test it run 3 test cases
TestResults
MyTest
My test
run test 1
run test 2
run test 3
ps. You can even do multiple test cases in the same foreach function using the same count variable.
You could write a base test class, and extend it for each data set. Something like this:
case class Person(name: String, age: Int)
abstract class MyTestBase extends WordSpec with Matchers {
def name: String
def dataSet: List[Person]
s"Data set $name" should {
"have no zero-length names" in {
dataSet.foreach { s => s.name should not be empty }
}
}
}
class TheTest extends MyTestBase {
override lazy val name = "Family" // note lazy, otherwise initialization fails
override val dataSet = List(Person("Mom", 53), Person("Dad", 50))
}
Which produces output like this:
TheTests:
Data set Family
- should have no zero-length names
You can use scala string substitutions in your test names. Using behavior functions, something like this would work:
case class Person(name: String, age: Int)
trait PersonBehaviors { this: FlatSpec =>
// or add data set name as a parameter to this function
def personBehavior(person: => Person): Unit = {
behavior of person.name
it should s"have non-negative age: ${person.age}" in {
assert(person.age >= 0)
}
}
}
class TheTest extends FlatSpec with PersonBehaviors {
val person = Person("John", 32)
personBehavior(person)
}
This produces output like this:
TheTest:
John
- should have non-negative age: 32
What about using ScalaTest's clue mechanism so that any test failures can report as a clue which data set was being used?
You can use the withClue construct provided by Assertions,
which is extended by every style trait in ScalaTest, to add
extra information to reports of failed or canceled tests.
See also the documentation on AppendedClues
I'm working with Scala project, using Specs2 for testing.
How could I know the test name i'm currently within it (form the test context)?
This is the code structure of my tests:
class TestsE2E{
trait Context extends E2EScope {
...
}
"Test" should {
"do stuff" in new Context {
...
}
}
}
If you just use the test name to log it, you can create a custom ExamplesFactory to intercept the name:
import org.specs2.mutable.{FragmentsBuilder, ExamplesFactory}
trait TestNameLogger extends ExamplesFactory { this: FragmentsBuilder =>
override implicit def exampleFactory: ExampleFactory = new TestNameLoggerMutableExampleFactory
class TestNameLoggerMutableExampleFactory extends MutableExampleFactory {
override def newExample(e: Example): Example = {
println(e.desc.raw)
super.newExample(e)
}
}
}
This solution is a bit involved and the API for doing this will change with the upcoming specs2 3.0. Another alternative is to create your own Notifier.
I have one superclass with WordSpecLike and several subclasses for each backend.
abstract class A extends TestKit(_system) with WordSpecLike {
"backend" must {
"do something useful" in {
//test useful stuff
}
"print something pretty" in {
//test pretty print
}
}
}
class B extends A {
}
class C extends A {
}
C and B tests different backends but the thing is I need to turn on/off each backend-test separately for integration tests (using exclude groups).
Obviously, I can't use taggedAs.
Using a separate traits like in a bellow example didn't work:
trait backendB { this: Tag =>
override val name = "backendB"
}
class B extends A with backendB {
//...
}
}
Pure java annotation didn't work ether:
#TagAnnotation
#Retention(RetentionPolicy.RUNTIME)
#Target({ElementType.METHOD, ElementType.TYPE})
public #interface backendB {
}
So, the question is: Have I any other options to run each backend test under a special group or I have to copy-paste code?
Scala version 2.10
ScalaTest version 2.2.2
Java version 1.8.0_20
Can't use sbt instead of maven, can't bump scala version.
If I understand correctly, you want sometimes to exclude class B and run just class C. Other times exclude class C and run just class B. If that's correct, then normally you would annotate B and C with a respective tag annotation (make using Java). For example, doing this would cause all tests of B to be tagged as backendB:
#backendB
class B extends A {
}
What you didn't show is where the actor system is coming from. One thing about tags is that it only excludes tests, an instance of B will still be created. That means any state it holds onto will be initialized, and that might in your case mean an actor system. This might be what you meant by "didn't work." If so, my usual advice is to make that state lazily initialized. The instance will still be created, but because no tests run, he state will never be accessed, and since it is lazy, will never be initialized.
Since there are a lot of unknowns in this reply, you might do better on scalatest-users, where it is easier to go back and forth.
Ok, my final solution is not so elegant as it could be but it's work at least (still kinda more java-way solution).
Several small changes:
trait A-suite extends WordSpecLike {
val tagToExclude = Tag("here we can write anything")
def createTestBackend: Backend
"backend" must {
"do something useful" taggedAs(tagToExclude) in {
val backend = createTestBackend
//test useful stuff
}
"print something pretty" taggedAs(tagToExclude) in {
val backend = createTestBackend
//test pretty print
}
}
}
And test classes for different backends:
`
class backendA(override val tagToExclude = "backendA") extends A-suite {
override def createTestBackend: Backend = new backendA
}
class backendB(override val tagToExclude = "backendB") extends A-suite {
override def createTestBackend: Backend = new backendB
}
Now exclude groups work properly with maven.