I have the following function which I want to test:
def filterFoo(FooColumnName: String, Foo: Seq[Any]): DataFrame = {
/* validate input parameters */
require(Option(FooColumnName).isDefined)
require(Option(Foo).isDefined)
require(df.columns.contains(FooColumnName))
df.filter(col(FooColumnName).isin(Foo:_*))
I have written the following test:
#Test(expected = classOf[IllegalArgumentException])
def testFilterFoorWrongColumnName(): Unit ={
val df = data.toDF(columns:_*)
df.filterFoo(FooColumnName = "foo", Foo = competitors)
}
If FooColumnName does not exists in Data frame it would throw IllegalArgumentException. I am getting this exception but it is failing the test because of this. I get this error when I run the test:
java.lang.IllegalArgumentException: requirement failed
The trick is to reify the exception into a value, which, thankfully, is something that is extremely idiomatic in Scala. scala.util.Try is in the standard library and exceptionally useful for this:
import scala.util.{ Failure, Success, Try }
#Test(expected = classOf[IllegalArgumentException])
def testFilterFoorWrongColumnName(): Unit ={
val df = data.toDF(columns:_*)
val attempt = Try { // note the capital T, this is not the keyword try...
df.filterFoo(FooColumnName = "foo", Foo = competitors)
}
// apologies, I don't know JUnit at all...
attempt match {
case _: Success =>
failTheTest // should have been a failure
case Failure(ex) => // ex is the thrown exception
// can check that the exception is an IllegalArgumentException or whatever
doChecksOnException
}
}
How it works is that a Try[T] is either a Success wrapping a T or a Failure wrapping a Throwable. Try { } wraps the code in a try block and catches the exception; if it's a NonFatal exception, that exception goes into the Failure. There are some other good reasons to use Try (it has some nicely compositional properties), but those are out of the scope of this answer.
Related
The Scala Try construct together with its flatMap does not work as I would expect or want it to. The TL;DR is that I want to do a series of operations that can fail in two ways: either by raising an exception, which should be promoted and caught higher up in the call stack, or by returning Failure, as the failure must logically be handled in different parts of the program.
I would expect something like this to do the trick:
def firstStepSucceeds(): Try[Int] = Try {
1
}
def secondStepThrows(input: Int) = {
throw new Exception("Exception thrown in second step")
}
// I expect this to propagate the exception thrown in secondStepThrows
firstStepSucceeds() flatMap (secondStepThrows _)
(Full Scastie with example)
However, in this case, the flatMap() call actually implicitly catches the uncaught exception thrown by secondStepThrows, which is not what I want (which is why I left out the Try block). Is there a way to get the same behaviour without the implicit exception-catching?
What happens in a Try should stay in a Try. Most Scala programmers would be very surprised if a function returning a Try also sometimes threw an exception.
The typical pattern if you want to handle exceptions in different places is to differentiate by the type of the exception. So
val partiallyRecoveredTry = originalTry.recover{
case _: SecondStepException => "second step had an exception"
}
// Further up the call stack
partiallyRecoveredTry.getOrElse("first step had an exception")
Try.flatMap() did not caught exceptions implicitely, it is the essence of Try.
When you use it, it is very explicit, and that's the goal.
I don't really understand what you want, but is something like that is possible for you ?
try {
val first = firstStepSucceeds()
val second = first.map(secondStepThrows).get
val third = secondStepFails(second)
// ...
}
catch {
case e: Exception => ???
}
I did some further experimentation, and what I ended up with was this reimplementation of Try as (the now right-biased and hence monadic) Either:
object CatchAll {
def apply[SomeType](block: => SomeType) = try { Right(block) }
catch { case e: Throwable => Left(e) }
}
def firstStepSucceeds() = CatchAll {
1
}
def firstStepFails() = CatchAll {
throw new Exception("First step failed")
}
def secondStepSucceeds(input: Int) = CatchAll {
input + 1
}
def secondStepFails(input: Int) = CatchAll {
throw new Exception("Second step failed in try block!")
}
def secondStepThrows(input: Int) = {
throw new Exception("Second step failed unexpectedly!")
}
firstStepSucceeds() flatMap (secondStepSucceeds _)
firstStepFails() flatMap (secondStepSucceeds _)
firstStepSucceeds() flatMap (secondStepFails _)
// This now throws an exception as expected
//firstStepSucceeds() flatMap (secondStepThrows _)
This method throws exception
#throws(classOf[QuestionNotFoundException])
def getQuestionFromQuestionID(questionKey: PracticeQuestionKeys) = {
logger.trace(s"getting question with keys ${questionKey}")
val practiceQuestionFuture: Future[Option[PracticeQuestion]] = findOne(questionKey)
for (questionOption <- practiceQuestionFuture) yield {
questionOption.fold(throw QuestionNotFoundException())(question => {
logger.trace("got question " + question)
question
})
}
}
I use it like follows
def function1 = {
...
val res = for{existingQuestion <- questionsRepository.getQuestionFromQuestionID(questionKey){..}
res.recover {
case exception => {
logger.trace(s"exception ${exception}")
Ok(Json.toJson(JsonResultError(messagesApi("error.answerAdditionFail")(langs.availables(0))+". "+exception.getMessage())))
}
...
}
I want to test that function1 handles the thrown exception. I have written the following test case
"function" should {
"should return error if the question id is not correct" in {
...
when(answerTestEnv.mockPracticeQuestionsRepository.getQuestionFromQuestionID(ArgumentMatchers.any[PracticeQuestionKeys])).thenThrow(
QuestionNotFoundException()
)
val response:Accumulator[ByteString,Result] = controller.function1(request)
//note that run is how accumulator gets the data and starts processing it. no parameter means no data
val resultOfAccumulatorRun:Future[Result] = response.run(ByteString(body))
val responseBody = contentAsJson(resultOfAccumulatorRun)(Timeout(Duration(5000,"millis")),TestEnvImplicits.mat)
println(s"received response ${responseBody}")
val result = (responseBody \ "result").get.as[String]
val additionalInfo = (responseBody \ "additional-info").get.as[String]
result mustBe "error"
additionalInfo mustBe components.messagesApi("error.answerAdditionFail")(components.langs.availables(0))+". "+QuestionNotFoundException().msg
}
}
But when I run my test case, the Exception gets thrown but it seems that res.recover doesn't handle it. I see this in the console.
Question not found
utilities.QuestionNotFoundException: Question not found
at controllers.AnswerController.$anonfun$newAnswer$1(AnswerController.scala:428)
Why doestres.recover` doesn't handle it.
the partial function passed to recover() gets invoked on a failed Future, but your Future isn't a failed Future, it is a Future that contains an exception as its 'successful' result type (actually it will be forced to be something more like Future[AnyRef], as it could be either an Exception or a PracticeQuestion).
You probably don't want to be using Option.fold() like that. Instead try:
practiceQuestionFuture.flatMap { qOpt =>
qOpt.map(q => Future(q))
.getOrElse(Future.failed(QuestionNotFoundException()))
}
(NB: I'm not near a compiler to try this out right now, so I may have made a syntactical mistake or two, but there should be enough there to get you going).
I want to test my method which returns an Either. This is how i do this:
#Test def `Test empty name is not valid`: Unit = {
val book = createBook()
val result = insertEntryToBook(book, "", "12345")
result match {
case Left(InvalidNameFormat) => true
case _ => fail()
}
}
Should i do some assert() call instead of fail() to make the test's fail message more explicit (e.g. for assertion side-to-side view)?
There is no reason to pattern match here. You can just assert the result equals the expected value wrapped in a Left().
assertEquals(Left(InvalidNameFormat), result)
In case your test fails, you get a precise error message.
ScalaTest can be used with JUnit, so consider mixing in EitherValues which should give more informative messages
import org.junit.Test
import org.scalatest.{EitherValues, Matchers}
class ExampleSuite extends Matchers with EitherValues {
#Test def matchEithers(): Unit = {
val result: Either[String, Int] = Right(42)
result.left.value should be ("Boom")
}
}
which gives
org.scalatest.exceptions.TestFailedException: The Either on which left.value was invoked was not defined as a Left.
at org.scalatest.EitherValues$LeftValuable.value(EitherValues.scala:124)
at example.ExampleSuite.matchEithers(ExampleSuite.scala:9)
...
I'm using scanamo to query a dynamodb table.
I want to create a healthcheck method that will return true as long as dynamo can be scanned (i don't care if there are records in the table, i just want to know that the table is there and can be scanned). And i want false to be returned if the table is not present (i fudge the table name as a test).
This is the scala code i have right now:
trait DynamoTestTrait extends AbstractDynamoConfig {
def test(): Future[List[Either[DynamoReadError, T]]] =
ScanamoAsync.exec(client)(table.consistently.limit(1).scan())
}
and it works when the table is present:
val r = reg.test()
.map(
_.headOption.forall(_.isRight)
)
val e = Await.result(r, scala.concurrent.duration.Duration(5, "seconds"))
but when the table name is incorrect i get an unexpected error:
Unexpected server error: 'Cannot do operations on a non-existent table (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException;
I would have thought that i was trapping that error in my Left.
How can i trap this error? does scala have a try/catch construct i could use?
I also tried this but the error still gets thrown and not my catch it seems:
trait DynamoTestTrait extends AbstractDynamoConfig {
def test(): Future[Boolean] =
try {
ScanamoAsync
.exec(client)(table.consistently.limit(1).scan())
.map(
_.headOption.forall(_.isRight)
)
} catch {
case e: ResourceNotFoundException => Future.successful(false)
}
}
this seems more scalathonic, but always returns true:
trait DynamoTestTrait extends AbstractDynamoConfig {
def test: Future[Boolean] = {
val result = Try(ScanamoAsync
.exec(client)(table.consistently.limit(1).scan())
.map(
_.headOption.forall(_.isRight)
))
result match{
case Success(v) => Future.successful(true)
case Failure(e) => Future.successful(false)
}
}
}
According to the Scala Language Specification (ยง6.19), "An enumerator sequence always starts with a generator". Why?
I sometimes find this restriction to be a hindrance when using for-comprehensions with monads, because it means you can't do things like this:
def getFooValue(): Future[Int] = {
for {
manager = Manager.getManager() // could throw an exception
foo <- manager.makeFoo() // method call returns a Future
value = foo.getValue()
} yield value
}
Indeed, scalac rejects this with the error message '<-' expected but '=' found.
If this was valid syntax in Scala, one advantage would be that any exception thrown by Manager.getManager() would be caught by the Future monad used within the for-comprehension, and would cause it to yield a failed Future, which is what I want. The workaround of moving the call to Manager.getManager() outside the for-comprehension doesn't have this advantage:
def getFooValue(): Future[Int] = {
val manager = Manager.getManager()
for {
foo <- manager.makeFoo()
value = foo.getValue()
} yield value
}
In this case, an exception thrown by foo.getValue() will yield a failed Future (which is what I want), but an exception thrown by Manager.getManager() will be thrown back to the caller of getFooValue() (which is not what I want). Other possible ways of handling the exception are more verbose.
I find this restriction especially puzzling because in Haskell's otherwise similar do notation, there is no requirement that a do block should begin with a statement containing <-. Can anyone explain this difference between Scala and Haskell?
Here's a complete working example showing how exceptions are caught by the Future monad in for-comprehensions:
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Try, Success, Failure}
class Foo(val value: Int) {
def getValue(crash: Boolean): Int = {
if (crash) {
throw new Exception("failed to get value")
} else {
value
}
}
}
class Manager {
def makeFoo(crash: Boolean): Future[Foo] = {
if (crash) {
throw new Exception("failed to make Foo")
} else {
Future(new Foo(10))
}
}
}
object Manager {
def getManager(crash: Boolean): Manager = {
if (crash) {
throw new Exception("failed to get manager")
} else {
new Manager()
}
}
}
object Main extends App {
def getFooValue(crashGetManager: Boolean,
crashMakeFoo: Boolean,
crashGetValue: Boolean): Future[Int] = {
for {
manager <- Future(Manager.getManager(crashGetManager))
foo <- manager.makeFoo(crashMakeFoo)
value = foo.getValue(crashGetValue)
} yield value
}
def waitForValue(future: Future[Int]): Unit = {
val result = Try(Await.result(future, Duration("10 seconds")))
result match {
case Success(value) => println(s"Got value: $value")
case Failure(e) => println(s"Got error: $e")
}
}
val future1 = getFooValue(false, false, false)
waitForValue(future1)
val future2 = getFooValue(true, false, false)
waitForValue(future2)
val future3 = getFooValue(false, true, false)
waitForValue(future3)
val future4 = getFooValue(false, false, true)
waitForValue(future4)
}
Here's the output:
Got value: 10
Got error: java.lang.Exception: failed to get manager
Got error: java.lang.Exception: failed to make Foo
Got error: java.lang.Exception: failed to get value
This is a trivial example, but I'm working on a project in which we have a lot of non-trivial code that depends on this behaviour. As far as I understand, this is one of the main advantages of using Future (or Try) as a monad. What I find strange is that I have to write
manager <- Future(Manager.getManager(crashGetManager))
instead of
manager = Manager.getManager(crashGetManager)
(Edited to reflect #RexKerr's point that the monad is doing the work of catching the exceptions.)
for comprehensions do not catch exceptions. Try does, and it has the appropriate methods to participate in for-comprehensions, so you can
for {
manager <- Try { Manager.getManager() }
...
}
But then it's expecting Try all the way down unless you manually or implicitly have a way to switch container types (e.g. something that converts Try to a List).
So I'm not sure your premises are right. Any assignment you made in a for-comprehension can just be made early.
(Also, there is no point doing an assignment inside a for comprehension just to yield that exact value. Just do the computation in the yield block.)
(Also, just to illustrate that multiple types can play a role in for comprehensions so there's not a super-obvious correct answer for how to wrap an early assignment in terms of later types:
// List and Option, via implicit conversion
for {i <- List(1,2,3); j <- Option(i).filter(_ <2)} yield j
// Custom compatible types with map/flatMap
// Use :paste in the REPL to define A and B together
class A[X] { def flatMap[Y](f: X => B[Y]): A[Y] = new A[Y] }
class B[X](x: X) { def map[Y](f: X => Y): B[Y] = new B(f(x)) }
for{ i <- (new A[Int]); j <- (new B(i)) } yield j.toString
Even if you take the first type you still have the problem of whether there is a unique "bind" (way to wrap) and whether to doubly-wrap things that are already the correct type. There could be rules for all these things, but for-comprehensions are already hard enough to learn, no?)
Haskell translates the equivalent of for { manager = Manager.getManager(); ... } to the equivalent of lazy val manager = Manager.getManager(); for { ... }. This seems to work:
scala> lazy val x: Int = throw new Exception("")
x: Int = <lazy>
scala> for { y <- Future(x + 1) } yield y
res8: scala.concurrent.Future[Int] = scala.concurrent.impl.Promise$DefaultPromise#fedb05d
scala> Try(Await.result(res1, Duration("10 seconds")))
res9: scala.util.Try[Int] = Failure(java.lang.Exception: )
I think the reason this can't be done is because for-loops are syntactic sugar for flatMap and map methods (except if you are using a condition in the for-loop, in that case it's desugared with the method withFilter). When you are storing in a immutable variable, you can't use these methods. That's the reason you would be ok using Try as pointed out by Rex Kerr. In that case, you should be able to use map and flatMap methods.