In the code below I have two Play for Scala functions, the first one catches an exception (this works fine) and in the second one I'm trying to rewrite it using Try.
I have two problems with Try: (1) when the number is negative the method doesn't fail, (2) I need to wrap all the responses with Future.successful.
How to fix this code?
class Test extends Controller {
def test1 = Action.async { request =>
val future = isPositive(-1)
future.map { result =>
Ok("OK, it's positive")
}
.recover {
case e => Ok(e.getMessage)
}
}
def isPositive(i: Int) = Future {
if (i<0)
throw new Exception ( "Number is negative" )
else
i
}
def test2 = Action.async { request =>
isPositiveWithTry(-1) match {
case Success(s) => Future.successful(Ok("OK, it's positive (Try succeded)"))
case Failure(f) => Future.successful(Ok(f.getMessage + " (Try failed)"))
}
}
def isPositiveWithTry(i: Int) : Try[Future[Int]] = Try {
isPositive(i)
}
}
In isPositive method exceptions are already caught by Future
def isPositive(i: Int) = Future {
if (i<0)
throw new Exception ( "Number is negative" )
else
i
}
In the below code
def isPositiveWithTry(i: Int) : Try[Future[Int]] = Try {
isPositive(i)
}
isPositive already catches all expections and Try is always a success.
So, when i is negative. Exception raised are handled by future and try gets a success value, resultant Try is a success. So you get successful Try with a failed Future inside.
Understanding using Grenade example
Assume throwing the exception as blowing up a Grenade.
Assume Future and Try as two layers. When grenade is blasted inside the double layer of Try[Future] i.e Try is around Future and grenade is gone off in the Future.
Now Future withstands the blast and becomes a failed value. As Future already took the damage caused by the damage of exception(grenade). Try will be a success but the value inside the Try is a failed future value. That failed future value is nothing but the exception raised.
Try is redundant when you are using Future
You can refactor your code to below one
Get rid of isPositiveWithTry. This method is not needed.
def isPositive(i: Int) = Future {
if (i<0)
throw new Exception ( "Number is negative" )
else
i
}
def test2 = Action.async { request =>
isPositive(-1).flatMap { _ =>
Future.successful(Ok("OK, it's positive (Try succeded)"))
}.recoverWith {
case f: Throwable => Future.successful(Ok(f.getMessage + " (Try failed)"))
}
}
Again test2 can also be written as
def test2 = Action.async { request =>
isPositive(-1).map { _ =>
Ok("OK, it's positive (Try succeded)")
}.recover {
case f: Throwable => Ok(f.getMessage + " (Try failed)")
}
}
In case isPositive returns Try
def isPositive(i: Int) = Try {
if (i<0)
throw new Exception ( "Number is negative" )
else
i
}
Now test2 will look like
def test2 = Action.async { request =>
isPositive(-1) match {
case Success(s) => Future.successful(Ok("OK, it's positive (Try succeded)"))
case Failure(f) => Future.successful(Ok(f.getMessage + " (Try failed)"))
}
}
Couple points:
1) You need to rewrite your isPositive such that it does not surround itself via a Future. The Future is catching the exception.
def isPositive(i: Int) ={
if (i<0)
throw new Exception ( "Number is negative" )
else
i
}
2) If you have a Try and you want a Future, then you can use the method on the companion object of Future, Future.fromTry. That will take a Try and turn it into the correct state of a Future.
Related
I have a method, which may throw an Exception depends on passed value:
private def transform(in: Int): Future[Boolean] = in match {
case i if i < 0 => Future.successful(true)
case i if i > 0 => Future.successful(false)
case i if i == 0 => throw new IllegalStateException()
}
And second method, which apply above method to each element of List in parallel way.
def massTransform(ints: List[Int])(implicit ex: ExecutionContext):
Future[List[Boolean]] = {
Future.traverse(ints){
i => transform(i).recover {
case e: IllegalStateException => false
}
}
}
I expected, recover will capture the IllegalStateException and return Future(false). But my code fails with IllegalStateException
The problem is that you are throwing outside of a Future. You need to wrap your exception in a Future, otherwise what happens is that the method itself throws instead of returning a failed Future. You can simplify your method as follows:
def transform(in: Int): Future[Boolean] =
if (in == 0) Future.failed(new IllegalStateException)
else Future.successful(in < 0)
You can play around with this code here on Scastie.
This looks like a simplification of your actual logic. If by any chance that's not (or not completely) the case, I recommend removing the unnecessary Future nesting as follows:
def transform(in: Int): Boolean =
if (in == 0) throw new IllegalStateException else in < 0
def massTransform(
ints: List[Int]
)(implicit ex: ExecutionContext): Future[List[Boolean]] =
Future(ints.map { i =>
try transform(i)
catch { case e: IllegalStateException => false }
})
Notice how now the transform method throws and exception handling is done in the Future constructor.
You can play around with this second version on Scastie as well.
transform is implemented in incorrectly, should be:
case i if i == 0 => Future.failed(new IllegalStateException())
I guess this is a simplified example, because this code does not really parallelise the computation btw, unless transform is in reality doing IO or quite expensive computation.
The Scala Try construct together with its flatMap does not work as I would expect or want it to. The TL;DR is that I want to do a series of operations that can fail in two ways: either by raising an exception, which should be promoted and caught higher up in the call stack, or by returning Failure, as the failure must logically be handled in different parts of the program.
I would expect something like this to do the trick:
def firstStepSucceeds(): Try[Int] = Try {
1
}
def secondStepThrows(input: Int) = {
throw new Exception("Exception thrown in second step")
}
// I expect this to propagate the exception thrown in secondStepThrows
firstStepSucceeds() flatMap (secondStepThrows _)
(Full Scastie with example)
However, in this case, the flatMap() call actually implicitly catches the uncaught exception thrown by secondStepThrows, which is not what I want (which is why I left out the Try block). Is there a way to get the same behaviour without the implicit exception-catching?
What happens in a Try should stay in a Try. Most Scala programmers would be very surprised if a function returning a Try also sometimes threw an exception.
The typical pattern if you want to handle exceptions in different places is to differentiate by the type of the exception. So
val partiallyRecoveredTry = originalTry.recover{
case _: SecondStepException => "second step had an exception"
}
// Further up the call stack
partiallyRecoveredTry.getOrElse("first step had an exception")
Try.flatMap() did not caught exceptions implicitely, it is the essence of Try.
When you use it, it is very explicit, and that's the goal.
I don't really understand what you want, but is something like that is possible for you ?
try {
val first = firstStepSucceeds()
val second = first.map(secondStepThrows).get
val third = secondStepFails(second)
// ...
}
catch {
case e: Exception => ???
}
I did some further experimentation, and what I ended up with was this reimplementation of Try as (the now right-biased and hence monadic) Either:
object CatchAll {
def apply[SomeType](block: => SomeType) = try { Right(block) }
catch { case e: Throwable => Left(e) }
}
def firstStepSucceeds() = CatchAll {
1
}
def firstStepFails() = CatchAll {
throw new Exception("First step failed")
}
def secondStepSucceeds(input: Int) = CatchAll {
input + 1
}
def secondStepFails(input: Int) = CatchAll {
throw new Exception("Second step failed in try block!")
}
def secondStepThrows(input: Int) = {
throw new Exception("Second step failed unexpectedly!")
}
firstStepSucceeds() flatMap (secondStepSucceeds _)
firstStepFails() flatMap (secondStepSucceeds _)
firstStepSucceeds() flatMap (secondStepFails _)
// This now throws an exception as expected
//firstStepSucceeds() flatMap (secondStepThrows _)
Considering a sequence of futures each returning Either[Status, Resp].
How would you propagate error status codes through a for comprehension which is using Future and not Either?
The code bellow does not work, since the parsing exception is not caught by .recover of the last future
The use case is Scala Play ActionRefiners which returns Future[Either[Status, TRequest[A]]].
def parseId(id: String):Future[Int] = {
Future.successful(Integer.parseInt(id))
}
def getItem(id: Int)(implicit ec: ExecutionContext): Future[Either[Status, String]] =
Future(Some("dummy res from db " + id)).transformWith {
case Success(opt) => opt match {
case Some(item) => Future.successful(Right(item))
case _ => Future.successful(Left(NotFound))
}
case Failure(_) => Future.successful(Left(InternalServerError))
}
(for {
id <- parseId("bad request")
resp <- getItem(id)
} yield resp).recover {
case _:NumberFormatException => Left(BadRequest)
}
I could move the .recover to parseId, but this makes the for comprehension very ugly - having to treat the Either[Status, id] in the middle
def parseId(id: String):Future[Either[Status, Int]] = {
Future.successful(Right(Integer.parseInt(id))).recover {
case _:NumberFormatException => Left(BadRequest)
}
}
Your exception is not caught because you are not throwing it inside the Future: Future.successful is immediately satisfied with the result of the expression you give it, if it throws an exception, it is executed on the current thread.
Try removing the .successful: Future(id.toInt) will do what you want.
Also, I would recommend to get rid of all the Eithers: these are highly overrated/overused, especially in the context of Future (that already wrap their result into Try anyhow), and just make the code more complicated and less readable without offering much benefit.
case class FailureReason(status: Status)
extends Exception(status.toString)
def notFound() = throw FailureReason(NotFound)
def internalError() = throw FailureReason(InternalError)
def badRequest() = throw FailureReason(BadRequest)
def parseId(id: String):Future[Int] = Future(id.toInt)
def getItem(id: Int): Future[String] = Future(Some("dummy"))
.map { _.getOrElse(notFound) }
.recover { _ => internalError }
// this is the same as your for-comprehension, just looking less ugly imo :)
parseId("foo").flatMap(getItem).recover {
case _: NumberFormatException => badRequest()
}
// if you still want `Either` in the end for some reason:
.map(Right.apply[Status, String])
.recover {
case _: NumberFormatException => Left(BadRequest) // no need for the first recover above if you do this
case FailureReason(status) => Left(status)
}
I am trying to read incremental data from my data source using Scala-Spark. Before hitting the source tables, I am trying to calculate the min & max of partition column that I use in my code in a Future which is present in a class: GetSourceMeta as given below.
def getBounds(keyIdMap:scala.collection.mutable.Map[String, String]): Future[scala.collection.mutable.Map[String, String]] = Future {
var boundsMap = scala.collection.mutable.Map[String, String]()
keyIdMap.keys.foreach(table => if(!keyIdMap(table).contains("Invalid")) {
val minMax = s"select max(insert_tms) maxTms, min(insert_tms) minTms from schema.${table} where source='DB2' and key_id in (${keyIdMap(table)})"
println("MinMax: " + minMax)
val boundsDF = spark.read.format("jdbc").option("url", con.getConUrl()).option("dbtable", s"(${minMax}) as ctids").option("user", con.getUserName()).option("password", con.getPwd()).load()
try {
val maxTms = boundsDF.select("minTms").head.getTimestamp(0).toString + "," + boundsDF.select("maxTms").head.getTimestamp(0).toString
println("Bounds: " + maxTms)
boundsMap += (table -> maxTms)
} catch {
case np: java.lang.NullPointerException => { println("No data found") }
case e: Exception => { println(s"Unknown exception: $e") }
}
}
)
boundsMap.foreach(println)
boundsMap
}
I am calling the above method in my main method as:
object LoadToCopyDB {
val conf = new SparkConf().setAppName("TEST_YEAR").set("some parameters")
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().config(conf).master("yarn").enableHiveSupport().config("hive.exec.dynamic.partition", "true").config("hive.exec.dynamic.partition.mode", "nonstrict").getOrCreate()
val gsm = new GetSourceMeta()
val minMaxKeyMap = gsm.getBounds(keyIdMap).onComplete {
case Success(values) => values.foreach(println)
case Failure(f) => f.printStackTrace
}
.
.
.
}
Well, the onComplete didn't print any values so I used andThen as below and that didn't help as well.
val bounds: Future[scala.collection.mutable.Map[String, String]] = gpMetaData.getBounds(incrementalIds) andThen {
case Success(outval) => outval.foreach(println)
case Failure(e) => println(e)
}
Earlier the main thread exits without letting the Future: getBounds execute. Hence I couldn't find any println statements from the Future displayed on the terminal. I found out that I need to keep the main thread Await inorder to complete the Future. But when I use Await in main along with onComplete:
Await.result(bounds, Duration.Inf)
The compiler gives an error:
Type mismatch, expected: Awaitable[NotInferedT], actual:Unit
If I declare the val minMaxKeyMap as Future[scala.collection.mutable.Map[String, String] the compiler says: Expression of type Unit doesn't conform to expected type Future[mutable.map[String,String]]
I tried to print the values of bounds after the Await statement but that just prints an empty Map.
I couldn't understand how can to fix this. Could anyone let me know what do I do to make the Future run properly ?
In this kind of cases, is always better to follow the types. The method onComplete only returns Unit, it won´t return a future hence it can´t be passed using Await.
In case you want to return a Future of any type you will have to map or flatmap the value and return an option, for example. In this case, does not matter what you return, you only want Await method to wait for this result and print a trace. You can treat the possible exception in the recover. It would be like that in your code:
val minMaxKeyMap:Future[Option[Any] = gsm.getBounds(keyIdMap).map { values =>
values.foreach(println)
None
}.recover{
case e: Throwable =>
e. printStackTrace
None
}
Note that the recover part has to return an instance of the type.
After that, you can apply the Await to the Future, and you will get the results printed. Is not the prettiest solution but it will work in your case.
I have a method that may return Future - successful or failed or can even throw an exception. I can avoid this by putting try catch block on entire method and return Future all the time but i would like to avoid it for now. I have few problems with calling such method:
1) In caller code, if I use map I expect execution of a method and expect a Future or an exception which I tried to handle in following manner:
object ETLCoordinator {
private def getBusinessListFromModules(modulePaths: Iterable[File]) : Future[String] = {
implicit val ec = ExecutionContext.global
println("Inside getBusinessListFromModules..")
throw new java.lang.RuntimeException("failed to get businesses") //Exception is thrown before future was constructed
Future("ok")
}
def main(args: Array[String]) {
println("Inside Future Test..")
implicit val ec = ExecutionContext.global
val modulePaths = Iterable(new File("mdrqaint/MDR/QAINT/QAINTX"))
val fut1 = getBusinessListFromModules(modulePaths) //This is outside of try and which should be okay
try {
fut1.map { res =>
println("things after Successful fut1")
}.recover{
case t: Throwable => println("Failed future in fut1: "+ t.getMessage)
}
} catch {
case t: Throwable => println("Exception in fut1: "+ t.getMessage)
}
}
}
Output: ( No execution of recover or catch block above)
Inside Future Test..
Inside getBusinessListFromModules..
Exception in thread "main" java.lang.RuntimeException: failed to get businesses
But if I put val fut1 = getBusinessListFromModules(modulePaths) inside Try block then Exception is get caught in Catch block and I get output:
Inside Future Test..
Inside getBusinessListFromModules..
Exception in fut1: failed to get businesses
Why is this? I though Future execution happens upon calling some of its methods like map, flatmap, onSuccess, onComplete etc. In this case call to map is already inside Try block.
2) What is the better way to define and call such methods? Try/catch block in a caller or try/catch in method itself? or any other way. I tried wrapping the calling method in Future so I get Future[Future[String]] in caller. I was able to avoid all try-catch.
val fut1 = Future(getBusinessListFromModules(modulePaths))
//try {
fut1.map { res =>
res.map{ str =>
println("things after Successful fut1")
}.recover{
case t: Throwable => println("Failed in future of fut1: "+ t.getMessage)
}
println("things after Successful fut1 wrapper")
}.recover{
case t: Throwable => println("Failed to create future in fut1: "+ t.getMessage)
}
3) If there is another method inbetween which does delegation to getBusinessListFromModules but it itself is non-future method.
object ETLController {
private def getBusinessListFromModules(modulePaths: Iterable[File]) : Future[String] = {
implicit val ec = ExecutionContext.global
println("Inside getBusinessListFromModules..")
//throw new java.lang.RuntimeException("failed to get businesses")
Future("ok")
}
private def callGetBusList(modulePaths: Iterable[File]) : String = {
implicit val ec = ExecutionContext.global
val etlF = getBusinessListFromModules(modulePaths)
etlF onComplete {
case Success(itr) => {
println("Future getBusinessListFromModules success: "+ itr)
throw new java.lang.RuntimeException("RTE from callGetBusList")
}
case Failure(t) => {
println("Future getBusinessListFromModules throws an error")
}
}
"callGetBusList was a success"
}
def main(args: Array[String]) {
println("Inside Future Test..")
implicit val ec = ExecutionContext.global
val modulePaths = Iterable(new File("mdrqaint/MDR/QAINT/QAINTX"))
try {
val fut = Future(callGetBusList(modulePaths))
fut.map { res =>
println("successful future!")
}.recover{
case t: Throwable => println("Failed future: "+ t.getMessage)
}
} catch {
case t: Throwable => println("callGetBusList failed:" + t.getMessage)
}
}
}
Output: (no recover or catch block execution!)
Inside Future Test..
Inside getBusinessListFromModules..
Future getBusinessListFromModules success: ok
java.lang.RuntimeException: RTE from callGetBusList
at ..
successful future!
I even try double wrapping Future calls :
val fut = Future(Future(callGetBusList(modulePaths)))
fut.map { res =>
res.map { str =>
println("successful inner future! "+ str)
}.recover{
case t: Throwable => println("Failed inner future: "+ t.getMessage)
}
println("successful outer future!")
}.recover{
case t: Throwable => println("Failed outer future: "+ t.getMessage)
}
Output:
Future getBusinessListFromModules success: ok
java.lang.RuntimeException: RTE from callGetBusList
at
successful inner future! callGetBusList was a success
successful outer future!
I get "callGetBusList was a success" which seems like RuntimeException inside onComplete method got lost! How do I catch it in a final caller? What are the better practice to handle such future dependencies?
UPDATE:
based on #dk14 explanation, Opted to convert middle method to return Future and basically all methods to return some kind of Future and not a plain Exception.
object ETLController {
private def getBusinessListFromModules(modulePaths: Iterable[File]) : Future[String] = {
implicit val ec = ExecutionContext.global
println("Inside getBusinessListFromModules..")
Future {
Thread.sleep(2000)
throw new java.lang.RuntimeException("failed to get businesses")
"ok"
}
}
private def callGetBusList(modulePaths: Iterable[File]) : Future[String] = {
implicit val ec = ExecutionContext.global
val etlF = getBusinessListFromModules(modulePaths)
etlF map { itr =>
println("Future getBusinessListFromModules success: "+ itr)
throw new java.lang.RuntimeException("RTE from callGetBusList")
} recover {
case t: Throwable => {
println("Future callGetBusList throws an error: " + t.getMessage)
throw t
}
}
}
def main(args: Array[String]) {
println("Inside Future Test..")
implicit val ec = ExecutionContext.global
val modulePaths = Iterable(new File("mdrqaint/MDR/QAINT/QAINTX"))
val fut = callGetBusList(modulePaths)
fut.map { str =>
println("successful future! "+ str)
}.recover{
case t: Throwable => println("Failed future: "+ t.getMessage)
}
println("Active threads: " +Thread.activeCount())
sys.allThreads().foreach(t => t.join())
}
}
1) Futures are firing eagerly and they aren't referentially transparent.
Answers to the referenced question also contain some insights about Future's internal behavior, so I'd like to skip it here.
In order to manage side-effects concerning execution pools/queues/threads in a more predictable way, you could consider scalaz/monix/fs2 Task or iteratee/scalaz/cats Eval (more abstract lazy evaluation, and intended for sync stuff) + Cont (continuations are abstracting over subscriptions) as alternative. All are referentially transparent and start execution lazily "on-demand".
2) The best way is the one you don't like: to not throw exception outisde of Future context.
You might also consider flatMap to avoid Future[Future[T]]
3) Double wrapping Futures directly a-la Future(Future(...)) doesn't change anything. Your method is executed on val etlF = g... (in the same thread) no matter what it returns. Future("ok")'s content (lambda) is executed eagerly (with "small" unpredictable delay) on a different thread but [execution task is being submitted to the pool] still inside getBusinessListFromModules.
One workaround (not really recommended) is val etlF = Future(getBusinessListFromModules(...)).flatMap(identity) which would return you a future wrapping any exception coming directly from getBusinessListFromModules and indirectly from getBusinessListFromModules's internal Future.
It's better to refactor getBusinessListFromModules itself however, also introduce different exception types for different kinds of trouble (validation, sync vs async, so on) your method might get into.
P.S. There are ways to mix async and sync exception handling, but in practice it's hard to analyze and predict such mixed behavior (which you probably noticed already). And code gets ugly.