Say I have a series of independent computations that need to be combined at the end:
def makeSauce(): Try[Sauce] = Success(Sauce("marinara"))
def makeCrust(): Try[Crust] = Failure(new RuntimeException("stale bread"))
def makeCheese(): Try[Cheese] = Failure(new IllegalArgumentException("moldy cheese"))
for {
sauce <- makeSauce()
crust <- makeCrust()
cheese <- makeCheese()
} yield {
Pizza(sauce, crust, cheese)
}
But if any of my computations fail, I want a comprehensive error message describing all of the failures, not just the first. The above code doesn't do what I want; it only shows me the first failure ("stale bread"). Instead, it should throw a super-error message combining both of the others, which could be a sequence of exception objects, sequence of string error messages, or even just one giant concatenated string.
Alternatively, instead of throwing the exception, it could be composed into a top-level Failure object:
(for {
sauce <- makeSauce()
crust <- makeCrust()
cheese <- makeCheese()
} yield (sauce, crust, cheese)) match {
case Success((sauce, crust, cheese)) => Pizza(sauce, crust, cheese)
case Failure(e) => throw e
}
The for-comprehension and the Try here are just for illustration; a working solution doesn't have to use for or Try. However, obviously the above syntax is expressive and clean, and almost does what I want, so ideally the solution will stick to it as closely as possible.
How can I achieve something like this in Scala?
Validated is what you are looking for. It is in Cats for example.
import cats.data.Validated
import cats.implicits.catsSyntaxTuple3Semigroupal
def valid(input: String) = if (input.toIntOption.isEmpty) Validated.Invalid(s"$input is not a number")
else Validated.Valid(input.toInt)
def sumThreeNumbers(one:String, two:String, three: String) = (valid(one), valid(two), valid(three)).mapN((oneInt, twoInt, threeInt)=> oneInt + twoInt + threeInt)
sumThreeNumbers("1","2","3")
sumThreeNumbers("1","two","xxx")
//results
val res0: cats.data.Validated[String,Int] = Valid(6)
val res1: cats.data.Validated[String,Int] = Invalid(two is not a number.xxx is not a number.)
scala.util.Try behaves in a fail-fast manner. In a for-comprehension context it executed sequentially and halts at the first failure.
What you need is a structure that aggregates all failures like cats.data.Validated or scalaz.Validation.
Related
How can I remove the option so it is just Try[Int] and not Try[Option[Int]]?
val m = Map("a" -> "1a", "b" -> "2")
Try(m.get("a").map(_.trim.toInt))
>>es17: scala.util.Try[Option[Int]] = Failure(java.lang.NumberFormatException: For input string: "1a")
Map#get returns an Option[String], but you can use Map#apply instead, which will return String, in this case.
scala> Try(m("a").trim.toInt)
res3: scala.util.Try[Int] = Failure(java.lang.NumberFormatException: For input string: "1a")
scala> Try(m("b").trim.toInt)
res4: scala.util.Try[Int] = Success(2)
apply throws an exception if the key you're looking for doesn't exist, but Try will catch it, anyway.
This answer goes in more detail about the comment:
I was wondering if there was a way to use flapmap? Your solution works for me, just want to learn of other alternatives.
As you've probably heard, Option and Try are monad instances and while monads are handy to represent sequence of computations, they don't compose with other monads. In other words, we can't compose Option and Try. We need to find a common ground.
The difference in semantics between Option and Try is that Try contains information about the case when a result is absent.
We can go from Try to Option using Try#toOption effectively loosing any failure information we may have.
If we wanted to go the other way, we need to add this information back: ne need to provide a failure reason when a value is absent in an Option. Something like this:
import scala.util.{Try, Success, Failure}
def optionToTry[T](opt:Option[T], failure: => Throwable): Try[T] = opt match {
case Some(v) => Success(v)
case None => Failure(failure)
}
With the help of that function, we can rewrite the original expression as:
val res: Try[Int] = for {
strValue <- optionToTry(m.get("a"), new NoSuchElementException("a"))
value <- Try(strValue.trim.toInt)
} yield value
which uses flatMap behind the scenes to compose the two Try instances like this:
val res = optionToTry(m.get("a"), new NoSuchElementException("a"))
.flatMap(strValue => Try(strValue.trim.toInt))
Note that we could save ourselves a bit of coding by using the unsafe map getter like so:
val res: Try[Int] = for {
strValue <- Try(m("a"))
value <- Try(strValue.trim.toInt)
} yield value
but this version would be computationally more expensive given the cost of handling exceptions in the JVM.
I have a sequence of parameters. For each parameter I have to perform DB query, which may or may not return a result. Simply speaking, I need to stop after the first result is non-empty. Of course, I would like to avoid doing unnecessary calls. The caveat is - I need to have this operation(s) contained as a another Future - or any "most reactive" approach.
Speaking of code:
//that what I have
def dbQuery(p:Param): Future[Option[Result]] = {}
//my list of params
val input = Seq(p1,p2,p3)
//that what I need to implements
def getFirstNonEmpty(params:Seq[Param]): Future[Option[Result]]
I know I can possibly just wrap entire function in yet another Future and execute code sequentially (Await? Brrr...), but that not the cleanest solution.
Can I somehow create lazy initialized collection of futures, like
params.map ( p => FutureWhichWontStartUnlessAskedWhichWrapsOtherFuture { dbQuery(p) }).findFirst(!_.isEmpty())
I believe it's possible!
What do you think about something like this?
def getFirstNonEmpty(params: Seq[Param]): Future[Option[Result]] = {
params.foldLeft(Future.successful(Option.empty[Result])) { (accuFtrOpt, param) =>
accuFtrOpt.flatMap {
case None => dbQuery(param)
case result => Future.successful(result)
}
}
}
This might be overkill, but if you are open to using scalaz we can do this using OptionT and foldMap.
With OptionT we sort of combine Future and Option into one structure. We can get the first of two Futures with a non-empty result using OptionT.orElse.
import scalaz._, Scalaz._
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
val someF: Future[Option[Int]] = Future.successful(Some(1))
val noneF: Future[Option[Int]] = Future.successful(None)
val first = OptionT(noneF) orElse OptionT(someF)
first.run // Future[Option[Int]] = Success(Some(1))
We could now get the first non-empty Future from a List with reduce from the standard library (this will however run all the Futures) :
List(noneF, noneF, someF).map(OptionT.apply).reduce(_ orElse _).run
But with a List (or other collection) we can't be sure that there is at least one element, so we need to use fold and pass a start value. Scalaz can do this work for us by using a Monoid. The Monoid[OptionT[Future, Int]] we will use will supply the start value and combine the Futures with the orElse used above.
type Param = Int
type Result = Int
type FutureO[x] = OptionT[Future, x]
def query(p: Param): Future[Option[Result]] =
Future.successful{ println(p); if (p > 2) Some(p) else None }
def getFirstNonEmpty(params: List[Param]): Future[Option[Result]] = {
implicit val monoid = PlusEmpty[FutureO].monoid[Result]
params.foldMap(p => OptionT(query(p))).run
}
val result = getFirstNonEmpty(List(1,2,3,4))
// prints 1, 2, 3
result.foreach(println) // Some(3)
This is an old question, but if someone comes looking for an answer, here is my take. I solved it for a use case that required me to loop through a limited number of futures sequentially and stop when the first of them returned a result.
I did not need a library for my use-case, a light-weight combination of recursion and pattern matching was sufficient. Although the question here does not have the same problem as a sequence of futures, looping through a sequence of parameters would be similar.
Here would be the pseudo-code based on recursion.
I have not compiled this, fix the types being matched/returned.
def getFirstNonEmpty(params: Seq[Param]): Future[Option[Result]] = {
if (params.isEmpty) {
Future.successful(None)
} else {
val head = params.head
dbQuery(head) match {
case Some(v) => Future.successful(Some(v))
case None => getFirstNonEmpty(params.tail)
}
}
}
I have to get a list of issues for each file of a given list from a REST API with Scala. I want to do the requests in parallel, and use the Dispatch library for this. My method is called from a Java framework and I have to wait at the end of this method for the result of all the futures to yield the overall result back to the framework. Here's my code:
def fetchResourceAsJson(filePath: String): dispatch.Future[json4s.JValue]
def extractLookupId(json: org.json4s.JValue): Option[String]
def findLookupId(filePath: String): Future[Option[String]] =
for (json <- fetchResourceAsJson(filePath))
yield extractLookupId(json)
def searchIssuesJson(lookupId: String): Future[json4s.JValue]
def extractIssues(json: org.json4s.JValue): Seq[Issue]
def findIssues(lookupId: String): Future[Seq[Issue]] =
for (json <- searchIssuesJson(componentId))
yield extractIssues(json)
def getFilePathsToProcess: List[String]
def thisIsCalledByJavaFramework(): java.util.Map[String, java.util.List[Issue]] = {
val finalResultPromise = Promise[Map[String, Seq[Issue]]]()
// (1) inferred type of issuesByFile not as expected, cannot get
// the type system happy, would like to have Seq[Future[(String, Seq[Issue])]]
val issuesByFile = getFilePathsToProcess map { f =>
findLookupId(f).flatMap { lookupId =>
(f, findIssues(lookupId)) // I want to yield a tuple (String, Seq[Issue]) here
}
}
Future.sequence(issuesByFile) onComplete {
case Success(x) => finalResultPromise.success(x) // (2) how to return x here?
case Failure(x) => // (3) how to return null from here?
}
//TODO transform finalResultPromise to Java Map
}
This code snippet has several issues. First, I'm not getting the type I would expect for issuesByFile (1). I would like to just ignore the result of findLookUpId if it is not able to find the lookUp ID (i.e., None). I've read in various tutorials that Future[Option[X]] is not easy to handle in function compositions and for expressions in Scala. So I'm also curious what the best practices are to handle these properly.
Second, I somehow have to wait for all futures to finish, but don't know how to return the result to the calling Java framework (2). Can I use a promise here to achieve this? If yes, how can I do it?
And last but not least, in case of any errors, I would just like to return null from thisIsCalledByJavaFramework but don't know how (3).
Any help is much appreciated.
Thanks,
Michael
Several points:
The first problem at (1) is that you don't handle the case where findLookupId returns None. You need to decide what to do in this case. Fail the whole process? Exclude that file from the list?
The second problem at (1) is that findIssues will itself return a Future, which you need to map before you can build the result tuple
There's a shortcut for map and then Future.sequence: Future.traverse
If you cannot change the result type of the method because the Java interface is fixed and cannot be changed to support Futures itself you must wait for the Future to be completed. Use Await.ready or Await.result to do that.
Taking all that into account and choosing to ignore files for which no id could be found results in this code:
// `None` in an entry for a file means that no id could be found
def entryForFile(file: String): Future[(String, Option[Seq[Issue]])] =
findLookupId(file).flatMap {
// the need for this kind of pattern match shows
// the difficulty of working with `Future[Option[T]]`
case Some(id) ⇒ findIssues(id).map(issues ⇒ file -> Some(issues))
case None ⇒ Future.successful(file -> None)
}
def thisIsCalledByJavaFramework(): java.util.Map[String, java.util.List[Issue]] = {
val issuesByFile: Future[Seq[(String, Option[Seq[Issue]])]] =
Future.traverse(getFilePathsToProcess)(entryForFile)
import scala.collection.JavaConverters._
try
Await.result(issuesByFile, 10.seconds)
.collect {
// here we choose to ignore entries where no id could be found
case (f, Some(issues)) ⇒ f -> issues
}
.toMap.mapValues(_.asJava).asJava
catch {
case NonFatal(_) ⇒ null
}
}
I have posted quite several questions about failure handling in Scala and I really thank you all for your answers.
I understand my options when dealing with Either and Scalaz or a for comprehension, and I have another (last?) question:
How to do a fail-fast sequence of operations when the operations are dealing with the outside non-functional world, like a DB?
I mean I have a method like that:
def insertItem(item: Item): Either[Error,Item]
Thanks to Either and these answers, I know how to do it with Either: Chaining method calls with Either and
Method parameters validation in Scala, with for comprehension and monads
But my Item case class is immutable and it doesn't really make sense to return it as a Right since the caller already has the value.
Thus how can I do the same kind of thing with:
def insertItem(item: Item): Option[Error]
In my application, when an user is created, we also create some items for him.
If an item fails to create, then the whole process should stop.
When I use directly Option[Error] in a for comprehension, I don't think I'll get the result I expect.
I guess it makes sense to do something like that:
for {
_ <- insertItem(item1).toLeft("???").right
_ <- insertItem(item2).toLeft("???").right
_ <- insertItem(item3).toLeft("???").right
}
But as the values "???" I put in my Right are never useful, I guess I'm missing the elegant solution which do not involve creating Rights that will never be used.
I think I'm looking for something that will continue the for comprehension only when the result is None, which is kind of weird because I just want to continue to the next operation, and not do a real map operation.
By the way, if possible, I would like both non-Scalaz and Scalaz solutions.
I'm not sure Scalaz handle such things, because it seems more focused on real functional programming and perhaps do not provide code for side-effect behaviors like my use case?
I don't see a principled reason not to use Either[Error, Unit] in a case like this (or at least I've done it, and I don't feel guilty about it). Say we have the following:
def launch(thing: String): Either[String, Unit] = Either.cond(
thing.nonEmpty,
println("Launching the " + thing),
"Need something to launch!"
)
We can show that the right projection monad is appropriately lazy:
scala> for {
| _ <- launch("flowers").right
| _ <- launch("").right
| r <- launch("MISSILES").right
| } yield r
Launching the flowers
res1: Either[String,Unit] = Left(Need something to launch!)
No missiles get launched, as desired.
It's worth noting that if you use Option instead of Either, the operation you're describing is just the sum given the "First" monoid instance for Option (where the addition operation is just orElse). For example, we can write the following with Scalaz 7:
import scalaz._, Scalaz._
def fst[A](x: Option[A]): Option[A] ## Tags.First = Tag(x)
def launch(thing: String): Option[String] = if (thing.isEmpty) Some(
"Need something to launch!"
) else {
println("Launching the " + thing)
None
}
And now:
scala> val r: Option[String] = fst(launch("flowers")) |+| fst(
| launch("")) |+| fst(launch("MISSILES"))
Launching the flowers
r: Option[String] = Some(Need something to launch!)
Once again, no missiles.
We do something similar where I work with scalaz to fail fast with message sending, though this is not idiomatic scalaz:
def insertItem(item: Item): Validation[Error, Unit]
val result = for {
_ <- insertItem(item1)
_ <- insertItem(item2)
_ <- insertItem(item3)
} yield Unit
If you want to chain together methods that contain Option[Error] and only execute the next step if the Option is None, you can use orElse for that.
If you are only interested in the first failure, than an Iterator can be your choice:
case class Item(i: Int)
case class Error(i: Item)
val items = Iterator(1,3,7,-5,-2,0) map Item
def insertItem(item: Item): Option[Error] =
if (item.i < 0) Some(Error(item)) else None
scala> val error = (items map insertItem find (_.isDefined)).flatten
error: Option[Error] = Some(Error(Item(-5)))
If insertItem is not your only method it is possible to call them separated with your values:
val items = Iterator(
() => insertItem(i1),
() => deleteItem(i2),
() => insertItem(i3),
() => updateItem(i4))
I'm developing a method that is supposed to persist an object, if it passes a list of conditions.
If any (or many) condition fail (or any other kind of error appears), a list with the errors should be returned, if everything goes well, a saved entity should be returned.
I was thinking about something like this (it's pseudocode, of course):
request.body.asJson.map { json =>
json.asOpt[Wine].map { wine =>
wine.save.map { wine =>
Ok(toJson(wine.update).toString)
}.getOrElse { errors => BadRequest(toJson(errors))}
}.getOrElse { BadRequest(toJson(Error("Invalid Wine entity")))}
}.getOrElse { BadRequest(toJson(Error("Expecting JSON data")))}
That is, I'd like to treat it like an Option[T], that if any validation fails, instead of returning None it gives me the list of errors...
The idea is to return an array of JSON errors...
So the question would be, is this the right way to handle these kind of situation? And what would be the way to accomplish it in Scala?
--
Oops, just posted the question and discovered Either
http://www.scala-lang.org/api/current/scala/Either.html
Anyway, I'd like to know what you think about the chosen approach, and if there's any other better alternative to handle it.
Using scalaz you have Validation[E, A], which is like Either[E, A] but has the property that if E is a semigroup (meaning things that can be concatenated, like lists) than multiple validated results can be combined in a way that keeps all the errors that occured.
Using Scala 2.10-M6 and Scalaz 7.0.0-M2 for example, where Scalaz has a custom Either[L, R] named \/[L, R] which is right-biased by default:
import scalaz._, Scalaz._
implicit class EitherPimp[E, A](val e: E \/ A) extends AnyVal {
def vnel: ValidationNEL[E, A] = e.validation.toValidationNEL
}
def parseInt(userInput: String): Throwable \/ Int = ???
def fetchTemperature: Throwable \/ Int = ???
def fetchTweets(count: Int): Throwable \/ List[String] = ???
val res = (fetchTemperature.vnel |#| fetchTweets(5).vnel) { case (temp, tweets) =>
s"In $temp degrees people tweet ${tweets.size}"
}
Here result is a Validation[NonEmptyList[Throwable], String], either containing all the errors occured (temp sensor error and/or twitter error or none) or the successful message. You can then switch back to \/ for convenience.
Note: The difference between Either and Validation is mainly that with Validation you can accumulate errors, but cannot flatMap to lose the accumulated errors, while with Either you can't (easily) accumulate but can flatMap (or in a for-comprehension) and possibly lose all but the first error message.
About error hierarchies
I think this might be of interest for you. Regardless of using scalaz/Either/\//Validation, I experienced that getting started was easy but going forward needs some additional work. The problem is, how do you collect errors from multiple erring functions in a meaningful way? Sure, you can just use Throwable or List[String] everywhere and have an easy time, but doesn't sound too much usable or interpretable. Imagine getting a list of errors like "child age missing" :: "IO error reading file" :: "division by zero".
So my choice is to create error hierarchies (using ADT-s), just like as one would wrap checked exceptions of Java into hierarchies. For example:
object errors {
object gamestart {
sealed trait Error
case class ResourceError(e: errors.resource.Error) extends Error
case class WordSourceError(e: errors.wordsource.Error) extends Error
}
object resource {
case class Error(e: GdxRuntimeException)
}
object wordsource {
case class Error(e: /*Ugly*/ Any)
}
}
Then when using result of erring functions with different error types, I join them under a relevant parent error type.
for {
wordSource <-
errors.gamestart.WordSourceError <-:
errors.wordsource.Error <-:
wordSourceCreator.doCreateWordSource(mtRandom).catchLeft.unsafePerformIO.toEither
resources <-
errors.gamestart.ResourceError <-:
GameViewResources(layout)
} yield ...
Here f <-: e maps the function f on the left of e: \/ since \/ is a Bifunctor. For se: scala.Either you might have se.left.map(f).
This may be further improved by providing shapeless HListIsos to be able to draw nice error trees.
Revisions
Updated: (e: \/).vnel lifts the failure side into a NonEmptyList so if we have a failure we have at least one error (was: or none).
If you have Option values, and you want to turn them into success/failure values, you can turn an Option into an Either using the toLeft or toRight method.
Usually a Right represents success, so use o.toRight("error message") to turn Some(value) into Right(value) and None into Left("error message").
Unfortunately Scala doesn't recognise this right-bias by default, so you have to jump through a hoop (by calling the .right method) in order to neatly compose your Eithers in a for-comprehension.
def requestBodyAsJson: Option[String] = Some("""{"foo":"bar"}""")
def jsonToWine(json: String): Option[Wine] = sys.error("TODO")
val wineOrError: Either[String, Wine] = for {
body <- requestBodyAsJson.toRight("Expecting JSON Data").right
wine <- jsonToWine(body).toRight("Invalid Wine entity").right
} yield wine
If you need an empty value, instead of using Either[A,Option[B]] you can use lift Box, which can have three values:
Full (there is a valid result)
Empty (no result, but no error either)
Failure (an error happened)
Box are more flexible than Either thanks to a rich API. Of course, although they were created for Lift, you can use them in any other framework.
well, this is my attemp using Either
def save() = CORSAction { request =>
request.body.asJson.map { json =>
json.asOpt[Wine].map { wine =>
wine.save.fold(
errors => JsonBadRequest(errors),
wine => Ok(toJson(wine).toString)
)
}.getOrElse (JsonBadRequest("Invalid Wine entity"))
}.getOrElse (JsonBadRequest("Expecting JSON data"))
}
And wine.save is like the following:
def save(wine: Wine): Either[List[Error],Wine] = {
val errors = validate(wine)
if (errors.length > 0) {
Left(errors)
} else {
DB.withConnection { implicit connection =>
val newId = SQL("""
insert into wine (
name, year, grapes, country, region, description, picture
) values (
{name}, {year}, {grapes}, {country}, {region}, {description}, {picture}
)"""
).on(
'name -> wine.name, 'year -> wine.year, 'grapes -> wine.grapes,
'country -> wine.country, 'region -> wine.region, 'description -> wine.description,
'picture -> wine.picture
).executeInsert()
val newWine = for {
id <- newId;
wine <- findById(id)
} yield wine
newWine.map { wine =>
Right(wine)
}.getOrElse {
Left(List(ValidationError("Could not create wine")))
}
}
}
}
Validate checks several preconditions. I still have to add a try/catch to catch any db error
I'm still looking for a way to improve the whole thing, it feels much to verbose to my taste...