I really feel like I'm doing something that is not possible by design, but I really want to fully understand how Play Iteratees work...
I'm currently writing a function that applies some transformation on an Enumerator through an Enumeratee. As of now, I have no problems doing this specific transformation. But I want to add some elements just before then end of the Enumerator. I thought of using mapInput or mapInputFlatten, since I can see which Input is being processed at the time. My current solution for this problems looks like:
import play.api.libs.iteratee.{Enumerator, Enumeratee, Input, Iteratee}
import scala.concurrent.Await
import scala.concurrent.duration.DurationInt
def appendingEnumeratee: Enumeratee[Int, Int] = Enumeratee.mapInputFlatten {
case Input.El(i) => Enumerator(i)
case Input.Empty => Enumerator.enumInput(Input.Empty)
case Input.EOF => Enumerator(42)
}
Await.result(Enumerator(1 to 41: _*).through(appendingEnumeratee).apply(Iteratee.getChunks), 5.seconds)
However, I end up with java.lang.RuntimeException: diverging iteratee after Input.EOF. This is on Scala 2.10.2 with play-iteratees 2.1.5
I think the problem with your code is the missing EOF. So modifying it to the following will work:
def appendingEnumeratee: Enumeratee[Int, Int] = Enumeratee.mapInputFlatten {
case Input.El(i) => Enumerator(i)
case Input.Empty => Enumerator.enumInput(Input.Empty)
case Input.EOF => Enumerator(42) andThen Enumerator.enumInput(Input.EOF)
}
Or a bit shorter:
def appendingEnumeratee: Enumeratee[Int, Int] = Enumeratee.mapInputFlatten {
case Input.EOF => Enumerator(42) >>> Enumerator.enumInput(Input.EOF)
case _else => Enumerator.enumInput(_else)
}
Related
I have a situation where I need to run a bunch of operations in parallel.
All operations have the same return value (say a Seq[String]).
Its possible that some of the operations may fail, and others successfully return results.
I want to return both the successful results, and any exceptions that happened, so I can log them for debugging.
Is there a built-in way, or easy way through any library (cats/scalaz) to do this, before I go and write my own class for doing this?
I was thinking of doing each operation in its own future, then checking each future, and returning a tuple of Seq[String] -> Seq[Throwable] where left value is the successful results (flattened / combined) and right is a list of any exceptions that occurred.
Is there a better way?
Using Await.ready, which you mention in a comment, generally loses most benefits from using futures. Instead you can do this just using the normal Future combinators. And let's do the more generic version, which works for any return type; flattening the Seq[String]s can be added easily.
def successesAndFailures[T](futures: Seq[Future[T]]): Future[(Seq[T], Seq[Throwable])] = {
// first, promote all futures to Either without failures
val eitherFutures: Seq[Future[Either[Throwable, T]]] =
futures.map(_.transform(x => Success(x.toEither)))
// then sequence to flip Future and Seq
val futureEithers: Future[Seq[Either[Throwable, T]]] =
Future.sequence(eitherFutures)
// finally, Seq of Eithers can be separated into Seqs of Lefts and Rights
futureEithers.map { seqOfEithers =>
val (lefts, rights) = seqOfEithers.partition(_.isLeft)
val failures = lefts.map(_.left.get)
val successes = rights.map(_.right.get)
(successes, failures)
}
}
Scalaz and Cats have separate to simplify the last step.
The types can be inferred by the compiler, they are shown just to help you see the logic.
Calling value on your Future returns an Option[Try[T]]. If the Future has not completed then the Option is None. If it has completed then it's easy to unwrap and process.
if (myFutr.isCompleted)
myFutr.value.map(_.fold( err: Throwable => //log the error
, ss: Seq[String] => //process results
))
else
// do something else, come back later
Sounds like a good use-case for the Try idiom (it's basically similar to the Either monad).
Example of usage from the doc:
import scala.util.{Success, Failure}
val f: Future[List[String]] = Future {
session.getRecentPosts
}
f onComplete {
case Success(posts) => for (post <- posts) println(post)
case Failure(t) => println("An error has occurred: " + t.getMessage)
}
It actually does a little bit more than what you asked because it is fully asynchronous. Does it fit your use-case?
I'd do it this way:
import scala.concurrent.{Future, ExecutionContext}
import scala.util.Success
def eitherify[A](f: Future[A])(implicit ec: ExecutionContext): Future[Either[Throwable, A]] = f.transform(tryResult => Success(tryResult.toEither))
def eitherifyF[A, B](f: A => Future[B])(implicit ec: ExecutionContext): A => Future[Either[Throwable, B]] = { a => eitherify(f(a)) }
// here we need some "cats" magic for `traverse` and `separate`
// instead of `traverse` you can use standard `Future.sequence`
// there is no analogue for `separate` in the standard library
import cats.implicits._
def myProgram[A, B](values: List[A], asyncF: A => Future[B])(implicit ec: ExecutionContext): Future[(List[Throwable], List[B])] = {
val appliedTransformations: Future[List[Either[Throwable, B]]] = values.traverse(eitherifyF(asyncF))
appliedTransformations.map(_.separate)
}
I have a method that returns Future[Try[Option[Int]]]. I want to extract value of Int for further computation. Any idea how to process it??
future.map(_.map(_.map(i => doSomethingWith(i))))
If you want use cats you can do fun (for certain definitions of fun) things like:
import scala.concurrent._
import scala.util._
import scala.concurrent.ExecutionContext.Implicits.global
import cats.Functor
import cats.instances.option._
import cats.implicits._
val x = Future { Try { Some(1) } } // your type
Functor[Future].compose[Try].compose[Option].map(x)(_ + 2)
This is suggested ONLY if you're already familiar with cats or scalaz.
Otherwise, you're great to go with any of the other valid answers here (I especially like the map-map-map one).
Just map the future and use match case to handle the different cases:
val result: Future[Try[Option[Int]]] = ???
result.map {
case Success(Some(r)) =>
println(s"Success. Result: $r")
//Further computation here
case Success(None) => //Success with None
case Failure(ex) => //Failed Try
}
Converting Future[Try[Option[Int]]] to Future[Int]
One hacky way is to convert the unfavourable results into failed future and flatMapping over.
Convert try failures to Future failures preserving the information that exception originated from Try and convert None to NoneFound exception.
val f: Future[Try[Option[Int]]] = ???
case class TryException(ex: Throwable) extends Exception(ex.getMessage)
case object NoneFound extends Exception("None found")
val result: Future[Int] = f.flatMap {
case Success(Some(value)) => Future.successful(value)
case Success(None) => Future.failed(NoneFound)
case Failure(th) => Future.failed(TryException(th))
}
result.map { extractedValue =>
processTheExtractedValue(extractedValue)
}.recover {
case NoneFound => "None case"
case TryException(th) => "try failures"
case th => "future failures"
}
Now in every case you know from where the exception has originated. In case of NoneFound exception you know Future and Try are successful but option is none. This way information is not lost and nested structure is flattened to Future[Int].
Now result type would be Future[Int]. Just use map, flatMap, recover and recoverWith to compose further actions.
If you really concerned about extraction see this, else go through the answer by #pamu to see how you actually use your Future.
Suppose your Future value is result.
Await.ready(result, 10.seconds).value.get.map { i => i.get}.get
Obviously this wont get through your failure and None cases and would throw exceptions and Await is not recommended.
So if you want to handle Failure and None case ->
val extractedValue = Await.ready(f, 10.seconds).value.get match {
case Success(i) => i match {
case Some(value) => value
case None => println("Handling None here")
}
case Failure(i) => println("Handling Failure here")
}
I have a Set of Future[T] that I want to manage into a single object for a library I am writing. In my current implementation I'm using a Future.sequence to collect them all and wait until they've resolved so I can do futurey things on them (map, collect, filter). However this only gives me the ability to match on Success or Failure, which is not necessarily the case for the collection of futures I'm dealing with. Some will fail and some will succeed and I would like to be able to extract the values I can from those that succeed and collect the exceptions and errors on the others so that I can escalate them appropriately. In pseudo code it would be something like
Future.sequence(Set[Future[T]]) andThen {
case FullSuccess => "woot"
case SomeErrors => "well, that's still ok."
case FullErrors => "Ok, who's the wise guy."
}
What I'm really looking for is to have data where there is data and not have to return a complete failure if only 1 of the futures in the sequence has failed.
Thanks for the help.
Unfortunately there is no builtin helper for your case, but it's easy to create your own:
import scala.concurrent.{Await, Future}
import scala.util.{Failure, Success, Try}
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration.DurationInt
def sequenceOfTries[T](futures: Seq[Future[T]]): Future[Seq[Try[T]]] =
futures.foldLeft(Future.successful(List[Try[T]]())) {
case (accF, f) => accF.flatMap {
acc => f.map(v => Success(v) :: acc).recover { case ex => Failure(ex) :: acc }
}
}.map(_.reverse)
val v = Seq(
Future.successful(1),
Future.failed(new IllegalStateException("2")),
Future.successful(3),
Future.failed(new IllegalStateException("4"))
)
Await.result(sequenceOfTries(v), 1.second)
Results:
v: Seq[scala.concurrent.Future[Int]] = List(scala.concurrent.impl.Promise$KeptPromise#2416f7e5, scala.concurrent.impl.Promise$KeptPromise#2aaf675d, scala.concurrent.impl.Promise$KeptPromise#360d48f, scala.concurrent.impl.Promise$KeptPromise#230f8be2)
res0: Seq[scala.util.Try[Int]] = List(Success(1), Failure(java.lang.IllegalStateException: 2), Success(3), Failure(java.lang.IllegalStateException: 4))
UPD. Alternatively you can utilize Future.sequence like this (with same result):
def sequenceOfTries[T](futures: Seq[Future[T]]): Future[Seq[Try[T]]] =
Future.sequence(futures.map(_.map(x => Success(x)).recover { case ex => Failure(ex) }))
Are there any conversions from Either to Try and vice versa in the Scala standard library ? Maybe I am missing something but I did not find them.
To the best of my knowledge this does not exist in the standard library. Although an Either is typically used with the Left being a failure and the Right being a success, it was really designed to support the concept of two possible return types with one not necessarily being a failure case. I'm guessing these conversions that one would expect to exist do not exist because Either was not really designed to be a Success/Fail monad like Try is. Having said that it would be pretty easy to enrich Either yourself and add these conversions. That could look something like this:
object MyExtensions {
implicit class RichEither[L <: Throwable,R](e:Either[L,R]){
def toTry:Try[R] = e.fold(Failure(_), Success(_))
}
implicit class RichTry[T](t:Try[T]){
def toEither:Either[Throwable,T] = t.transform(s => Success(Right(s)), f => Success(Left(f))).get
}
}
object ExtensionsExample extends App{
import MyExtensions._
val t:Try[String] = Success("foo")
println(t.toEither)
val t2:Try[String] = Failure(new RuntimeException("bar"))
println(t2.toEither)
val e:Either[Throwable,String] = Right("foo")
println(e.toTry)
val e2:Either[Throwable,String] = Left(new RuntimeException("bar"))
println(e2.toTry)
}
In Scala 2.12.x Try has a toEither method: http://www.scala-lang.org/api/2.12.x/scala/util/Try.html#toEither:scala.util.Either[Throwable,T]
import scala.util.{ Either, Failure, Left, Right, Success, Try }
implicit def eitherToTry[A <: Exception, B](either: Either[A, B]): Try[B] = {
either match {
case Right(obj) => Success(obj)
case Left(err) => Failure(err)
}
}
implicit def tryToEither[A](obj: Try[A]): Either[Throwable, A] = {
obj match {
case Success(something) => Right(something)
case Failure(err) => Left(err)
}
}
The answer depends on how to convert the Failure to Left (and vice versa). If you don't need to use the details of the exception, then Try can be converted to Either by going the intermediate route of an Option:
val tried = Try(1 / 0)
val either = tried.toOption.toRight("arithmetic error")
The conversion the other way requires you to construct some Throwable. It could be done like this:
either.fold(left => Failure(new Exception(left)), right => Success(right))
Very often i end up with lots of nested .map and .getOrElse when validating several consecutives conditions
for example:
def save() = CORSAction { request =>
request.body.asJson.map { json =>
json.asOpt[Feature].map { feature =>
MaxEntitiyValidator.checkMaxEntitiesFeature(feature).map { rs =>
feature.save.map { feature =>
Ok(toJson(feature.update).toString)
}.getOrElse {
BadRequest(toJson(
Error(status = BAD_REQUEST, message = "Error creating feature entity")
))
}
}.getOrElse {
BadRequest(toJson(
Error(status = BAD_REQUEST, message = "You have already reached the limit of feature.")
))
}
}.getOrElse {
BadRequest(toJson(
Error(status = BAD_REQUEST, message = "Invalid feature entity")
))
}
}.getOrElse {
BadRequest(toJson(
Error(status = BAD_REQUEST, message = "Expecting JSON data")
))
}
}
You get the idea
I just wanted to know if there's some idiomatic way to keep it more clear
If you hadn't had to return a different message for the None case this would be an ideal use-case for for comprehension. In your case , you probably want to use the Validation monad, as the one you can find in Scalaz. Example ( http://scalaz.github.com/scalaz/scalaz-2.9.0-1-6.0/doc.sxr/scalaz/Validation.scala.html ).
In functional programming, you should not throw exceptions but let functions which can fail return an Either[A,B], where by convention A is the type of result in case of failure and B is the type of result in case of success. You can then match against Left(a) or Right(b) to handle, reespectively, the two cases.
You can think of the Validation monad as an extended Either[A,B] where applying subsequent functions to a Validation will either yield a result, or the first failure in the execution chain.
sealed trait Validation[+E, +A] {
import Scalaz._
def map[B](f: A => B): Validation[E, B] = this match {
case Success(a) => Success(f(a))
case Failure(e) => Failure(e)
}
def foreach[U](f: A => U): Unit = this match {
case Success(a) => f(a)
case Failure(e) =>
}
def flatMap[EE >: E, B](f: A => Validation[EE, B]): Validation[EE, B] = this match {
case Success(a) => f(a)
case Failure(e) => Failure(e)
}
def either : Either[E, A] = this match {
case Success(a) => Right(a)
case Failure(e) => Left(e)
}
def isSuccess : Boolean = this match {
case Success(_) => true
case Failure(_) => false
}
def isFailure : Boolean = !isSuccess
def toOption : Option[A] = this match {
case Success(a) => Some(a)
case Failure(_) => None
}
}
final case class Success[E, A](a: A) extends Validation[E, A]
final case class Failure[E, A](e: E) extends Validation[E, A]
Your code now can be refactored by using the Validation monad into three validation layers. You should basically replace your map with a validation like the following:
def jsonValidation(request:Request):Validation[BadRequest,String] = request.asJson match {
case None => Failure(BadRequest(toJson(
Error(status = BAD_REQUEST, message = "Expecting JSON data")
)
case Some(data) => Success(data)
}
def featureValidation(validatedJson:Validation[BadRequest,String]): Validation[BadRequest,Feature] = {
validatedJson.flatMap {
json=> json.asOpt[Feature] match {
case Some(feature)=> Success(feature)
case None => Failure( BadRequest(toJson(
Error(status = BAD_REQUEST, message = "Invalid feature entity")
)))
}
}
}
And then you chain them like the following featureValidation(jsonValidation(request))
This is a classic example of where using a monad can clean up your code. For example you could use Lift's Box, which is not tied to Lift in any way. Then your code would look something like this:
requestBox.flatMap(asJSON).flatMap(asFeature).flatMap(doSomethingWithFeature)
where asJson is a Function from a request to a Box[JSON] and asFeature is a function from a Feature to some other Box. The box can contain either a value, in which case flatMap calls the function with that value, or it can be an instance of Failure and in that case flatMap does not call the function passed to it.
If you had posted some example code that compiles, I could have posted an answer that compiles.
I tried this to see if pattern matching offered someway to adapt the submitted code sample (in style, if not literally) to something more coherent.
object MyClass {
case class Result(val datum: String)
case class Ok(val _datum: String) extends Result(_datum)
case class BadRequest(_datum: String) extends Result(_datum)
case class A {}
case class B(val a: Option[A])
case class C(val b: Option[B])
case class D(val c: Option[C])
def matcher(op: Option[D]) = {
(op,
op.getOrElse(D(None)).c,
op.getOrElse(D(None)).c.getOrElse(C(None)).b,
op.getOrElse(D(None)).c.getOrElse(C(None)).b.getOrElse(B(None)).a
) match {
case (Some(d), Some(c), Some(b), Some(a)) => Ok("Woo Hoo!")
case (Some(d), Some(c), Some(b), None) => BadRequest("Missing A")
case (Some(d), Some(c), None, None) => BadRequest("Missing B")
case (Some(d), None, None, None) => BadRequest("Missing C")
case (None, None, None, None) => BadRequest("Missing D")
case _ => BadRequest("Egads")
}
}
}
Clearly there are ways to write this more optimally; this is left as an exercise for the reader.
I agree with Edmondo suggestion of using for comprehension but not with the part about using a validation library (At least not anymore given the new features added to scala standard lib since 2012). From my experience with scala, dev that struggle to come up with nice statement with the standard lib will also end up doing the same of even worst when using libs like cats or scalaz. Maybe not at the same place, but ideally we would solve the issue rather than just moving it.
Here is your code rewritten with for comprehension and either that is part of scala standard lib :
def save() = CORSAction { request =>
// Helper to generate the error
def badRequest(message: String) = Error(status = BAD_REQUEST, message)
//Actual validation
val updateEither = for {
json <- request.body.asJson.toRight(badRequest("Expecting JSON data"))
feature <- json.asOpt[Feature].toRight(badRequest("Invalid feature entity"))
rs <- MaxEntitiyValidator
.checkMaxEntitiesFeature(feature)
.toRight(badRequest("You have already reached the limit"))
} yield toJson(feature.update).toString
// Turn the either into an OK/BadRequest
featureEither match {
case Right(update) => Ok(update)
case Left(error) => BadRequest(toJson(error))
}
}
Explanations
Error handling
I'm not sure how much you know about either but they are pretty similar in behaviour as Validation presented by Edmondo or Try object from the scala library. Main difference between those object regard their capability and behaviour with errors, but beside that they all can be mapped and flat mapped the same way.
You can also see that I use toRight to immediately convert the option into Either instead of doing it at the end. I see that java dev have the reflex to throw exception as far as they physically can, but they mostly do so because the try catch mechanism is unwieldy: in case of success, to get data out of a try block you either need to return them or put them in a variable initialized to null out of the block. But this is not the case is scala: you can map a try or an either, so in general, you get a more legible code if you turn results into error representation as soon as have identified it as they are identified as incorrect.
For comprehension
I also know that dev discovering scala are often quite puzzled by for comprehension. This is quite understandable as in most other language, for is only used for iteration over collections while is scala, it seem to use usable on a lot of unrelated types. In scala for is actually more nicer way to call the function flatMap. The compiler may decide to optimize it with map or foreach but it remain correct assume that you will get a flatMap behavior when you use for.
Calling flatMap on a collection will behave like the for each would in other language, so scala for may be used like a standard for when dealing with collection. But you can also use it on any other type of object that provide an implementation for flatMap with the correct signature. If your OK/BadRequest also implement the flatMap, you may be able to use in directly in the for comprehension instead of usong an intermediate Either representation.
For the people are not at ease with using for on anything that do not look like a collection, here is is how the function would look like if explicitly using flatMap instead of for :
def save() = CORSAction { request =>
def badRequest(message: String) = Error(status = BAD_REQUEST, message)
val updateEither = request.body.asJson.toRight(badRequest("Expecting JSON data"))
.flatMap { json =>
json
.asOpt[Feature]
.toRight(badRequest("Invalid feature entity"))
}
.flatMap { feature =>
MaxEntitiyValidator
.checkMaxEntitiesFeature(feature)
.map(_ => feature)
.toRight(badRequest("You have already reached the limit"))
}
.map { rs =>
toJson(feature.update).toString
}
featureEither match {
case Right(update) => Ok(update)
case Left(error) => BadRequest(toJson(error))
}
}
Note that in term of parameter scope, for behave live if the function where nested, not chained.
Conclusion
I think that more than not using the right framework or the right language feature, the main issue with the code your provided is how errors are dealt with. In general, you should not write error paths as after thought that you pile up at the end of the method. If you can deal with the error immediately as they occur, that allow you to move to something else. On the contrary, the more you push them back, the more you will have code with inextricable nesting. They are actually a materialization of all the pending error cases that scala expect you to deal with at some point.