Combining List, Future and Option in for-comprehension - scalaz - scala

I've got following problem:
val sth: Future[Seq[T, S]] = for {
x <- whatever: Future[List[T]]
y <- x: List[T]
z <- f(y): Future[Option[S]]
n <- z: Option[S]
} yield y: T -> n: S
I would want to make this code to work(I guess everyone understands the idea as I've added types).
By "to work" I mean, that I would want to stay with the for-comprehension structure and fulfil expected types in the end. I know there are "ugly" ways to do it, but I want to learn how to do it pure :)
As I read the internet I've reached the conclusion that my problem may be solved by the monad transformers & scalaz. Unfortunately, I couldn't find an example to help understand better how should I proceed.
At the moment I've tried scalaz and Eff monad libs, but I guess I still don't understand how it works because I couldn't solve my problem.
I will be grateful for any help.
EDIT: It supposed to be future of sequence, also regarding the "whatever" I get it as a parameter of the function, sorry for misleading you

You could do something like what you need using the scalaz ListT monad transformer
object Test {
import scalaz._
import ListT._
type T = String
type S = Int
val whatever: Future[List[T]] = ??? // you get this somewhere
def f(y: T): Future[Option[S]] = ??? // function that returns future of option
val sth: Future[List[(T, S)]] = (for {
y <- listT(whatever)
// you cannot mix list and option, but you can convert the option to a list of 1 item
n <- listT(f(y).map(_.toList))
} yield y -> n).run
}
N.B.: Since you start with a future, you cannot return a Seq[(T,S)], you can only have a future. You have to call Await.result if you want to block and get the result.

The problem with for comprehension is that it's not some kind of magic monadic "unwrapper", it's just a sequence of map, flatMap and filter.
As you may know map and flatMap operate only on "inner" type, leaving "outer" type of monad unchanged. This means you can't do this:
for {
x <- whatever: Future[List[T]]
y <- x: List[T]
} yield y
inside single for. Instead, you can do something like this:
for (x <- whatever: Future[List[T]])
yield for (y <- x: List[T]) yield y
Which looks kinda ugly.
Back to your case, I's easier to write whole transformation explicitly using map and flatMap, as it gives you greater visibility and control:
whatever.flatMap {
x: List[T] =>
Future.sequence(x.map {
y: T => f(y).map(y -> _)
}).map(_.collect {
case (y, Some(n)) => y -> n
})
}
Also, #trustnoone mentioned, you can't get rid of the Future without explicitly calling Await.

Related

Safer way to extract future result [duplicate]

I have two functions which return Futures. I'm trying to feed a modified result from first function into the other using a for-yield comprehension.
This approach works:
val schoolFuture = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- schoolStore.getSchool(sid.get) if sid.isDefined
} yield s
However I'm not happy with having the "if" in there, it seems that I should be able to use a map instead.
But when I try with a map:
val schoolFuture: Future[Option[School]] = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- sid.map(schoolStore.getSchool(_))
} yield s
I get a compile error:
[error] found : Option[scala.concurrent.Future[Option[School]]]
[error] required: scala.concurrent.Future[Option[School]]
[error] s <- sid.map(schoolStore.getSchool(_))
I've played around with a few variations, but haven't found anything attractive that works. Can anyone suggest a nicer comprehension and/or explain what's wrong with my 2nd example?
Here is a minimal but complete runnable example with Scala 2.10:
import concurrent.{Future, Promise}
case class User(userId: Int)
case class UserDetails(userId: Int, schoolId: Option[Int])
case class School(schoolId: Int, name: String)
trait Error
class UserStore {
def getUserDetails(userId: Int): Future[Either[Error, UserDetails]] = Promise.successful(Right(UserDetails(1, Some(1)))).future
}
class SchoolStore {
def getSchool(schoolId: Int): Future[Option[School]] = Promise.successful(Option(School(1, "Big School"))).future
}
object Demo {
import concurrent.ExecutionContext.Implicits.global
val userStore = new UserStore
val schoolStore = new SchoolStore
val user = User(1)
val schoolFuture: Future[Option[School]] = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- sid.map(schoolStore.getSchool(_))
} yield s
}
(Edited to give a correct answer!)
The key here is that Future and Option don't compose inside for because there aren't the correct flatMap signatures. As a reminder, for desugars like so:
for ( x0 <- c0; w1 = d1; x1 <- c1 if p1; ... ; xN <- cN) yield f
c0.flatMap{ x0 =>
val w1 = d1
c1.filter(x1 => p1).flatMap{ x1 =>
... cN.map(xN => f) ...
}
}
(where any if statement throws a filter into the chain--I've given just one example--and the equals statements just set variables before the next part of the chain). Since you can only flatMap other Futures, every statement c0, c1, ... except the last had better produce a Future.
Now, getUserDetails and getSchool both produce Futures, but sid is an Option, so we can't put it on the right-hand side of a <-. Unfortunately, there's no clean out-of-the-box way to do this. If o is an option, we can
o.map(Future.successful).getOrElse(Future.failed(new Exception))
to turn an Option into an already-completed Future. So
for {
ud <- userStore.getUserDetails(user.userId) // RHS is a Future[Either[...]]
sid = ud.right.toOption.flatMap(_.schoolId) // RHS is an Option[Int]
fid <- sid.map(Future.successful).getOrElse(Future.failed(new Exception)) // RHS is Future[Int]
s <- schoolStore.getSchool(fid)
} yield s
will do the trick. Is that better than what you've got? Doubtful. But if you
implicit class OptionIsFuture[A](val option: Option[A]) extends AnyVal {
def future = option.map(Future.successful).getOrElse(Future.failed(new Exception))
}
then suddenly the for-comprehension looks reasonable again:
for {
ud <- userStore.getUserDetails(user.userId)
sid <- ud.right.toOption.flatMap(_.schoolId).future
s <- schoolStore.getSchool(sid)
} yield s
Is this the best way to write this code? Probably not; it relies upon converting a None into an exception simply because you don't know what else to do at that point. This is hard to work around because of the design decisions of Future; I'd suggest that your original code (which invokes a filter) is at least as good of a way to do it.
This answer to a similar question about Promise[Option[A]] might help. Just substitute Future for Promise.
I'm inferring the following types for getUserDetails and getSchool from your question:
getUserDetails: UserID => Future[Either[??, UserDetails]]
getSchool: SchoolID => Future[Option[School]]
Since you ignore the failure value from the Either, transforming it to an Option instead, you effectively have two values of type A => Future[Option[B]].
Once you've got a Monad instance for Future (there may be one in scalaz, or you could write your own as in the answer I linked), applying the OptionT transformer to your problem would look something like this:
for {
ud <- optionT(getUserDetails(user.userID) map (_.right.toOption))
sid <- optionT(Future.successful(ud.schoolID))
s <- optionT(getSchool(sid))
} yield s
Note that, to keep the types compatible, ud.schoolID is wrapped in an (already completed) Future.
The result of this for-comprehension would have type OptionT[Future, SchoolID]. You can extract a value of type Future[Option[SchoolID]] with the transformer's run method.
What behavior would you like to occur in the case that the Option[School] is None? Would you like the Future to fail? With what kind of exception? Would you like it to never complete? (That sounds like a bad idea).
Anyways, the if clause in a for-expression desugars to a call to the filter method. The contract on Future#filteris thus:
If the current future contains a value which satisfies the predicate,
the new future will also hold that value. Otherwise, the resulting
future will fail with a NoSuchElementException.
But wait:
scala> None.get
java.util.NoSuchElementException: None.get
As you can see, None.get returns the exact same thing.
Thus, getting rid of the if sid.isDefined should work, and this should return a reasonable result:
val schoolFuture = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- schoolStore.getSchool(sid.get)
} yield s
Keep in mind that the result of schoolFuture can be in instance of scala.util.Failure[NoSuchElementException]. But you haven't described what other behavior you'd like.
We've made small wrapper on Future[Option[T]] which acts like one monad (nobody even checked none of monad laws, but there is map, flatMap, foreach, filter and so on) - MaybeLater. It behaves much more than an async option.
There are a lot of smelly code there, but maybe it will be usefull at least as an example.
BTW: there are a lot of open questions(here for ex.)
It's easier to use https://github.com/qifun/stateless-future or https://github.com/scala/async to do A-Normal-Form transform.

Scala, a cross between a foldLeft and a map supporting lazy evaluation

I have a collection which I want to map to a new collection, however each resulting value is dependent on the value before it in some way.I could solve this with a leftFold
val result:List[B] = (myList:List[A]).foldLeft(C -> List.empty[B]){
case ((c, list), a) =>
..some function returning something like..
C -> (B :: list)
}
The problem here is I need to iterate through the entire list to retrieve the resultant list. Say I wanted a function that maps TraversableOnce[A] to TraversableOnce[B] and only evaluate members as I call them?
It seems to me to be a fairly conventional problem so Im wondering if there is a common approach to this. What I currently have is:
implicit class TraversableOnceEx[T](val self : TraversableOnce[T]) extends AnyVal {
def foldyMappyFunction[A, U](a:A)(func:(A,T) => (A,U)):TraversableOnce[U] = {
var currentA = a
self.map { t =>
val result = func(currentA, t)
currentA = result._1
result._2
}
}
}
As far as functional purity goes, you couldn't run it in parallel, but otherwise it seems sound.
An example would be;
Return me each element and if it is the first time that element has appeared before.
val elements:TraversableOnce[E]
val result = elements.mappyFoldyFunction(Set.empty[E]) {
(s, e) => (s + e) -> (e -> s.contains(e))
}
result:TraversableOnce[(E,Boolean)]
You might be able to make use of the State Monad. Here is your example re-written using scalaz:
import scalaz._, Scalaz._
def foldyMappy(i: Int) = State[Set[Int], (Int, Boolean)](s => (s + i, (i, s contains(i))))
val r = List(1, 2, 3, 3, 6).traverseS(foldyMappy)(Set.empty[Int])._2
//List((1,false), (2,false), (3,false), (3,true), (6,false))
println(r)
It is look like you need SeqView. Use view or view(from: Int, until: Int) methods for create a non-strict view of list.
I really don't understand your example as your contains check will always result to false.
foldLeft is different. It will result in a single value by aggregating all elements of the list.
You clearly need map (List => List).
Anyway, answering your question about laziness:
you should use Stream instead of List. Stream doesn't evaluate the tail before actually calling it.
Stream API

Future[Option] in Scala for-comprehensions

I have two functions which return Futures. I'm trying to feed a modified result from first function into the other using a for-yield comprehension.
This approach works:
val schoolFuture = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- schoolStore.getSchool(sid.get) if sid.isDefined
} yield s
However I'm not happy with having the "if" in there, it seems that I should be able to use a map instead.
But when I try with a map:
val schoolFuture: Future[Option[School]] = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- sid.map(schoolStore.getSchool(_))
} yield s
I get a compile error:
[error] found : Option[scala.concurrent.Future[Option[School]]]
[error] required: scala.concurrent.Future[Option[School]]
[error] s <- sid.map(schoolStore.getSchool(_))
I've played around with a few variations, but haven't found anything attractive that works. Can anyone suggest a nicer comprehension and/or explain what's wrong with my 2nd example?
Here is a minimal but complete runnable example with Scala 2.10:
import concurrent.{Future, Promise}
case class User(userId: Int)
case class UserDetails(userId: Int, schoolId: Option[Int])
case class School(schoolId: Int, name: String)
trait Error
class UserStore {
def getUserDetails(userId: Int): Future[Either[Error, UserDetails]] = Promise.successful(Right(UserDetails(1, Some(1)))).future
}
class SchoolStore {
def getSchool(schoolId: Int): Future[Option[School]] = Promise.successful(Option(School(1, "Big School"))).future
}
object Demo {
import concurrent.ExecutionContext.Implicits.global
val userStore = new UserStore
val schoolStore = new SchoolStore
val user = User(1)
val schoolFuture: Future[Option[School]] = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- sid.map(schoolStore.getSchool(_))
} yield s
}
(Edited to give a correct answer!)
The key here is that Future and Option don't compose inside for because there aren't the correct flatMap signatures. As a reminder, for desugars like so:
for ( x0 <- c0; w1 = d1; x1 <- c1 if p1; ... ; xN <- cN) yield f
c0.flatMap{ x0 =>
val w1 = d1
c1.filter(x1 => p1).flatMap{ x1 =>
... cN.map(xN => f) ...
}
}
(where any if statement throws a filter into the chain--I've given just one example--and the equals statements just set variables before the next part of the chain). Since you can only flatMap other Futures, every statement c0, c1, ... except the last had better produce a Future.
Now, getUserDetails and getSchool both produce Futures, but sid is an Option, so we can't put it on the right-hand side of a <-. Unfortunately, there's no clean out-of-the-box way to do this. If o is an option, we can
o.map(Future.successful).getOrElse(Future.failed(new Exception))
to turn an Option into an already-completed Future. So
for {
ud <- userStore.getUserDetails(user.userId) // RHS is a Future[Either[...]]
sid = ud.right.toOption.flatMap(_.schoolId) // RHS is an Option[Int]
fid <- sid.map(Future.successful).getOrElse(Future.failed(new Exception)) // RHS is Future[Int]
s <- schoolStore.getSchool(fid)
} yield s
will do the trick. Is that better than what you've got? Doubtful. But if you
implicit class OptionIsFuture[A](val option: Option[A]) extends AnyVal {
def future = option.map(Future.successful).getOrElse(Future.failed(new Exception))
}
then suddenly the for-comprehension looks reasonable again:
for {
ud <- userStore.getUserDetails(user.userId)
sid <- ud.right.toOption.flatMap(_.schoolId).future
s <- schoolStore.getSchool(sid)
} yield s
Is this the best way to write this code? Probably not; it relies upon converting a None into an exception simply because you don't know what else to do at that point. This is hard to work around because of the design decisions of Future; I'd suggest that your original code (which invokes a filter) is at least as good of a way to do it.
This answer to a similar question about Promise[Option[A]] might help. Just substitute Future for Promise.
I'm inferring the following types for getUserDetails and getSchool from your question:
getUserDetails: UserID => Future[Either[??, UserDetails]]
getSchool: SchoolID => Future[Option[School]]
Since you ignore the failure value from the Either, transforming it to an Option instead, you effectively have two values of type A => Future[Option[B]].
Once you've got a Monad instance for Future (there may be one in scalaz, or you could write your own as in the answer I linked), applying the OptionT transformer to your problem would look something like this:
for {
ud <- optionT(getUserDetails(user.userID) map (_.right.toOption))
sid <- optionT(Future.successful(ud.schoolID))
s <- optionT(getSchool(sid))
} yield s
Note that, to keep the types compatible, ud.schoolID is wrapped in an (already completed) Future.
The result of this for-comprehension would have type OptionT[Future, SchoolID]. You can extract a value of type Future[Option[SchoolID]] with the transformer's run method.
What behavior would you like to occur in the case that the Option[School] is None? Would you like the Future to fail? With what kind of exception? Would you like it to never complete? (That sounds like a bad idea).
Anyways, the if clause in a for-expression desugars to a call to the filter method. The contract on Future#filteris thus:
If the current future contains a value which satisfies the predicate,
the new future will also hold that value. Otherwise, the resulting
future will fail with a NoSuchElementException.
But wait:
scala> None.get
java.util.NoSuchElementException: None.get
As you can see, None.get returns the exact same thing.
Thus, getting rid of the if sid.isDefined should work, and this should return a reasonable result:
val schoolFuture = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- schoolStore.getSchool(sid.get)
} yield s
Keep in mind that the result of schoolFuture can be in instance of scala.util.Failure[NoSuchElementException]. But you haven't described what other behavior you'd like.
We've made small wrapper on Future[Option[T]] which acts like one monad (nobody even checked none of monad laws, but there is map, flatMap, foreach, filter and so on) - MaybeLater. It behaves much more than an async option.
There are a lot of smelly code there, but maybe it will be usefull at least as an example.
BTW: there are a lot of open questions(here for ex.)
It's easier to use https://github.com/qifun/stateless-future or https://github.com/scala/async to do A-Normal-Form transform.

Handling fail-fast failures when the return type is Option[Error]

I have posted quite several questions about failure handling in Scala and I really thank you all for your answers.
I understand my options when dealing with Either and Scalaz or a for comprehension, and I have another (last?) question:
How to do a fail-fast sequence of operations when the operations are dealing with the outside non-functional world, like a DB?
I mean I have a method like that:
def insertItem(item: Item): Either[Error,Item]
Thanks to Either and these answers, I know how to do it with Either: Chaining method calls with Either and
Method parameters validation in Scala, with for comprehension and monads
But my Item case class is immutable and it doesn't really make sense to return it as a Right since the caller already has the value.
Thus how can I do the same kind of thing with:
def insertItem(item: Item): Option[Error]
In my application, when an user is created, we also create some items for him.
If an item fails to create, then the whole process should stop.
When I use directly Option[Error] in a for comprehension, I don't think I'll get the result I expect.
I guess it makes sense to do something like that:
for {
_ <- insertItem(item1).toLeft("???").right
_ <- insertItem(item2).toLeft("???").right
_ <- insertItem(item3).toLeft("???").right
}
But as the values "???" I put in my Right are never useful, I guess I'm missing the elegant solution which do not involve creating Rights that will never be used.
I think I'm looking for something that will continue the for comprehension only when the result is None, which is kind of weird because I just want to continue to the next operation, and not do a real map operation.
By the way, if possible, I would like both non-Scalaz and Scalaz solutions.
I'm not sure Scalaz handle such things, because it seems more focused on real functional programming and perhaps do not provide code for side-effect behaviors like my use case?
I don't see a principled reason not to use Either[Error, Unit] in a case like this (or at least I've done it, and I don't feel guilty about it). Say we have the following:
def launch(thing: String): Either[String, Unit] = Either.cond(
thing.nonEmpty,
println("Launching the " + thing),
"Need something to launch!"
)
We can show that the right projection monad is appropriately lazy:
scala> for {
| _ <- launch("flowers").right
| _ <- launch("").right
| r <- launch("MISSILES").right
| } yield r
Launching the flowers
res1: Either[String,Unit] = Left(Need something to launch!)
No missiles get launched, as desired.
It's worth noting that if you use Option instead of Either, the operation you're describing is just the sum given the "First" monoid instance for Option (where the addition operation is just orElse). For example, we can write the following with Scalaz 7:
import scalaz._, Scalaz._
def fst[A](x: Option[A]): Option[A] ## Tags.First = Tag(x)
def launch(thing: String): Option[String] = if (thing.isEmpty) Some(
"Need something to launch!"
) else {
println("Launching the " + thing)
None
}
And now:
scala> val r: Option[String] = fst(launch("flowers")) |+| fst(
| launch("")) |+| fst(launch("MISSILES"))
Launching the flowers
r: Option[String] = Some(Need something to launch!)
Once again, no missiles.
We do something similar where I work with scalaz to fail fast with message sending, though this is not idiomatic scalaz:
def insertItem(item: Item): Validation[Error, Unit]
val result = for {
_ <- insertItem(item1)
_ <- insertItem(item2)
_ <- insertItem(item3)
} yield Unit
If you want to chain together methods that contain Option[Error] and only execute the next step if the Option is None, you can use orElse for that.
If you are only interested in the first failure, than an Iterator can be your choice:
case class Item(i: Int)
case class Error(i: Item)
val items = Iterator(1,3,7,-5,-2,0) map Item
def insertItem(item: Item): Option[Error] =
if (item.i < 0) Some(Error(item)) else None
scala> val error = (items map insertItem find (_.isDefined)).flatten
error: Option[Error] = Some(Error(Item(-5)))
If insertItem is not your only method it is possible to call them separated with your values:
val items = Iterator(
() => insertItem(i1),
() => deleteItem(i2),
() => insertItem(i3),
() => updateItem(i4))

for..else for Option types in Scala?

Suppose I have two Options and, if both are Some, execute one code path, and if note, execute another. I'd like to do something like
for (x <- xMaybe; y <- yMaybe) {
// do something
}
else {
// either x or y were None, handle this
}
Outside of if statements or pattern matching (which might not scale if I had more than two options), is there a better way of handling this?
Very close to your syntax proposal by using yield to wrap the for output in an Option:
val result = {
for (x <- xMaybe; y <- yMaybe) yield {
// do something
}
} getOrElse {
// either x or y were None, handle this
}
The getOrElse block is executed only if one or both options are None.
You could pattern match both Options at the same time:
(xMaybe, yMaybe) match {
case (Some(x), Some(y)) => "x and y are there"
case _ => "x and/or y were None"
}
The traverse function in Scalaz generalises your problem here. It takes two arguments:
T[F[A]]
A => F[B]
and returns F[T[B]]. The T is any traversable data structure such as List and the F is any applicative functor such as Option. Therefore, to specialise, your desired function has this type:
List[Option[A]] => (A => Option[B]) => Option[List[B]]
So put all your Option values in a List
val z = List(xMaybe, yMaybe)
Construct the function got however you want to collection the results:
val f: X => Option[Y] = ...
and call traverse
val r = z traverse f
This programming patterns occurs very often. It has a paper that talks all about it, The Essence of the Iterator Pattern.
note: I just wanted to fix the URL but the CLEVER edit help tells me I need to change at least 6 characters so I include this useful link too (scala examples):
http://etorreborre.blogspot.com/2011/06/essence-of-iterator-pattern.html
Why would something like this not work?
val opts = List[Option[Int]](Some(1), None, Some(2))
if (opts contains None) {
// Has a None
} else {
// Launch the missiles
val values = opts.map(_.get) // We know that there is no None in the list so get will not throw
}
If you don't know the number of values you are dealing with, then Tony's answer is the best. If you do know the number of values you are dealing with then I would suggest using an applicative functor.
((xMaybe |#| yMaybe) { (x, y) => /* do something */ }).getOrElse(/* something else */)
You said you want the solution to be scalable:
val optional = List(Some(4), Some(3), None)
if(optional forall {_.isDefined}) {
//All defined
} else {
//At least one not defined
}
EDIT: Just saw that Emil Ivanov's solution is a bit more elegant.
Starting Scala 2.13, we can alternatively use Option#zip which concatenates two options to Some tuple of their values if both options are defined or else None:
opt1 zip opt2 match {
case Some((x, y)) => "x and y are there"
case None => "x and/or y were None"
}
Or with Option#fold:
(opt1 zip opt2).fold("x and/or y were None"){ case (x, y) => "x and y are there" }
For scaling to many options, try something along these lines:
def runIfAllSome[A](func:(A)=>Unit, opts:Option[A]*) = {
if(opts.find((o)=>o==None) == None) for(opt<-opts) func(opt.get)
}
With this, you can do:
scala> def fun(i:Int) = println(i)
fun: (i: Int)Unit
scala> runIfAllSome(fun, Some(1), Some(2))
1
2
scala> runIfAllSome(fun, None, Some(1))
scala>
I think the key point here is to think in term of types as what you want to do. As I understand it you want to iterate over a list of Option pairs and then do something based on a certain condition. So the interesting bit of your question would be , what would the return type look like you would except? I think it would look something like this: Either[List[Option], List [Option,Option]] . on the error side (left) you would accumulate the option which was paired with a None (and was left alone so to speak) . On the right side you sum the non empty options which represent your successful values. So we would just need a function which does exactly that. Validate each pair and accumulate it according to it's result( success - failure) . I hope this helps , if not please explain in more detail your usecase. Some links to implement what I described : http://applicative-errors-scala.googlecode.com/svn/artifacts/0.6/pdf/index.pdf and : http://blog.tmorris.net/automated-validation-with-applicatives-and-semigroups-for-sanjiv/