This is a follow-up to my previous question. I copied the example below from Haxl
Suppose I am fetching data from a blog server to render a blog page, which contains the recent posts, popular posts and posts topics.
I have the following Data Fetching API:
val getRecent : Server => Seq[Post] = ...
val getPopular : Server => Seq[Post] = ...
val getTopics : Server => Seq[Topic] = ...
Now I need to compose them to implement a new function getPageData
val getPageData: Server => (Seq[Post], Seq[Post], Seq[Topic])
Haxl suggests using a new monad Fetch to make the API composable.
val getRecent : Fetch[Seq[Posts]] = ...
val getPopular : Fetch[Seq[Posts]] = ...
val getTopics : Fetch[Seq[Topic]] = ...
Now I can define my getPageData: Fetch[A] with monadic composition
val getPageData = for {
recent <- getRecent
popular <- getPopular
topics <- getTopics
} yield (recent, popular, topics)
but it does not run getRecent, getPopular, and getTopics concurrently.
Haxl suggests using applicative composition <*> to compose "concurrent" functions (i.e. the functions that can run concurrently). So my questions are:
How to implement getPageData assuming Fetch[A] is an Applicative ?
How to implement Fetch as an Applicative but not a Monad ?
How to implement getPageData assuming Fetch[A] is an Applicative ?
All we need to do is drop the monadic bind >>= in favour of the applicative <*>. So instead of
val getPageData = for {
recent <- getRecent
popular <- getPopular
topics <- getTopics
} yield (recent, popular, topics)
we would write something like (in Haskell syntax; sorry, I can't do Scala off the top of my head):
getPageData = makeTriple <$> getRecent <*> getPopular <*> getTopics
where
makeTriple x y z = (x, y, z)
But whether this has the desired effect is contingent upon the second question!
How to implement Fetch as an Applicative but not a Monad ?
The key distinction between monadic and applicative sequencing is that the monadic one can depend upon the value inside a monadic value, whereas the applicative <*> cannot. Notice how the monadic expression for getPageData above binds the names recent and popular before reaching getTopics. Those names could have been used to change the structure of the expression, for example by getting some other data source in case recent is empty. But with the applicative expression, the results of getRecent and getPopular are not factors in the structure of the expression itself. This property allows us to fire off each term in the applicative expression concurrently, because we know the structure of the expression statically.
So, using the observation above, and obviously the particular shape of the Fetch datatype, we can come up with a suitable definition for <*>. I think the following illustrates the general idea:
data Fetch a = Fetch { runFetch :: IO a }
fetchF <*> fetchX = Fetch $ do
-- Fire off both IOs concurrently.
resultF <- async $ runFetch fetchF
resultX <- async $ runFetch fetchX
-- Wait for both results to be ready.
f <- wait resultF
x <- wait resultX
return $ f x
For comparison, suppose we tried to do monadic bind with concurrent evaluation:
fetchF >>= fetchK = Fetch $ do
resultF <- async $ runFetch fetchF
-- Oh no, we need resultF in order to produce the next
-- Fetch value! We just have to wait...
f <- wait resultF
fetchX <- async $ runFetch (fetchK f)
x <- wait $ runFetch fetchX
return $ f x
Related
I'm kind of new to Scala/functional so I'm not yet able to use technical language.
I'm experiencing problems with a for-comprehension
val queries =
for {
_ <- createBanco
_ <- createBancoMedio
bankInsertions <- Update[Banco](insertStr).updateMany(NonEmptyList.fromList(createBankList(1, maxBanks)).get)
mediumInsertions <- Update[BancoMedio](mediumInsert).updateMany(NonEmptyList.fromList(mediumList).get)
bankCount <- BancoStatements.getCount().unique
bankGetIds <- BancoStatements.getIds(0, maxBanks).to[List]
bankSome <- BancoStatements.getSome(halfBanks).to[List]
} yield (bankCount, bankGetIds, bankSome)
//Execute database queries, saves them on tuple
val transactionResults : (Int, List[String], List[Banco]) =
queries.transact(h2Transactor).unsafeRunSync()
I'm trying to refactor the _ <- createBanco & _ <- createBancoMedio, which are both a ConnectionIO[Int] object.
Id like to convert those to a single List(createBanco, createBancoMedio) and then execute transact.
However, i'd be altering the return type of the for-comprehension by doing that. I'd like to know if there is any way on doing that without affecting the for output value
Basically, treat the list as if I was writing multiple anonymous parameters manually.
You can use .sequence to turn a List[G[A]] into a G[List[A]] if G has an Applicative instance, which ConnectionIO does:
val queries =
for {
_ <- List(createBanco, createBancoMedio).sequence
...
Just solved it, did another for comprehension for the List
val createList = for {
m <- createBancoMedio
b <- createBanco
} yield List(b, m)
val queries =
for {
_ <- createList ....
This way i had a ConnectionIO[List[Int]]
With Scalaz Task I make this with scalaz.Nondeterminism.both:
Nondeterminism[Task]
.both(
Task.now("Hello"),
Task.now("world")
)
or with Nondeterminism[Task].gatherUnordered().
How can I do the same thing with fs2 0.9.x version tasks?
I'm assuming you're on fs2 version 0.9.x.
To execute several Tasks in parallel, you can simply call Task.start.
Here's an example from the docs:
for {
f <- Task.start { expensiveTask1 }
// at this point, `expensive1` is evaluating in background
g <- Task.start { expensiveTask2 }
// now both `expensiveTask2` and `expensiveTask1` are running
result1 <- f
// we have forced `f`, so now only `expensiveTask2` may be running
result2 <- g
// we have forced `g`, so now nothing is running and we have both results
} yield (result1 + result2)
So in your case it would look like this:
for {
ta <- Task.start(Task.now("Hello"))
tb <- Task.start(Task.now("World"))
a <- ta
b <- tb
} yield (a, b)
Note that in the future it might be possible to do something like this with much less boilerplate. There's a PR in the works to add a Parallel type class, which would allow us to write something like this:
(taskA, taskB).parMapN((a, b) => ...)
Starting playing with Scala futures, I get stuck with dependent futures.
Let's get a example. I search for places and get a Future[Seq[Place]]. For each of theses places, I search for the closest subway stations (the service resurns a Future[List[Station]]).
I would write this:
Place.get()
.map { places =>
places.map { place =>
Station.closestFrom(place).map { stations =>
SearchResult(place, stations)
}
}
}
That thing will make me get a Future[Seq[Future[SearchResult]]]... which is... not what I would have expected.
What did I miss to get a Future[Seq[SearchResult]] ?
Thanks for all,
Alban
You are missing two Future concepts in your solution: flatMap and Future.sequence
To explain each:
flatMap is like map except instead of giving it a function from future.map(A => B) you give it a function from future.flatMap(A => Future[B]). This way you can chain Futures together.
Future.sequence is a helper function that combines a list of futures to a future of a list: Seq[Future[A]] => Future[Seq[A]]
Using these two features of the Future API we can change your answer to be:
Place.get().flatMap { places =>
Future.sequence(places.map { place =>
Station.closestFrom(place).map { stations =>
SearchResult(place, stations)
}
})
}
Short version
Working with futures is generaly easier using for-comprehension rather than directly map/flatMap. In your situation it should look like this:
for {places <- Place.get()
searchResults <- Future.traverse(places)(place => for (stations <- Station.closestFrom(place))
yield SearchResult(place,stations)
)
} yield searchResults
Detailed Version
Future being a monad, it offers you several ways to chain your operations.
If you want to apply an 'regular' function f : A => B to what's inside the box myfuture : Future[A], indeed map is the way to get a Future[B]. But in the present situation Station.closestFrom a does not give you a List[Stattion] but a Future[List[Station]].
If you want to apply a monadic operation h : A => Future[B] or chain several of them (here Places.get and Station.closestFrom), flatMap is the way to go. Apply h to a Future[A] gives you a Future[B].
If you want to apply a monadic operation h : A => Future[B] to a collection like a places : Seq[A], you should use Future.traverse : Seq[A] => (A => Future[B]) => Future[Seq[B]].
Furthermore, Scala's for-compresention is just syntactic sugar for flatMap/map so instead of writing complex code using those directly you can use a clean and clear for loop. The loop:
for { variable1 <- f1
variable2 <- f2
} yield expression
is equivalent to (without optimisations) :
f1.flatMap( variable1 => f2.map(variable2 => expression))
Don't hesitate to use for-comprehension, it really helps.
I have two functions which return Futures. I'm trying to feed a modified result from first function into the other using a for-yield comprehension.
This approach works:
val schoolFuture = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- schoolStore.getSchool(sid.get) if sid.isDefined
} yield s
However I'm not happy with having the "if" in there, it seems that I should be able to use a map instead.
But when I try with a map:
val schoolFuture: Future[Option[School]] = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- sid.map(schoolStore.getSchool(_))
} yield s
I get a compile error:
[error] found : Option[scala.concurrent.Future[Option[School]]]
[error] required: scala.concurrent.Future[Option[School]]
[error] s <- sid.map(schoolStore.getSchool(_))
I've played around with a few variations, but haven't found anything attractive that works. Can anyone suggest a nicer comprehension and/or explain what's wrong with my 2nd example?
Here is a minimal but complete runnable example with Scala 2.10:
import concurrent.{Future, Promise}
case class User(userId: Int)
case class UserDetails(userId: Int, schoolId: Option[Int])
case class School(schoolId: Int, name: String)
trait Error
class UserStore {
def getUserDetails(userId: Int): Future[Either[Error, UserDetails]] = Promise.successful(Right(UserDetails(1, Some(1)))).future
}
class SchoolStore {
def getSchool(schoolId: Int): Future[Option[School]] = Promise.successful(Option(School(1, "Big School"))).future
}
object Demo {
import concurrent.ExecutionContext.Implicits.global
val userStore = new UserStore
val schoolStore = new SchoolStore
val user = User(1)
val schoolFuture: Future[Option[School]] = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- sid.map(schoolStore.getSchool(_))
} yield s
}
(Edited to give a correct answer!)
The key here is that Future and Option don't compose inside for because there aren't the correct flatMap signatures. As a reminder, for desugars like so:
for ( x0 <- c0; w1 = d1; x1 <- c1 if p1; ... ; xN <- cN) yield f
c0.flatMap{ x0 =>
val w1 = d1
c1.filter(x1 => p1).flatMap{ x1 =>
... cN.map(xN => f) ...
}
}
(where any if statement throws a filter into the chain--I've given just one example--and the equals statements just set variables before the next part of the chain). Since you can only flatMap other Futures, every statement c0, c1, ... except the last had better produce a Future.
Now, getUserDetails and getSchool both produce Futures, but sid is an Option, so we can't put it on the right-hand side of a <-. Unfortunately, there's no clean out-of-the-box way to do this. If o is an option, we can
o.map(Future.successful).getOrElse(Future.failed(new Exception))
to turn an Option into an already-completed Future. So
for {
ud <- userStore.getUserDetails(user.userId) // RHS is a Future[Either[...]]
sid = ud.right.toOption.flatMap(_.schoolId) // RHS is an Option[Int]
fid <- sid.map(Future.successful).getOrElse(Future.failed(new Exception)) // RHS is Future[Int]
s <- schoolStore.getSchool(fid)
} yield s
will do the trick. Is that better than what you've got? Doubtful. But if you
implicit class OptionIsFuture[A](val option: Option[A]) extends AnyVal {
def future = option.map(Future.successful).getOrElse(Future.failed(new Exception))
}
then suddenly the for-comprehension looks reasonable again:
for {
ud <- userStore.getUserDetails(user.userId)
sid <- ud.right.toOption.flatMap(_.schoolId).future
s <- schoolStore.getSchool(sid)
} yield s
Is this the best way to write this code? Probably not; it relies upon converting a None into an exception simply because you don't know what else to do at that point. This is hard to work around because of the design decisions of Future; I'd suggest that your original code (which invokes a filter) is at least as good of a way to do it.
This answer to a similar question about Promise[Option[A]] might help. Just substitute Future for Promise.
I'm inferring the following types for getUserDetails and getSchool from your question:
getUserDetails: UserID => Future[Either[??, UserDetails]]
getSchool: SchoolID => Future[Option[School]]
Since you ignore the failure value from the Either, transforming it to an Option instead, you effectively have two values of type A => Future[Option[B]].
Once you've got a Monad instance for Future (there may be one in scalaz, or you could write your own as in the answer I linked), applying the OptionT transformer to your problem would look something like this:
for {
ud <- optionT(getUserDetails(user.userID) map (_.right.toOption))
sid <- optionT(Future.successful(ud.schoolID))
s <- optionT(getSchool(sid))
} yield s
Note that, to keep the types compatible, ud.schoolID is wrapped in an (already completed) Future.
The result of this for-comprehension would have type OptionT[Future, SchoolID]. You can extract a value of type Future[Option[SchoolID]] with the transformer's run method.
What behavior would you like to occur in the case that the Option[School] is None? Would you like the Future to fail? With what kind of exception? Would you like it to never complete? (That sounds like a bad idea).
Anyways, the if clause in a for-expression desugars to a call to the filter method. The contract on Future#filteris thus:
If the current future contains a value which satisfies the predicate,
the new future will also hold that value. Otherwise, the resulting
future will fail with a NoSuchElementException.
But wait:
scala> None.get
java.util.NoSuchElementException: None.get
As you can see, None.get returns the exact same thing.
Thus, getting rid of the if sid.isDefined should work, and this should return a reasonable result:
val schoolFuture = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- schoolStore.getSchool(sid.get)
} yield s
Keep in mind that the result of schoolFuture can be in instance of scala.util.Failure[NoSuchElementException]. But you haven't described what other behavior you'd like.
We've made small wrapper on Future[Option[T]] which acts like one monad (nobody even checked none of monad laws, but there is map, flatMap, foreach, filter and so on) - MaybeLater. It behaves much more than an async option.
There are a lot of smelly code there, but maybe it will be usefull at least as an example.
BTW: there are a lot of open questions(here for ex.)
It's easier to use https://github.com/qifun/stateless-future or https://github.com/scala/async to do A-Normal-Form transform.
If I have some computation that takes a while I might place it in a scala.concurrent.Future:
val f = Future { someLongRunningFunction() }
and let's say I want to do something else asynchronously once that computation is completed:
f.flatMap{ _ => anotherLongRunningFunction() }
In the event that f's initial block fails, how do I "idiomatically" handle this when using flatMap or other combinators? Is it merely a case of using recover or onFailure before the flatMap?
I like the elegance and simplicity of using a flatMap but it seems failure scenarios get in the way of it.
Edit: the second future is reliant on the first, hence the flatMap. I'm looking for a solution that'll elegantly let me chain like I would with flatMap but also handle failures of the first.
To quote the scaladoc for flatMap:
Creates a new future by applying a function to the successful result
of this future, and returns the result of the function as the new
future. If this future is completed with an exception then the new
future will also contain this exception.
Notice the bold, meaning that whatever you pass to flatMap will only be executed if the initial future completes successfully.
So you should only handle the result of the entire execution:
val result = future1.flatMap {
result => functionReturningFuture2(result)
}
and then:
result.onFailure // or
result.onSuccess // or
result.recover
If you have several futures you can put them in a for comprehension.
val futureResult = for {
result1 <- future1
result2 <- future2
...
} yield {
//computation with results
}
You can add a recover at the end in case you want to process any exception you may find:
futureResult.recover{
case exceptionResult: Throwable => // Process exception
}
I think this is more clean that using flatMap.
I'm only starting with Future, but here are some ideas.
If you really want to use flatMap, you have to turn the failure into a success.
for{ a <- f recover r
b <- another(a)
} yield b
This works if the return type of r is :> the result type of f.
Or you can pass the problem of what to do with the failure on to the next process
for{ a <- f map (x => Success(x)) recover (ex => Failure(ex))
b <- another(a)
} yield b
Here the argument type of another would be Try[T] where the type of f is Future[T].