The code sends a request to MongoDB using ReactiveMongo and returns Future[BSONDocument] but my code handles lists of data, so I need to get the value of Future[BSONDocument] and then turn it into a list.
How do I do that preferably without blocking?
Upadte:
I am using ReactiveMongo RawCommand
def findLByDistance()(implicit ec: ExecutionContext) = db.command(RawCommand(
BSONDocument(
"aggregate" -> collName,
"pipeline" -> BSONArray(BSONDocument(
"$geoNear" -> BSONDocument(
"near" -> BSONArray(44.72,25.365),
"distanceField" -> "location.distance",
"maxDistance" -> 0.08,
"uniqueDocs" -> true)))
)))
And the result comes out in Future[BSONDocument]. For some simple queries I used default query builder which allowed for simple conversion
def findLimitedEvents()(implicit ec: ExecutionContext) =
collection.find(BSONDocument.empty)
.query(BSONDocument("tags" -> "lazy"))
.options(QueryOpts().batchSize(10))
.cursor.collect[List](10, true)
I basically I need the RawCommand output type to match previously used.
Not sure about your exact use-case (showing some more code would help), but it might be useful to convert from List[Future[BSONDocument]] to one Future[List[BsonDocument]], which you can then more easily onSuccess or map on, you can do that via:
val futures: List[Future[A]] = List(f1, f2, f3)
val futureOfThose: Future[List[A]] = Future.sequence(futures)
You cannot "get" a future without blocking. if you want to wait for a Future to complete then you must block.
What you can do is map a Future into another Future:
val futureDoc: Future[BSONDocument] = ...
val futureList = futureDoc map { doc => docToList(doc) }
Eventually, you'll hit a point where you've combined, mapped, recovered, etc. all your futures and want something to happen with the result. This is where you either block, or establish a handler to do something with the eventual result:
val futureThing: Future[Thing] = ...
//this next bit will be executed at some later time,
//probably on a different thread
futureThing onSuccess {
case thing => doWhateverWith(thing)
}
Related
I want to log in the event that a record doesn't have an adjoining record. Is there a purely functional way to do this? One that separates the side effect from the data transformation?
Here's an example of what I need to do:
val records: Seq[Record] = Seq(record1, record2, ...)
val accountsMap: Map[Long, Account] = Map(record1.id -> account1, ...)
def withAccount(accountsMap: Map[Long, Account])(r: Record): (Record, Option[Account]) = {
(r, accountsMap.get(r.id))
}
def handleNoAccounts(tuple: (Record, Option[Account]) = {
val (r, a) = tuple
if (a.isEmpty) logger.error(s"no account for ${record.id}")
tuple
}
def toRichAccount(tuple: (Record, Option[Account]) = {
val (r, a) = tuple
a.map(acct => RichAccount(r, acct))
}
records
.map(withAccount(accountsMap))
.map(handleNoAccounts) // if no account is found, log
.flatMap(toRichAccount)
So there are multiple issues with this approach that I think make it less than optimal.
The tuple return type is clumsy. I have to destructure the tuple in both of the latter two functions.
The logging function has to handle the logging and then return the tuple with no changes. It feels weird that this is passed to .map even though no transformation is taking place -- maybe there is a better way to get this side effect.
Is there a functional way to clean this up?
I could be wrong (I often am) but I think this does everything that's required.
records
.flatMap(r =>
accountsMap.get(r.id).fold{
logger.error(s"no account for ${r.id}")
Option.empty[RichAccount]
}{a => Some(RichAccount(r,a))})
If you're using scala 2.13 or newer you could use tapEach, which takes function A => Unit to apply side effect on every element of function and then passes collection unchanged:
//you no longer need to return tuple in side-effecting function
def handleNoAccounts(tuple: (Record, Option[Account]): Unit = {
val (r, a) = tuple
if (a.isEmpty) logger.error(s"no account for ${record.id}")
}
records
.map(withAccount(accountsMap))
.tapEach(handleNoAccounts) // if no account is found, log
.flatMap(toRichAccount)
In case you're using older Scala, you could provide extension method (updated according to Levi's Ramsey suggestion):
implicit class SeqOps[A](s: Seq[A]) {
def tapEach(f: A => Unit): Seq[A] = {
s.foreach(f)
s
}
}
I have a mutable map that contains users:
val userMap = mutable.Map.empty[Int, User] // Int is user.Id
Now I need to load the new users, and add them to the map. I have the following api methods:
def getNewUsers(): Seq[Int]
def getUser(userId: Int): Future[Option[User]]
So I first get all the users I need to load:
val newUserIds: Set[Int] = api.getNewUsers
I now need to load each user, but not sure how to do getUser returns a Future[Option[User]].
I tried this:
api.getNewUsers().map( getUser(_) )
But that returns a Set[Future[Option[User]]]
I'm not sure how to use Set[Future[Option[User]]] to update my userMap now.
You'll have to wait for all of the Futures to finish. You can use Future.sequence to transform your Set[Future[_]] into a Future[Set], so you can wait for them all to finish:
val s: Set[scala.concurrent.Future[Some[User]]] = Set(Future(Some(User(1))), Future(Some(User(2))))
val f: Future[Set[Some[User]]] = Future.sequence(s)
f.map(users => users.foreach(u => /* your code here */))
However, using a mutable Map may be dangerous because it's possible to open yourself up to race conditions. Futures are executed in different threads, and if you altering a mutable object's state in different threads, bad things will happen.
You can use Future.sequence:
transforms a TraversableOnce[Future[A]] into a
Future[TraversableOnce[A]]. Useful for reducing many Futures into a
single Future
from Scala Future
You can try:
val result : Future[Seq[Option[User]]] =
Future.sequence(
api.getNewUsers().map( getUser )
)
result.andThen {
case Success(users) =>
users.flatten.foreach(u => yourMap += u.id -> u)
}
How can I remove the option so it is just Try[Int] and not Try[Option[Int]]?
val m = Map("a" -> "1a", "b" -> "2")
Try(m.get("a").map(_.trim.toInt))
>>es17: scala.util.Try[Option[Int]] = Failure(java.lang.NumberFormatException: For input string: "1a")
Map#get returns an Option[String], but you can use Map#apply instead, which will return String, in this case.
scala> Try(m("a").trim.toInt)
res3: scala.util.Try[Int] = Failure(java.lang.NumberFormatException: For input string: "1a")
scala> Try(m("b").trim.toInt)
res4: scala.util.Try[Int] = Success(2)
apply throws an exception if the key you're looking for doesn't exist, but Try will catch it, anyway.
This answer goes in more detail about the comment:
I was wondering if there was a way to use flapmap? Your solution works for me, just want to learn of other alternatives.
As you've probably heard, Option and Try are monad instances and while monads are handy to represent sequence of computations, they don't compose with other monads. In other words, we can't compose Option and Try. We need to find a common ground.
The difference in semantics between Option and Try is that Try contains information about the case when a result is absent.
We can go from Try to Option using Try#toOption effectively loosing any failure information we may have.
If we wanted to go the other way, we need to add this information back: ne need to provide a failure reason when a value is absent in an Option. Something like this:
import scala.util.{Try, Success, Failure}
def optionToTry[T](opt:Option[T], failure: => Throwable): Try[T] = opt match {
case Some(v) => Success(v)
case None => Failure(failure)
}
With the help of that function, we can rewrite the original expression as:
val res: Try[Int] = for {
strValue <- optionToTry(m.get("a"), new NoSuchElementException("a"))
value <- Try(strValue.trim.toInt)
} yield value
which uses flatMap behind the scenes to compose the two Try instances like this:
val res = optionToTry(m.get("a"), new NoSuchElementException("a"))
.flatMap(strValue => Try(strValue.trim.toInt))
Note that we could save ourselves a bit of coding by using the unsafe map getter like so:
val res: Try[Int] = for {
strValue <- Try(m("a"))
value <- Try(strValue.trim.toInt)
} yield value
but this version would be computationally more expensive given the cost of handling exceptions in the JVM.
Let's say I have some iterator:
val nextElemIter: Iterator[Future[Int]] = Iterator.continually(...)
And I want to build a source from that iterator:
val source: Source[Future[Int], NotUsed] =
Source.fromIterator(() => nextElemIter)
So now my source emits Futures. I have never seen futures being passed between stages in Akka docs or anywhere else, so instead, I could always do something like this:
val source: Source[Int, NotUsed] =
Source.fromIterator(() => nextElemIter).mapAsync(1)(identity /* was n => n */)
And now I have a regular source that emits T instead of Future[T]. But this feels hacky and wrong.
What's the proper way to deal with such situations?
Answering your question directly: I agree with Vladimir's comment that there is nothing "hacky" about using mapAsync for the purpose you described. I can't think of any more direct way to unwrap the Future from around your underlying Int values.
Answering your question indirectly...
Try to stick with Futures
Streams, as a concurrency mechanism, are incredibly useful when backpressure is required. However, pure Future operations have their place in applications as well.
If your Iterator[Future[Int]] is going to produce a known, limited, number of Future values then you may want to stick with using the Futures for concurrency.
Imagine you want to filter, map, & reduce the Int values.
def isGoodInt(i : Int) : Boolean = ??? //filter
def transformInt(i : Int) : Int = ??? //map
def combineInts(i : Int, j : Int) : Int = ??? //reduce
Futures provide a direct way of using these functions:
val finalVal : Future[Int] =
Future sequence {
for {
f <- nextElemIter.toSeq //launch all of the Futures
i <- f if isGoodInt(i)
} yield transformInt(i)
} map (_ reduce combineInts)
Compared with a somewhat indirect way of using the Stream as you suggested:
val finalVal : Future[Int] =
Source.fromIterator(() => nextElemIter)
.via(Flow[Future[Int]].mapAsync(1)(identity))
.via(Flow[Int].filter(isGoodInt))
.via(Flow[Int].map(transformInt))
.to(Sink.reduce(combineInts))
.run()
I have to get a list of issues for each file of a given list from a REST API with Scala. I want to do the requests in parallel, and use the Dispatch library for this. My method is called from a Java framework and I have to wait at the end of this method for the result of all the futures to yield the overall result back to the framework. Here's my code:
def fetchResourceAsJson(filePath: String): dispatch.Future[json4s.JValue]
def extractLookupId(json: org.json4s.JValue): Option[String]
def findLookupId(filePath: String): Future[Option[String]] =
for (json <- fetchResourceAsJson(filePath))
yield extractLookupId(json)
def searchIssuesJson(lookupId: String): Future[json4s.JValue]
def extractIssues(json: org.json4s.JValue): Seq[Issue]
def findIssues(lookupId: String): Future[Seq[Issue]] =
for (json <- searchIssuesJson(componentId))
yield extractIssues(json)
def getFilePathsToProcess: List[String]
def thisIsCalledByJavaFramework(): java.util.Map[String, java.util.List[Issue]] = {
val finalResultPromise = Promise[Map[String, Seq[Issue]]]()
// (1) inferred type of issuesByFile not as expected, cannot get
// the type system happy, would like to have Seq[Future[(String, Seq[Issue])]]
val issuesByFile = getFilePathsToProcess map { f =>
findLookupId(f).flatMap { lookupId =>
(f, findIssues(lookupId)) // I want to yield a tuple (String, Seq[Issue]) here
}
}
Future.sequence(issuesByFile) onComplete {
case Success(x) => finalResultPromise.success(x) // (2) how to return x here?
case Failure(x) => // (3) how to return null from here?
}
//TODO transform finalResultPromise to Java Map
}
This code snippet has several issues. First, I'm not getting the type I would expect for issuesByFile (1). I would like to just ignore the result of findLookUpId if it is not able to find the lookUp ID (i.e., None). I've read in various tutorials that Future[Option[X]] is not easy to handle in function compositions and for expressions in Scala. So I'm also curious what the best practices are to handle these properly.
Second, I somehow have to wait for all futures to finish, but don't know how to return the result to the calling Java framework (2). Can I use a promise here to achieve this? If yes, how can I do it?
And last but not least, in case of any errors, I would just like to return null from thisIsCalledByJavaFramework but don't know how (3).
Any help is much appreciated.
Thanks,
Michael
Several points:
The first problem at (1) is that you don't handle the case where findLookupId returns None. You need to decide what to do in this case. Fail the whole process? Exclude that file from the list?
The second problem at (1) is that findIssues will itself return a Future, which you need to map before you can build the result tuple
There's a shortcut for map and then Future.sequence: Future.traverse
If you cannot change the result type of the method because the Java interface is fixed and cannot be changed to support Futures itself you must wait for the Future to be completed. Use Await.ready or Await.result to do that.
Taking all that into account and choosing to ignore files for which no id could be found results in this code:
// `None` in an entry for a file means that no id could be found
def entryForFile(file: String): Future[(String, Option[Seq[Issue]])] =
findLookupId(file).flatMap {
// the need for this kind of pattern match shows
// the difficulty of working with `Future[Option[T]]`
case Some(id) ⇒ findIssues(id).map(issues ⇒ file -> Some(issues))
case None ⇒ Future.successful(file -> None)
}
def thisIsCalledByJavaFramework(): java.util.Map[String, java.util.List[Issue]] = {
val issuesByFile: Future[Seq[(String, Option[Seq[Issue]])]] =
Future.traverse(getFilePathsToProcess)(entryForFile)
import scala.collection.JavaConverters._
try
Await.result(issuesByFile, 10.seconds)
.collect {
// here we choose to ignore entries where no id could be found
case (f, Some(issues)) ⇒ f -> issues
}
.toMap.mapValues(_.asJava).asJava
catch {
case NonFatal(_) ⇒ null
}
}