I have a set of operations that are completed in sequence, but if an intermediate sequence returns "null" I would like to abort the operation early (skip the subsequent steps).
I conjured up a function like this which given an input parameter, performs several operations against Redis and will return a product if it exists. Since it is possible that one of the intermediate requests returns a null value, the complete operation could "fail" and I would like to short circuit the unnecessary steps that come afterwards.
The nesting here is becoming crazy, and I would like to make it more legible. Is there a proper "functional" way to perform this type of "if/else" short circuiting?
def getSingleProduct(firstSku: String): Option[Product] = {
val jedis = pool.getResource
val sid: Array[Byte] = jedis.get(Keys.sidForSku(firstSku, sectionId, feedId).getBytes)
Option(sid).flatMap {
sid: Array[Byte] =>
Option(jedis.get(Keys.latestVersionForSid(sectionId, feedId, sid))) flatMap {
version: Array[Byte] =>
Option(Keys.dataForSid(sectionId, feedId, version, sid)) flatMap {
getDataKey: Array[Byte] =>
Option(jedis.get(getDataKey)) flatMap {
packedData: Array[Byte] =>
val data = doSomeStuffWith(packedData)
Option(Product(data, "more interesting things"))
}
}
}
}
}
The way to do this is to use for:
for {
sid <- Option(jedis.get(...))
version <- Option(jedis.get(..., sid, ...))
getDataKey <- Option(jedis.get(...version,...))
packedData <- Option(jedis.get(getDataKey))
} yield {
// Do stuff with packedData
}
This will return None if any of the get calls returns None, otherwise it will return Some(x) where x is the result of the yeild expression.
You might also want to consider writing a wrapper for jedis.get which returns Option(x) rather than using null as the error result.
Related
I want to log in the event that a record doesn't have an adjoining record. Is there a purely functional way to do this? One that separates the side effect from the data transformation?
Here's an example of what I need to do:
val records: Seq[Record] = Seq(record1, record2, ...)
val accountsMap: Map[Long, Account] = Map(record1.id -> account1, ...)
def withAccount(accountsMap: Map[Long, Account])(r: Record): (Record, Option[Account]) = {
(r, accountsMap.get(r.id))
}
def handleNoAccounts(tuple: (Record, Option[Account]) = {
val (r, a) = tuple
if (a.isEmpty) logger.error(s"no account for ${record.id}")
tuple
}
def toRichAccount(tuple: (Record, Option[Account]) = {
val (r, a) = tuple
a.map(acct => RichAccount(r, acct))
}
records
.map(withAccount(accountsMap))
.map(handleNoAccounts) // if no account is found, log
.flatMap(toRichAccount)
So there are multiple issues with this approach that I think make it less than optimal.
The tuple return type is clumsy. I have to destructure the tuple in both of the latter two functions.
The logging function has to handle the logging and then return the tuple with no changes. It feels weird that this is passed to .map even though no transformation is taking place -- maybe there is a better way to get this side effect.
Is there a functional way to clean this up?
I could be wrong (I often am) but I think this does everything that's required.
records
.flatMap(r =>
accountsMap.get(r.id).fold{
logger.error(s"no account for ${r.id}")
Option.empty[RichAccount]
}{a => Some(RichAccount(r,a))})
If you're using scala 2.13 or newer you could use tapEach, which takes function A => Unit to apply side effect on every element of function and then passes collection unchanged:
//you no longer need to return tuple in side-effecting function
def handleNoAccounts(tuple: (Record, Option[Account]): Unit = {
val (r, a) = tuple
if (a.isEmpty) logger.error(s"no account for ${record.id}")
}
records
.map(withAccount(accountsMap))
.tapEach(handleNoAccounts) // if no account is found, log
.flatMap(toRichAccount)
In case you're using older Scala, you could provide extension method (updated according to Levi's Ramsey suggestion):
implicit class SeqOps[A](s: Seq[A]) {
def tapEach(f: A => Unit): Seq[A] = {
s.foreach(f)
s
}
}
I am trying to incorporate a database into my http-microservice.
The microservice has a function getValueFromInternet(val: Foo): Future[Value] which was being called by my microservice on a GET request. Now, I want it to happen such that, a function getValue(val: Foo): Future[Value] would first query a db and if the database returns no results, call getValueFromInternet. The database query returns a Future[Seq[Value2]] where I can convert Value2 to Value using a function. And if no entry is found corresponding to that value, an empty Vector is returned.
This is what I have tried so far:
def getValue(val: Foo): Future[Value] = {
val resultFuture = db.getValue(val)
// 1st attempt. Clearly wrong
resultFuture onComplete {
case Success(Vector()) => getValueFromInternet(val)
case Success(vec) => convertValue2to1(vec.head)
}
// 2nd attempt. This is also wrong
resultFuture match {
case Future(Success(Vector())) => getValueFromInternet(val)
case Future(Success(vec)) => convertValue2to1(vec.head)
}
}
I would be grateful for any help suggesting how I can do this.
I have implemented the database and microservice independently and you can find them here and here
You have to use flatMap, since the thing you want to do if the first operation does not return a result also returns a future.
This is as close to your code as possible while still compiling. Note that you can't have identifiers called val in scala, since that is a keyword.
def getValue(v: Foo)(implicit ec: ExecutionContext): Future[Value] = {
val resultFuture: Future[Seq[Value2]] = db.getValue(v)
resultFuture.flatMap { vec =>
if(vec.isEmpty)
getValueFromInternet(v)
else
Future.successful(convertValue2to1(vec.head))
}
}
How would you do, using for comprehensions, with futures returning an option, that is Future[Option[T]], and executing subsequent futures if and only if a condition is met, like for example, the second future should be executed only if the first future has a non empty value on a given field?
Not sure if I'm explaining myself well enough what i'm trying to achieve but here is some code that isn't working for me:
def show(id: Int, slug: String) = Action.async { implicit request =>
for {
aWithbTuple <- someDAO.findByIDJoined(id)
cID = aWithbTuple.map(_._1.cID).get if aWithbTuple.isDefined
c <- cDAO.findByID(cID) //Load the c table record only if the cID field is present in the first member of the tuple of aWithbTuple
} yield {
aWithbTuple.fold(NotFound) { case (a, b) =>
Ok(views.html.show(a, b, c))
}
}
}
It compiles well but I get an exception if the first future's result is None, what i'd like to get is a None as result if the first future returns None, but it seems that even if the first future returns None, the second future still gets executed and I get an exception as a result.
The problem you are running into is that the if in the for-comprehension isn't a postfix conditional as you might find in some other languages. It is instead, a filter operation.
The difference is that as a postfix conditional, it would prevent the action preceeding it from being executed if the conditional fails. As a filter, however, the action is executed and then then filter is used to determine whether to proceed. Because of this you are failing on aWithbTuple.map(_._1.cID).get since the get throws when the option is a None.
If you used the if as a filter on the future someDAO.findByIDJoined(id) instead, it will short-circuit the for-comprehension with a failed future when aWithbTuple is None, thus allowing you to safely call get:
def show(id: Int, slug: String) = Action.async { implicit request =>
for {
aWithbTuple <- someDAO.findByIDJoined(id) if aWithbTuple.isDefined
cID = aWithbTuple.map(_._1.cID).get
c <- cDAO.findByID(cID) //Load the c table record only if the cID field is present in the first member of the tuple of aWithbTuple
} yield {
aWithbTuple.fold(NotFound) { case (a, b) =>
Ok(views.html.show(a, b, c))
}
}
}
I have to get a list of issues for each file of a given list from a REST API with Scala. I want to do the requests in parallel, and use the Dispatch library for this. My method is called from a Java framework and I have to wait at the end of this method for the result of all the futures to yield the overall result back to the framework. Here's my code:
def fetchResourceAsJson(filePath: String): dispatch.Future[json4s.JValue]
def extractLookupId(json: org.json4s.JValue): Option[String]
def findLookupId(filePath: String): Future[Option[String]] =
for (json <- fetchResourceAsJson(filePath))
yield extractLookupId(json)
def searchIssuesJson(lookupId: String): Future[json4s.JValue]
def extractIssues(json: org.json4s.JValue): Seq[Issue]
def findIssues(lookupId: String): Future[Seq[Issue]] =
for (json <- searchIssuesJson(componentId))
yield extractIssues(json)
def getFilePathsToProcess: List[String]
def thisIsCalledByJavaFramework(): java.util.Map[String, java.util.List[Issue]] = {
val finalResultPromise = Promise[Map[String, Seq[Issue]]]()
// (1) inferred type of issuesByFile not as expected, cannot get
// the type system happy, would like to have Seq[Future[(String, Seq[Issue])]]
val issuesByFile = getFilePathsToProcess map { f =>
findLookupId(f).flatMap { lookupId =>
(f, findIssues(lookupId)) // I want to yield a tuple (String, Seq[Issue]) here
}
}
Future.sequence(issuesByFile) onComplete {
case Success(x) => finalResultPromise.success(x) // (2) how to return x here?
case Failure(x) => // (3) how to return null from here?
}
//TODO transform finalResultPromise to Java Map
}
This code snippet has several issues. First, I'm not getting the type I would expect for issuesByFile (1). I would like to just ignore the result of findLookUpId if it is not able to find the lookUp ID (i.e., None). I've read in various tutorials that Future[Option[X]] is not easy to handle in function compositions and for expressions in Scala. So I'm also curious what the best practices are to handle these properly.
Second, I somehow have to wait for all futures to finish, but don't know how to return the result to the calling Java framework (2). Can I use a promise here to achieve this? If yes, how can I do it?
And last but not least, in case of any errors, I would just like to return null from thisIsCalledByJavaFramework but don't know how (3).
Any help is much appreciated.
Thanks,
Michael
Several points:
The first problem at (1) is that you don't handle the case where findLookupId returns None. You need to decide what to do in this case. Fail the whole process? Exclude that file from the list?
The second problem at (1) is that findIssues will itself return a Future, which you need to map before you can build the result tuple
There's a shortcut for map and then Future.sequence: Future.traverse
If you cannot change the result type of the method because the Java interface is fixed and cannot be changed to support Futures itself you must wait for the Future to be completed. Use Await.ready or Await.result to do that.
Taking all that into account and choosing to ignore files for which no id could be found results in this code:
// `None` in an entry for a file means that no id could be found
def entryForFile(file: String): Future[(String, Option[Seq[Issue]])] =
findLookupId(file).flatMap {
// the need for this kind of pattern match shows
// the difficulty of working with `Future[Option[T]]`
case Some(id) ⇒ findIssues(id).map(issues ⇒ file -> Some(issues))
case None ⇒ Future.successful(file -> None)
}
def thisIsCalledByJavaFramework(): java.util.Map[String, java.util.List[Issue]] = {
val issuesByFile: Future[Seq[(String, Option[Seq[Issue]])]] =
Future.traverse(getFilePathsToProcess)(entryForFile)
import scala.collection.JavaConverters._
try
Await.result(issuesByFile, 10.seconds)
.collect {
// here we choose to ignore entries where no id could be found
case (f, Some(issues)) ⇒ f -> issues
}
.toMap.mapValues(_.asJava).asJava
catch {
case NonFatal(_) ⇒ null
}
}
I am using play2 and reactivemongo to fetch a result from mongodb. Each item of the result needs to be transformed to add some metadata. Afterwards I need to apply some sorting to it.
To deal with the transformation step I use enumerate():
def ideasEnumerator = collection.find(query)
.options(QueryOpts(skipN = page))
.sort(Json.obj(sortField -> -1))
.cursor[Idea]
.enumerate()
Then I create an Iteratee as follows:
val processIdeas: Iteratee[Idea, Unit] =
Iteratee.foreach[Idea] { idea =>
resolveCrossLinks(idea) flatMap { idea =>
addMetaInfo(idea.copy(history = None))
}
}
Finally I feed the Iteratee:
ideasEnumerator(processIdeas)
And now I'm stuck. Every example I saw does some println inside foreach, but seems not to care about a final result.
So when all documents are returned and transformed how do I get a Sequence, a List or some other datatype I can further deal with?
Change the signature of your Iteratee from Iteratee[Idea, Unit] to Iteratee[Idea, Seq[A]] where A is the type. Basically the first param of Iteratee is Input type and second param is Output type. In your case you gave the Output type as Unit.
Take a look at the below code. It may not compile but it gives you the basic usage.
ideasEnumerator.run(
Iteratee.fold(List.empty[MyObject]) { (accumulator, next) =>
accumulator + resolveCrossLinks(next) flatMap { next =>
addMetaInfo(next.copy(history = None))
}
}
) // returns Future[List[MyObject]]
As you can see, Iteratee is a simply a state machine. Just extract that Iteratee part and assign it to a val:
val iteratee = Iteratee.fold(List.empty[MyObject]) { (accumulator, next) =>
accumulator + resolveCrossLinks(next) flatMap { next =>
addMetaInfo(next.copy(history = None))
}
}
and feel free to use it where ever you need to convert from your Idea to List[MyObject]
With the help of your answers I ended up with
val processIdeas: Iteratee[Idea, Future[Vector[Idea]]] =
Iteratee.fold(Future(Vector.empty[Idea])) { (accumulator: Future[Vector[Idea]], next:Idea) =>
resolveCrossLinks(next) flatMap { next =>
addMetaInfo(next.copy(history = None))
} flatMap (ideaWithMeta => accumulator map (acc => acc :+ ideaWithMeta))
}
val ideas = collection.find(query)
.options(QueryOpts(page, perPage))
.sort(Json.obj(sortField -> -1))
.cursor[Idea]
.enumerate(perPage).run(processIdeas)
This later needs a ideas.flatMap(identity) to remove the returning Future of Futures but I'm fine with it and everything looks idiomatic and elegant I think.
The performance gained compared to creating a list and iterate over it afterwards is negligible though.