Let's say I have some Repository API where I have wrapped the transactions in a (Scalaz) Reader monad. Now I want to run computations over the results, and save the results back into the Repository. I tried something like:
type UOW[A] = Reader[Transaction, A]
object Record1Repo {
override def findAll: UOW[Seq[Record1]] = Reader(t => {
...
})
}
...
repo.run {
for {
all: Seq[Record1] <- Record1Repo.findAll
record: Record <- all
encoding: Encoding <- Processor.encode(record)
_ <- Record2Repo.save(Record2(encoding))
} yield {
logger.info(s"processed record id=${record.id}")
}}
But it falls apart with the futile attempt to map over the results in record <- all.
I'm quite new to this type of functional programming and couldn't find how express my intention properly. Any suggestions is welcome.
It fails because you are breaking out of the Reader monad.
You start with a Reader and then you extract from a Seq so this cannot be translated in a flatMap/map chain within the Reader structure.
Related
I'm trying to use cats effect in scala and in the end of the world I have the type:
IO[Vector[IO[Vector[IO[Unit]]]]]
I found only one method to run it:
for {
row <- rows.unsafeRunSync()
} yield
for {
cell <- row.unsafeRunSync()
} yield cell.handleErrorWith(errorHandlingFunc).unsafeRunSync()
But it looks pretty ugly. Please help me to understand how I can perform complex side effects.
UPDATE:
1)First IO - I open excel file and get the vector of rows i.e IO[Vector[Row]].
2)Second IO - I perform query to DB for each row. I can't compose IO monad with Vector[_],
3)Third IO - I create PDF file for each row from excel using Vector[Results] from DB.
So I have such functions as:
1) String=>IO[Vector[Row]]
2) Row=>IO[Vector[Results]]
3) Vector[Results] => IO[Unit]
For the sake of example here's a nonsense action I've just made up off the top of my head with the same type:
import cats.effect.IO
val actions: IO[Vector[IO[Vector[IO[Unit]]]]] =
IO(readLine).flatMap(in => IO(in.toInt)).map { count =>
(0 until count).toVector.map { _ =>
IO(System.nanoTime).map { t =>
(0 until 2).toVector.map { _ =>
IO(println(t.toString))
}
}
}
}
Here we're reading a string from standard input, parsing it as an integer, looking at the current time that many times, and printing it twice each time.
The correct way to flatten this type would be to use sequence to rearrange the layers:
import cats.implicits._
val program = actions.flatMap(_.sequence).flatMap(_.flatten.sequence_)
(Or something similar—there are lots of reasonable ways you could write this.)
This program has type IO[Unit], and works as we'd expect:
scala> program.unsafeRunSync
// I typed "3" here
8058983807657
8058983807657
8058984254443
8058984254443
8058984270434
8058984270434
Any time you see a deeply nested type involving multiple layers of IO and collections like this, though, it's likely that the best thing to do is to avoid getting in that situation in the first place (usually by using traverse). In this case we could rewrite our original actions like this:
val actions: IO[Unit] =
IO(readLine).flatMap(in => IO(in.toInt)).flatMap { count =>
(0 until count).toVector.traverse_ { _ =>
IO(System.nanoTime).flatMap { t =>
(0 until 2).toVector.traverse { _ =>
IO(println(t.toString))
}
}
}
}
This will work exactly the same way as our program, but we've avoided the nesting by replacing the maps in our original actions with either flatMap or traverse. Knowing which you need where is something that you learn through practice, but when you're starting out it's best to go in the smallest steps possible and follow the types.
I have a set of operations that are completed in sequence, but if an intermediate sequence returns "null" I would like to abort the operation early (skip the subsequent steps).
I conjured up a function like this which given an input parameter, performs several operations against Redis and will return a product if it exists. Since it is possible that one of the intermediate requests returns a null value, the complete operation could "fail" and I would like to short circuit the unnecessary steps that come afterwards.
The nesting here is becoming crazy, and I would like to make it more legible. Is there a proper "functional" way to perform this type of "if/else" short circuiting?
def getSingleProduct(firstSku: String): Option[Product] = {
val jedis = pool.getResource
val sid: Array[Byte] = jedis.get(Keys.sidForSku(firstSku, sectionId, feedId).getBytes)
Option(sid).flatMap {
sid: Array[Byte] =>
Option(jedis.get(Keys.latestVersionForSid(sectionId, feedId, sid))) flatMap {
version: Array[Byte] =>
Option(Keys.dataForSid(sectionId, feedId, version, sid)) flatMap {
getDataKey: Array[Byte] =>
Option(jedis.get(getDataKey)) flatMap {
packedData: Array[Byte] =>
val data = doSomeStuffWith(packedData)
Option(Product(data, "more interesting things"))
}
}
}
}
}
The way to do this is to use for:
for {
sid <- Option(jedis.get(...))
version <- Option(jedis.get(..., sid, ...))
getDataKey <- Option(jedis.get(...version,...))
packedData <- Option(jedis.get(getDataKey))
} yield {
// Do stuff with packedData
}
This will return None if any of the get calls returns None, otherwise it will return Some(x) where x is the result of the yeild expression.
You might also want to consider writing a wrapper for jedis.get which returns Option(x) rather than using null as the error result.
How can I make a class support for keywords in scala?
e.g:
class A(data: String) {
...
}
val a = A("I'm A")
for {
data <- a
} yield {
data
}
Thanks
The compiler rewrites all for comprehensions into the necessary constituent parts: map(), flatMap(), withFilter(), foreach(). That's why many Scala syntax rules are suspended inside the for comprehension, e.g. can't create variables in the standard fashion, val x = 2, and can't throw in println() statements.
In your example, this will work.
class A(data: String) {
def map[B](f: (String) => B) = f(data)
}
val a = new A("I'm A")
for {
data <- a
} yield {
data
} // res0: String = I'm A
But note that if you have multiple generators (the <- is a generator) then only the final one is turned into a map() call. The previous generators are all flatMap() calls.
If your for comprehension includes an if condition then you'll need a withFilter() as well.
I recommend avoiding for comprehensions until you have a good feel for how they work.
I have to get a list of issues for each file of a given list from a REST API with Scala. I want to do the requests in parallel, and use the Dispatch library for this. My method is called from a Java framework and I have to wait at the end of this method for the result of all the futures to yield the overall result back to the framework. Here's my code:
def fetchResourceAsJson(filePath: String): dispatch.Future[json4s.JValue]
def extractLookupId(json: org.json4s.JValue): Option[String]
def findLookupId(filePath: String): Future[Option[String]] =
for (json <- fetchResourceAsJson(filePath))
yield extractLookupId(json)
def searchIssuesJson(lookupId: String): Future[json4s.JValue]
def extractIssues(json: org.json4s.JValue): Seq[Issue]
def findIssues(lookupId: String): Future[Seq[Issue]] =
for (json <- searchIssuesJson(componentId))
yield extractIssues(json)
def getFilePathsToProcess: List[String]
def thisIsCalledByJavaFramework(): java.util.Map[String, java.util.List[Issue]] = {
val finalResultPromise = Promise[Map[String, Seq[Issue]]]()
// (1) inferred type of issuesByFile not as expected, cannot get
// the type system happy, would like to have Seq[Future[(String, Seq[Issue])]]
val issuesByFile = getFilePathsToProcess map { f =>
findLookupId(f).flatMap { lookupId =>
(f, findIssues(lookupId)) // I want to yield a tuple (String, Seq[Issue]) here
}
}
Future.sequence(issuesByFile) onComplete {
case Success(x) => finalResultPromise.success(x) // (2) how to return x here?
case Failure(x) => // (3) how to return null from here?
}
//TODO transform finalResultPromise to Java Map
}
This code snippet has several issues. First, I'm not getting the type I would expect for issuesByFile (1). I would like to just ignore the result of findLookUpId if it is not able to find the lookUp ID (i.e., None). I've read in various tutorials that Future[Option[X]] is not easy to handle in function compositions and for expressions in Scala. So I'm also curious what the best practices are to handle these properly.
Second, I somehow have to wait for all futures to finish, but don't know how to return the result to the calling Java framework (2). Can I use a promise here to achieve this? If yes, how can I do it?
And last but not least, in case of any errors, I would just like to return null from thisIsCalledByJavaFramework but don't know how (3).
Any help is much appreciated.
Thanks,
Michael
Several points:
The first problem at (1) is that you don't handle the case where findLookupId returns None. You need to decide what to do in this case. Fail the whole process? Exclude that file from the list?
The second problem at (1) is that findIssues will itself return a Future, which you need to map before you can build the result tuple
There's a shortcut for map and then Future.sequence: Future.traverse
If you cannot change the result type of the method because the Java interface is fixed and cannot be changed to support Futures itself you must wait for the Future to be completed. Use Await.ready or Await.result to do that.
Taking all that into account and choosing to ignore files for which no id could be found results in this code:
// `None` in an entry for a file means that no id could be found
def entryForFile(file: String): Future[(String, Option[Seq[Issue]])] =
findLookupId(file).flatMap {
// the need for this kind of pattern match shows
// the difficulty of working with `Future[Option[T]]`
case Some(id) ⇒ findIssues(id).map(issues ⇒ file -> Some(issues))
case None ⇒ Future.successful(file -> None)
}
def thisIsCalledByJavaFramework(): java.util.Map[String, java.util.List[Issue]] = {
val issuesByFile: Future[Seq[(String, Option[Seq[Issue]])]] =
Future.traverse(getFilePathsToProcess)(entryForFile)
import scala.collection.JavaConverters._
try
Await.result(issuesByFile, 10.seconds)
.collect {
// here we choose to ignore entries where no id could be found
case (f, Some(issues)) ⇒ f -> issues
}
.toMap.mapValues(_.asJava).asJava
catch {
case NonFatal(_) ⇒ null
}
}
I have two functions which return Futures. I'm trying to feed a modified result from first function into the other using a for-yield comprehension.
This approach works:
val schoolFuture = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- schoolStore.getSchool(sid.get) if sid.isDefined
} yield s
However I'm not happy with having the "if" in there, it seems that I should be able to use a map instead.
But when I try with a map:
val schoolFuture: Future[Option[School]] = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- sid.map(schoolStore.getSchool(_))
} yield s
I get a compile error:
[error] found : Option[scala.concurrent.Future[Option[School]]]
[error] required: scala.concurrent.Future[Option[School]]
[error] s <- sid.map(schoolStore.getSchool(_))
I've played around with a few variations, but haven't found anything attractive that works. Can anyone suggest a nicer comprehension and/or explain what's wrong with my 2nd example?
Here is a minimal but complete runnable example with Scala 2.10:
import concurrent.{Future, Promise}
case class User(userId: Int)
case class UserDetails(userId: Int, schoolId: Option[Int])
case class School(schoolId: Int, name: String)
trait Error
class UserStore {
def getUserDetails(userId: Int): Future[Either[Error, UserDetails]] = Promise.successful(Right(UserDetails(1, Some(1)))).future
}
class SchoolStore {
def getSchool(schoolId: Int): Future[Option[School]] = Promise.successful(Option(School(1, "Big School"))).future
}
object Demo {
import concurrent.ExecutionContext.Implicits.global
val userStore = new UserStore
val schoolStore = new SchoolStore
val user = User(1)
val schoolFuture: Future[Option[School]] = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- sid.map(schoolStore.getSchool(_))
} yield s
}
(Edited to give a correct answer!)
The key here is that Future and Option don't compose inside for because there aren't the correct flatMap signatures. As a reminder, for desugars like so:
for ( x0 <- c0; w1 = d1; x1 <- c1 if p1; ... ; xN <- cN) yield f
c0.flatMap{ x0 =>
val w1 = d1
c1.filter(x1 => p1).flatMap{ x1 =>
... cN.map(xN => f) ...
}
}
(where any if statement throws a filter into the chain--I've given just one example--and the equals statements just set variables before the next part of the chain). Since you can only flatMap other Futures, every statement c0, c1, ... except the last had better produce a Future.
Now, getUserDetails and getSchool both produce Futures, but sid is an Option, so we can't put it on the right-hand side of a <-. Unfortunately, there's no clean out-of-the-box way to do this. If o is an option, we can
o.map(Future.successful).getOrElse(Future.failed(new Exception))
to turn an Option into an already-completed Future. So
for {
ud <- userStore.getUserDetails(user.userId) // RHS is a Future[Either[...]]
sid = ud.right.toOption.flatMap(_.schoolId) // RHS is an Option[Int]
fid <- sid.map(Future.successful).getOrElse(Future.failed(new Exception)) // RHS is Future[Int]
s <- schoolStore.getSchool(fid)
} yield s
will do the trick. Is that better than what you've got? Doubtful. But if you
implicit class OptionIsFuture[A](val option: Option[A]) extends AnyVal {
def future = option.map(Future.successful).getOrElse(Future.failed(new Exception))
}
then suddenly the for-comprehension looks reasonable again:
for {
ud <- userStore.getUserDetails(user.userId)
sid <- ud.right.toOption.flatMap(_.schoolId).future
s <- schoolStore.getSchool(sid)
} yield s
Is this the best way to write this code? Probably not; it relies upon converting a None into an exception simply because you don't know what else to do at that point. This is hard to work around because of the design decisions of Future; I'd suggest that your original code (which invokes a filter) is at least as good of a way to do it.
This answer to a similar question about Promise[Option[A]] might help. Just substitute Future for Promise.
I'm inferring the following types for getUserDetails and getSchool from your question:
getUserDetails: UserID => Future[Either[??, UserDetails]]
getSchool: SchoolID => Future[Option[School]]
Since you ignore the failure value from the Either, transforming it to an Option instead, you effectively have two values of type A => Future[Option[B]].
Once you've got a Monad instance for Future (there may be one in scalaz, or you could write your own as in the answer I linked), applying the OptionT transformer to your problem would look something like this:
for {
ud <- optionT(getUserDetails(user.userID) map (_.right.toOption))
sid <- optionT(Future.successful(ud.schoolID))
s <- optionT(getSchool(sid))
} yield s
Note that, to keep the types compatible, ud.schoolID is wrapped in an (already completed) Future.
The result of this for-comprehension would have type OptionT[Future, SchoolID]. You can extract a value of type Future[Option[SchoolID]] with the transformer's run method.
What behavior would you like to occur in the case that the Option[School] is None? Would you like the Future to fail? With what kind of exception? Would you like it to never complete? (That sounds like a bad idea).
Anyways, the if clause in a for-expression desugars to a call to the filter method. The contract on Future#filteris thus:
If the current future contains a value which satisfies the predicate,
the new future will also hold that value. Otherwise, the resulting
future will fail with a NoSuchElementException.
But wait:
scala> None.get
java.util.NoSuchElementException: None.get
As you can see, None.get returns the exact same thing.
Thus, getting rid of the if sid.isDefined should work, and this should return a reasonable result:
val schoolFuture = for {
ud <- userStore.getUserDetails(user.userId)
sid = ud.right.toOption.flatMap(_.schoolId)
s <- schoolStore.getSchool(sid.get)
} yield s
Keep in mind that the result of schoolFuture can be in instance of scala.util.Failure[NoSuchElementException]. But you haven't described what other behavior you'd like.
We've made small wrapper on Future[Option[T]] which acts like one monad (nobody even checked none of monad laws, but there is map, flatMap, foreach, filter and so on) - MaybeLater. It behaves much more than an async option.
There are a lot of smelly code there, but maybe it will be usefull at least as an example.
BTW: there are a lot of open questions(here for ex.)
It's easier to use https://github.com/qifun/stateless-future or https://github.com/scala/async to do A-Normal-Form transform.