Scala's for comprehension for Futures and Options - scala

I have recently read Manuel Bernhardt's new book Reactive Web Applications. In his book, he states that Scala developers should never use .get to retrieve an optional value.
I want to pick up his suggestions but I am struggling to avoid .get when using for comprehensions for Futures.
Let's say I have the following code:
for {
avatarUrl <- avatarService.retrieve(email)
user <- accountService.save(Account(profiles = List(profile.copy(avatarUrl = avatarUrl)))
userId <- user.id
_ <- accountTokenService.save(AccountToken.create(userId, email))
} yield {
Logger.info("Foo bar")
}
Normally, I would have used AccountToken.create(user.id.get, email) instead of AccountToken.create(userId, email). However, when trying to avoid this bad practice, I get the following exception:
[error] found : Option[Nothing]
[error] required: scala.concurrent.Future[?]
[error] userId <- user.id
[error] ^
How can I solve this?

First option
If you really want to use for comprehension you'll have to separate it to several fors, where each works with the same monad type:
for {
avatarUrl <- avatarService.retrieve(email)
user <- accountService.save(Account(profiles = List(profile.copy(avatarUrl = avatarUrl)))
} yield for {
userId <- user.id
} yield for {
_ <- accountTokenService.save(AccountToken.create(userId, email))
}
Second option
Another option is to avoid Future[Option[T]] altogether and use Future[T] which can materialize into Failure(e) where e is a NoSuchElementException whenever you expect a None (in your case, the accountService.save() method):
def saveWithoutOption(account: Account): Future[User] = {
this.save(account) map { userOpt =>
userOpt.getOrElse(throw new NoSuchElementException)
}
}
Then you'll have:
(for {
avatarUrl <- avatarService.retrieve(email)
user <- accountService.saveWithoutOption(Account(profiles = List(profile.copy(avatarUrl = avatarUrl)))
_ <- accountTokenService.save(AccountToken.create(user.id, email))
} yield {
Logger.info("Foo bar")
}) recover {
case t: NoSuchElementException => Logger.error("boo")
}
Third option
Fall back to map/flatMap and introduce intermediate results.

Let's take a step back and explore the meaning of our expression:
A Future is "eventually a value (but might fail)"
An Option is "maybe a value"
What are the semantics of Future[Option]? Let's explore the values to gain some intuition:
Future[Option]
Success(Some(x)) => Good. Let's do stuff with x
Success(None) => Finished but got nothing => This is probably an application-level error
Failure(_) => Something went wrong, so we don't have a value
We can flatten Success(None) into a Failure(SomeApplicationException) and eliminate the need of handling the Option separately.
For that, we can define a generic function to turn an Option into a Future and use the for-comprehension to apply the flattening.
def optionToFuture[T](opt:Option[T], ex: ()=>Exception):Future[T] = opt match {
case Some(v) => Future.successful(v)
case None => Future.failed(ex())
}
We can now express our computation fluently with a for-comprehension:
for {
avatarUrl <- avatarService.retrieve(email)
user <- accountService.save(Account(profiles = List(profile.copy(avatarUrl = avatarUrl)))
userId <- optionToFuture(user.id, () => new UserNotFoundException(email))
_ <- accountTokenService.save(AccountToken.create(userId, email))
} yield {
Logger.info("Foo bar")
}

Stop Option propogation by failing the Future when option is None
Fail the future when id is none and abort
for {
....
accountOpt <-
user.id.map { id =>
Account.create(id, ...)
}.getOrElse {
Future.failed(new Exception("could not create account."))
}
...
} yield result
Better to have a custom exception like
case class NoIdException(msg: String) extends Exception(msg)
invoking .get on Option should be done only if you are sure that option is Some(x) otherwise .get will throw an exception.
Thats by using .get is not good practise because it may cause an exception in the code.
Instead of .get its good practice to use getOrElse.
You can map or flatMap the option to get access to the inner value.
Good practice
val x: Option[Int] = giveMeOption()
x.getOrElse(defaultValue)
Get can be used here
val x: Option[Int] = giveMeOption()
x.OrElse(Some(1)).get

Related

How to return upon encountering first "true" in a List[IO[Boolean]] in Scala Cats Effect

Say I have a set of rules that have a validation function that returns IO[Boolean] at runtime.
case class Rule1() {
def validate(): IO[Boolean] = IO.pure(false)
}
case class Rule2() {
def validate(): IO[Boolean] = IO.pure(false)
}
case class Rule3() {
def validate(): IO[Boolean] = IO.pure(true)
}
val rules = List(Rule1(), Rule2(), Rule3())
Now I have to iterate through these rules and see "if any of these rules" hold valid and if not then throw exception!
for {
i <- rules.map(_.validate()).sequence
_ <- if (i.contains(true)) IO.unit else IO.raiseError(new RuntimeException("Failed"))
} yield ()
The problem with the code snippet above is that it is trying to evaluate all the rules! What I really want is to exit at the encounter of the first true validation.
Not sure how to achieve this using cats effects in Scala.
I claim that existsM is the most direct way to achieve what you want. It behaves pretty much the same as exists, but for monadic predicates:
for {
t <- rules.existsM(_.validate())
_ <- IO.raiseUnless(t)(new RuntimeException("Failed"))
} yield ()
It also stops the search as soon as it finds the first true.
The raiseUnless is just some syntactic sugar that's equivalent to the if-else from your question.
If you take a look at list of available extension methods in your IDE, you can find findM:
for {
opt <- rules.findM(_.validate())
_ <- opt match {
case Some(_) => IO.unit
case None => IO.raiseError(new RuntimeException("Failed")
}
} yield ()
Doing it manually could be done with foldLeft and flatMap:
rules.foldLeft(IO.pure(false)) { (valueSoFar, nextValue) =>
valueSoFar.flatMap {
case true => IO.pure(true) // can skip evaluating nextValue
case false => nextValue.validate() // need to find the first true IO yet
}
}.flatMap {
case true => IO.unit
case false => IO.raiseError(new RuntimeException("Failed")
}
The former should have the additional advantage that it doesn't have to iterate over whole collection when it finds the first match, while the latter will still go through all items, even if will start discarding them at some point. findM solves that by using tailRecM internally to terminate the iteration on first met condition.
You can try recursive
def firstTrue(rules: List[{def validate(): IO[Boolean]}]): IO[Unit] = rules match {
case r :: rs => for {
b <- r.validate()
res <- if (b) IO.unit else firstTrue(rs)
} yield res
case _ => IO.raiseError(new RuntimeException("Failed"))
}
Another approach is not using booleans at all, but the monad capabilities of IO
def validateRules(rules: List[Rule]): IO[Unit] =
rules.traverse_ { rule =>
rule.validate().flatMap { flag =>
IO.raiseUnless(flag)(new RuntimeException("Failed"))
}
}

Avoiding deeply nested Option cascades in Scala

Say I have three database access functions foo, bar, and baz that can each return Option[A] where A is some model class, and the calls depend on each other.
I would like to call the functions sequentially and in each case, return an appropriate error message if the value is not found (None).
My current code looks like this:
Input is a URL: /x/:xID/y/:yID/z/:zID
foo(xID) match {
case None => Left(s"$xID is not a valid id")
case Some(x) =>
bar(yID) match {
case None => Left(s"$yID is not a valid id")
case Some(y) =>
baz(zID) match {
case None => Left(s"$zID is not a valid id")
case Some(z) => Right(process(x, y, z))
}
}
}
As can be seen, the code is badly nested.
If instead, I use a for comprehension, I cannot give specific error messages, because I do not know which step failed:
(for {
x <- foo(xID)
y <- bar(yID)
z <- baz(zID)
} yield {
Right(process(x, y, z))
}).getOrElse(Left("One of the IDs was invalid, but we do not know which one"))
If I use map and getOrElse, I end up with code almost as nested as the first example.
Is these some better way to structure this to avoid the nesting while allowing specific error messages?
You can get your for loop working by using right projections.
def ckErr[A](id: String, f: String => Option[A]) = (f(id) match {
case None => Left(s"$id is not a valid id")
case Some(a) => Right(a)
}).right
for {
x <- ckErr(xID, foo)
y <- ckErr(yID, bar)
z <- ckErr(zID, baz)
} yield process(x,y,z)
This is still a little clumsy, but it has the advantage of being part of the standard library.
Exceptions are another way to go, but they slow things down a lot if the failure cases are common. I'd only use that if failure was truly exceptional.
It's also possible to use non-local returns, but it's kind of awkward for this particular setup. I think right projections of Either are the way to go. If you really like working this way but dislike putting .right all over the place, there are various places you can find a "right-biased Either" which will act like the right projection by default (e.g. ScalaUtils, Scalaz, etc.).
Instead of using an Option I would instead use a Try. That way you have the Monadic composition that you'd like mixed with the ability to retain the error.
def myDBAccess(..args..) =
thingThatDoesStuff(args) match{
case Some(x) => Success(x)
case None => Failure(new IdError(args))
}
I'm assuming in the above that you don't actually control the functions and can't refactor them to give you a non-Option. If you did, then simply substitute Try.
I know this question was answered some time back, but I wanted to give an alternative to the accepted answer.
Given that, in your example, the three Options are independent, you can treat them as Applicative Functors and use ValidatedNel from Cats to simplify and aggregate the handling of the unhappy path.
Given the code:
import cats.data.Validated.{invalidNel, valid}
def checkOption[B, T](t : Option[T])(ifNone : => B) : ValidatedNel[B, T] = t match {
case None => invalidNel(ifNone)
case Some(x) => valid(x)
def processUnwrappedData(a : Int, b : String, c : Boolean) : String = ???
val o1 : Option[Int] = ???
val o2 : Option[String] = ???
val o3 : Option[Boolean] = ???
You can then replicate obtain what you want with:
//import cats.syntax.cartesian._
(
checkOption(o1)(s"First option is not None") |#|
checkOption(o2)(s"Second option is not None") |#|
checkOption(o3)(s"Third option is not None")
) map (processUnwrappedData)
This approach will allow you to aggregate failures, which was not possible in your solution (as using for-comprehensions enforces sequential evaluation). More examples and documentation can be found here and here.
Finally this solution uses Cats Validated but could easily be translated to Scalaz Validation
I came up with this solution (based on #Rex's solution and his comments):
def ifTrue[A](boolean: Boolean)(isFalse: => A): RightProjection[A, Unit.type] =
Either.cond(boolean, Unit, isFalse).right
def none[A](option: Option[_])(isSome: => A): RightProjection[A, Unit.type] =
Either.cond(option.isEmpty, Unit, isSome).right
def some[A, B](option: Option[A])(ifNone: => B): RightProjection[B, A] =
option.toRight(ifNone).right
They do the following:
ifTrue is used when a function returns a Boolean, with true being the "success" case (e.g.: isAllowed(userId)). It actually returns Unit so should be used as _ <- ifTrue(...) { error } in a for comprehension.
none is used when a function returns an Option with None being the "success" case (e.g.: findUser(email) for creating accounts with unique email addresses). It actually returns Unit so should be used as _ <- none(...) { error } in a for comprehension.
some is used when a function returns an Option with Some() being the "success" case (e.g.: findUser(userId) for a GET /users/userId). It returns the contents of the Some: user <- some(findUser(userId)) { s"user $userId not found" }.
They are used in a for comprehension:
for {
x <- some(foo(xID)) { s"$xID is not a valid id" }
y <- some(bar(yID)) { s"$yID is not a valid id" }
z <- some(baz(zID)) { s"$zID is not a valid id" }
} yield {
process(x, y, z)
}
This returns an Either[String, X] where the String is an error message and the X is the result of calling process.

Why do futures get called in a for comprehension if they aren't used?

I'm trying to implement a system that caches requests to an external API. If the response is in cache, then no request should be made to the external site.
I have two methods:
// Check to see if the response is in the database
def checkCache(searchParameters: JsValue): Future[Option[JsValue]]
// Call the external API and get the JSON response
def getResponse(path: String): Future[JsValue]
And then I try to do the following:
val json: Future[JsValue] = for {
databaseJson <- checkCache(searchParameters)
externalJson <- getResponse(path)
} yield databaseJson match {
case None => externalJson
case Some(x) => x
}
This works, but a request to the external API is made all the time, even when the cached result is returned. This is obviously not what I want because it's slow.
How do I fix this?
The for comprehension maps over the futures, not the Option within it. Your code would translate to this
checkCache(searchParameters) flatMap { databaseJson =>
getResponse(path) map { externalJson =>
databaseJson match {
case None => externalJson
case Some(x) => x
}
}
}
So you are always calling getResponse() obviously.
You need something along the lines of this (not tested):
checkCache(searchParameters) flatMap { databaseJson =>
databaseJson match {
case None => getResponse(path)
case Some(x) => Future.successful(x)
}
}
You could also give this a shot:
val json: Future[JsValue] = for {
databaseJson <- checkCache(searchParameters)
json <- databaseJson.fold(getResponse(path))(js =>Future.successful(js))
} yield json
Similar in spirit to Marius Soutier's answer, but doing the Option checking via a fold right in the for comprehension.
This is a little cheesy, but still:
checkCache(params) map (_.get) fallbackTo getResponse(path)
Also not tested.
Update:
I didn't like the conversion to failure by None.get, but actually this isn't cheesy at all but is very natural. More naturally:
checkCache(params) filter (_.nonEmpty) map (_.get) fallbackTo getResponse(path)
The Some.get is now just a wart due to the asymmetry in the two futures (that they are not both Option).

Chaining validation in Scala

I have a Scala case class containing command-line configuration information:
case class Config(emailAddress: Option[String],
firstName: Option[String]
lastName: Option[String]
password: Option[String])
I am writing a validation function that checks that each of the values is a Some:
def validateConfig(config: Config): Try[Config] = {
if (config.emailAddress.isEmpty) {
Failure(new IllegalArgumentException("Email Address")
} else if (config.firstName.isEmpty) {
Failure(new IllegalArgumentException("First Name")
} else if (config.lastName.isEmpty) {
Failure(new IllegalArgumentException("Last Name")
} else if (config.password.isEmpty) {
Failure(new IllegalArgumentException("Password")
} else {
Success(config)
}
}
but if I understand monads from Haskell, it seems that I should be able to chain the validations together (pseudo syntax):
def validateConfig(config: Config): Try[Config] = {
config.emailAddress.map(Success(config)).
getOrElse(Failure(new IllegalArgumentException("Email Address")) >>
config.firstName.map(Success(config)).
getOrElse(Failure(new IllegalArgumentException("First Name")) >>
config.lastName.map(Success(config)).
getOrElse(Failure(new IllegalArgumentException("Last Name")) >>
config.password.map(Success(config)).
getOrElse(Failure(new IllegalArgumentException("Password"))
}
If any of the config.XXX expressions returns Failure, the whole thing (validateConfig) should fail, otherwise Success(config) should be returned.
Is there some way to do this with Try, or maybe some other class?
It's pretty straightforward to convert each Option to an instance of the right projection of Either:
def validateConfig(config: Config): Either[String, Config] = for {
_ <- config.emailAddress.toRight("Email Address").right
_ <- config.firstName.toRight("First Name").right
_ <- config.lastName.toRight("Last Name").right
_ <- config.password.toRight("Password").right
} yield config
Either isn't a monad in the standard library's terms, but its right projection is, and will provide the behavior you want in the case of failure.
If you'd prefer to end up with a Try, you could just convert the resulting Either:
import scala.util._
val validate: Config => Try[Config] = (validateConfig _) andThen (
_.fold(msg => Failure(new IllegalArgumentException(msg)), Success(_))
)
I wish that the standard library provided a nicer way to make this conversion, but it doesn't.
It's a case class, so why aren't you doing this with pattern matching?
def validateConfig(config: Config): Try[Config] = config match {
case Config(None, _, _, _) => Failure(new IllegalArgumentException("Email Address")
case Config(_, None, _, _) => Failure(new IllegalArgumentException("First Name")
case Config(_, _, None, _) => Failure(new IllegalArgumentException("Last Name")
case Config(_, _, _, None) => Failure(new IllegalArgumentException("Password")
case _ => Success(config)
}
In your simple example, my priority would be to forget monads and chaining, just get rid of that nasty if...else smell.
However, while a case class works perfectly well for a short list, for a large number of configuration options, this becomes tedious and the risk of error increases. In this case, I would consider something like this:
Add a method that returns a key->value map of the configuration options, using the option names as the keys.
Have the Validate method check if any value in the map is None
If no such value, return success.
If at least one value matches, return that value name with the error.
So assuming that somewhere is defined
type OptionMap = scala.collection.immutable.Map[String, Option[Any]]
and the Config class has a method like this:
def optionMap: OptionMap = ...
then I would write Config.validate like this:
def validate: Either[List[String], OptionMap] = {
val badOptions = optionMap collect { case (s, None) => s }
if (badOptions.size > 0)
Left(badOptions)
else
Right(optionMap)
}
So now Config.validate returns either a Left containing the name of all the bad options or a Right containing the full map of options and their values. Frankly, it probably doesn't matter what you put in the Right.
Now, anything that wants to validate a Config just calls Config.validate and examines the result. If it's a Left, it can throw an IllegalArgumentException containing one or more of the names of bad options. If it's a Right, it can do whatever it wanted to do, knowing the Config is valid.
So we could rewrite your validateConfig function as
def validateConfig(config: Config): Try[Config] = config.validate match {
case Left(l) => Failure(new IllegalArgumentException(l.toString))
case _ => Success(config)
}
Can you see how much more functional and OO this is getting?
No imperative chain of if...else
The Config object validates itself
The consequences of a Config object being invalid are left to the larger program.
I think a real example would be more complex yet, though. You are validating options by saying "Does it contain Option[String] or None?" but not checking the validity of the string itself. Really, I think your Config class should contain a map of options where the name maps to the value and to an anonymous function that validates the string. I could describe how to extend the above logic to work with that model, but I think I'll leave that as an exercise for you. I will give you a hint: you might want to return not just the list of failed options, but also the reason for failure in each case.
Oh, by the way... I hope none of the above implies that I think you should actually store the options and their values as an optionMap inside the object. I think it's useful to be able to retrieve them like that, but I wouldn't ever encourage such exposure of the actual internal representation ;)
Here's a solution that I came up with after some searching and scaladocs reading:
def validateConfig(config: Config): Try[Config] = {
for {
_ <- Try(config.emailAddress.
getOrElse(throw new IllegalArgumentException("Email address missing")))
_ <- Try(config.firstName.
getOrElse(throw new IllegalArgumentException("First name missing")))
_ <- Try(config.lastName.
getOrElse(throw new IllegalArgumentException("Last name missing")))
_ <- Try(config.password.
getOrElse(throw new IllegalArgumentException("Password missing")))
} yield config
}
Similar to Travis Brown's answer.

Using Either to process failures in Scala code

Option monad is a great expressive way to deal with something-or-nothing things in Scala. But what if one needs to log a message when "nothing" occurs? According to the Scala API documentation,
The Either type is often used as an
alternative to scala.Option where Left
represents failure (by convention) and
Right is akin to Some.
However, I had no luck to find best practices using Either or good real-world examples involving Either for processing failures. Finally I've come up with the following code for my own project:
def logs: Array[String] = {
def props: Option[Map[String, Any]] = configAdmin.map{ ca =>
val config = ca.getConfiguration(PID, null)
config.properties getOrElse immutable.Map.empty
}
def checkType(any: Any): Option[Array[String]] = any match {
case a: Array[String] => Some(a)
case _ => None
}
def lookup: Either[(Symbol, String), Array[String]] =
for {val properties <- props.toRight('warning -> "ConfigurationAdmin service not bound").right
val logsParam <- properties.get("logs").toRight('debug -> "'logs' not defined in the configuration").right
val array <- checkType(logsParam).toRight('warning -> "unknown type of 'logs' confguration parameter").right}
yield array
lookup.fold(failure => { failure match {
case ('warning, msg) => log(LogService.WARNING, msg)
case ('debug, msg) => log(LogService.DEBUG, msg)
case _ =>
}; new Array[String](0) }, success => success)
}
(Please note this is a snippet from a real project, so it will not compile on its own)
I'd be grateful to know how you are using Either in your code and/or better ideas on refactoring the above code.
Either is used to return one of possible two meaningful results, unlike Option which is used to return a single meaningful result or nothing.
An easy to understand example is given below (circulated on the Scala mailing list a while back):
def throwableToLeft[T](block: => T): Either[java.lang.Throwable, T] =
try {
Right(block)
} catch {
case ex => Left(ex)
}
As the function name implies, if the execution of "block" is successful, it will return "Right(<result>)". Otherwise, if a Throwable is thrown, it will return "Left(<throwable>)". Use pattern matching to process the result:
var s = "hello"
throwableToLeft { s.toUpperCase } match {
case Right(s) => println(s)
case Left(e) => e.printStackTrace
}
// prints "HELLO"
s = null
throwableToLeft { s.toUpperCase } match {
case Right(s) => println(s)
case Left(e) => e.printStackTrace
}
// prints NullPointerException stack trace
Hope that helps.
Scalaz library has something alike Either named Validation. It is more idiomatic than Either for use as "get either a valid result or a failure".
Validation also allows to accumulate errors.
Edit: "alike" Either is complettly false, because Validation is an applicative functor, and scalaz Either, named \/ (pronounced "disjonction" or "either"), is a monad.
The fact that Validation can accumalate errors is because of that nature. On the other hand, / has a "stop early" nature, stopping at the first -\/ (read it "left", or "error") it encounters. There is a perfect explanation here: http://typelevel.org/blog/2014/02/21/error-handling.html
See: http://scalaz.googlecode.com/svn/continuous/latest/browse.sxr/scalaz/example/ExampleValidation.scala.html
As requested by the comment, copy/paste of the above link (some lines removed):
// Extracting success or failure values
val s: Validation[String, Int] = 1.success
val f: Validation[String, Int] = "error".fail
// It is recommended to use fold rather than pattern matching:
val result: String = s.fold(e => "got error: " + e, s => "got success: " + s.toString)
s match {
case Success(a) => "success"
case Failure(e) => "fail"
}
// Validation is a Monad, and can be used in for comprehensions.
val k1 = for {
i <- s
j <- s
} yield i + j
k1.toOption assert_≟ Some(2)
// The first failing sub-computation fails the entire computation.
val k2 = for {
i <- f
j <- f
} yield i + j
k2.fail.toOption assert_≟ Some("error")
// Validation is also an Applicative Functor, if the type of the error side of the validation is a Semigroup.
// A number of computations are tried. If the all success, a function can combine them into a Success. If any
// of them fails, the individual errors are accumulated.
// Use the NonEmptyList semigroup to accumulate errors using the Validation Applicative Functor.
val k4 = (fNel <**> fNel){ _ + _ }
k4.fail.toOption assert_≟ some(nel1("error", "error"))
The snippet you posted seems very contrived. You use Either in a situation where:
It's not enough to just know the data isn't available.
You need to return one of two distinct types.
Turning an exception into a Left is, indeed, a common use case. Over try/catch, it has the advantage of keeping the code together, which makes sense if the exception is an expected result. The most common way of handling Either is pattern matching:
result match {
case Right(res) => ...
case Left(res) => ...
}
Another interesting way of handling Either is when it appears in a collection. When doing a map over a collection, throwing an exception might not be viable, and you may want to return some information other than "not possible". Using an Either enables you to do that without overburdening the algorithm:
val list = (
library
\\ "books"
map (book =>
if (book \ "author" isEmpty)
Left(book)
else
Right((book \ "author" toList) map (_ text))
)
)
Here we get a list of all authors in the library, plus a list of books without an author. So we can then further process it accordingly:
val authorCount = (
(Map[String,Int]() /: (list filter (_ isRight) map (_.right.get)))
((map, author) => map + (author -> (map.getOrElse(author, 0) + 1)))
toList
)
val problemBooks = list flatMap (_.left.toSeq) // thanks to Azarov for this variation
So, basic Either usage goes like that. It's not a particularly useful class, but if it were you'd have seen it before. On the other hand, it's not useless either.
Cats has a nice way to create an Either from exception-throwing code:
val either: Either[NumberFormatException, Int] =
Either.catchOnly[NumberFormatException]("abc".toInt)
// either: Either[NumberFormatException,Int] = Left(java.lang.NumberFormatException: For input string: "abc")
in https://typelevel.org/cats/datatypes/either.html#working-with-exception-y-code