I'm trying to implement a system that caches requests to an external API. If the response is in cache, then no request should be made to the external site.
I have two methods:
// Check to see if the response is in the database
def checkCache(searchParameters: JsValue): Future[Option[JsValue]]
// Call the external API and get the JSON response
def getResponse(path: String): Future[JsValue]
And then I try to do the following:
val json: Future[JsValue] = for {
databaseJson <- checkCache(searchParameters)
externalJson <- getResponse(path)
} yield databaseJson match {
case None => externalJson
case Some(x) => x
}
This works, but a request to the external API is made all the time, even when the cached result is returned. This is obviously not what I want because it's slow.
How do I fix this?
The for comprehension maps over the futures, not the Option within it. Your code would translate to this
checkCache(searchParameters) flatMap { databaseJson =>
getResponse(path) map { externalJson =>
databaseJson match {
case None => externalJson
case Some(x) => x
}
}
}
So you are always calling getResponse() obviously.
You need something along the lines of this (not tested):
checkCache(searchParameters) flatMap { databaseJson =>
databaseJson match {
case None => getResponse(path)
case Some(x) => Future.successful(x)
}
}
You could also give this a shot:
val json: Future[JsValue] = for {
databaseJson <- checkCache(searchParameters)
json <- databaseJson.fold(getResponse(path))(js =>Future.successful(js))
} yield json
Similar in spirit to Marius Soutier's answer, but doing the Option checking via a fold right in the for comprehension.
This is a little cheesy, but still:
checkCache(params) map (_.get) fallbackTo getResponse(path)
Also not tested.
Update:
I didn't like the conversion to failure by None.get, but actually this isn't cheesy at all but is very natural. More naturally:
checkCache(params) filter (_.nonEmpty) map (_.get) fallbackTo getResponse(path)
The Some.get is now just a wart due to the asymmetry in the two futures (that they are not both Option).
Related
Below is this code snippet working well but with Await.result
def validate(loginRequest: LoginRequest): Option[AuthResponse] = {
val response:Option[User] = Await.result(userDao.getUserByUsernameAndPassword(loginRequest.username,Utilities.encrypt(loginRequest.password)),Duration.Inf )
response match {
case Some(value) =>populateResponse(value)
case None => None
}
}
I want to use Futures instead of await but then return the response. below is a non blocking snippet
val response:Future[Option[User]]= userDao.getUserByUsernameAndPassword(loginRequest.username,Utilities.encrypt(loginRequest.password))
How would u get a similar response without blocking
response match {
case Some(value) =>populateResponse(value)
case None => None
}
You just need to call map on the Future to change the result, and inside that call map on the Option to change the contents of the option:
response.map(_.map(populateResponse))
If the Option is None then the second map will do nothing, and if the Future had failed then the first map will do nothing.
so based on your great answers I came up with a solution. because it was an Option response.
response.map{
res => res match {
case Some(value) =>populateResponse(value)
case None => None
}
I have recently read Manuel Bernhardt's new book Reactive Web Applications. In his book, he states that Scala developers should never use .get to retrieve an optional value.
I want to pick up his suggestions but I am struggling to avoid .get when using for comprehensions for Futures.
Let's say I have the following code:
for {
avatarUrl <- avatarService.retrieve(email)
user <- accountService.save(Account(profiles = List(profile.copy(avatarUrl = avatarUrl)))
userId <- user.id
_ <- accountTokenService.save(AccountToken.create(userId, email))
} yield {
Logger.info("Foo bar")
}
Normally, I would have used AccountToken.create(user.id.get, email) instead of AccountToken.create(userId, email). However, when trying to avoid this bad practice, I get the following exception:
[error] found : Option[Nothing]
[error] required: scala.concurrent.Future[?]
[error] userId <- user.id
[error] ^
How can I solve this?
First option
If you really want to use for comprehension you'll have to separate it to several fors, where each works with the same monad type:
for {
avatarUrl <- avatarService.retrieve(email)
user <- accountService.save(Account(profiles = List(profile.copy(avatarUrl = avatarUrl)))
} yield for {
userId <- user.id
} yield for {
_ <- accountTokenService.save(AccountToken.create(userId, email))
}
Second option
Another option is to avoid Future[Option[T]] altogether and use Future[T] which can materialize into Failure(e) where e is a NoSuchElementException whenever you expect a None (in your case, the accountService.save() method):
def saveWithoutOption(account: Account): Future[User] = {
this.save(account) map { userOpt =>
userOpt.getOrElse(throw new NoSuchElementException)
}
}
Then you'll have:
(for {
avatarUrl <- avatarService.retrieve(email)
user <- accountService.saveWithoutOption(Account(profiles = List(profile.copy(avatarUrl = avatarUrl)))
_ <- accountTokenService.save(AccountToken.create(user.id, email))
} yield {
Logger.info("Foo bar")
}) recover {
case t: NoSuchElementException => Logger.error("boo")
}
Third option
Fall back to map/flatMap and introduce intermediate results.
Let's take a step back and explore the meaning of our expression:
A Future is "eventually a value (but might fail)"
An Option is "maybe a value"
What are the semantics of Future[Option]? Let's explore the values to gain some intuition:
Future[Option]
Success(Some(x)) => Good. Let's do stuff with x
Success(None) => Finished but got nothing => This is probably an application-level error
Failure(_) => Something went wrong, so we don't have a value
We can flatten Success(None) into a Failure(SomeApplicationException) and eliminate the need of handling the Option separately.
For that, we can define a generic function to turn an Option into a Future and use the for-comprehension to apply the flattening.
def optionToFuture[T](opt:Option[T], ex: ()=>Exception):Future[T] = opt match {
case Some(v) => Future.successful(v)
case None => Future.failed(ex())
}
We can now express our computation fluently with a for-comprehension:
for {
avatarUrl <- avatarService.retrieve(email)
user <- accountService.save(Account(profiles = List(profile.copy(avatarUrl = avatarUrl)))
userId <- optionToFuture(user.id, () => new UserNotFoundException(email))
_ <- accountTokenService.save(AccountToken.create(userId, email))
} yield {
Logger.info("Foo bar")
}
Stop Option propogation by failing the Future when option is None
Fail the future when id is none and abort
for {
....
accountOpt <-
user.id.map { id =>
Account.create(id, ...)
}.getOrElse {
Future.failed(new Exception("could not create account."))
}
...
} yield result
Better to have a custom exception like
case class NoIdException(msg: String) extends Exception(msg)
invoking .get on Option should be done only if you are sure that option is Some(x) otherwise .get will throw an exception.
Thats by using .get is not good practise because it may cause an exception in the code.
Instead of .get its good practice to use getOrElse.
You can map or flatMap the option to get access to the inner value.
Good practice
val x: Option[Int] = giveMeOption()
x.getOrElse(defaultValue)
Get can be used here
val x: Option[Int] = giveMeOption()
x.OrElse(Some(1)).get
Say I have three database access functions foo, bar, and baz that can each return Option[A] where A is some model class, and the calls depend on each other.
I would like to call the functions sequentially and in each case, return an appropriate error message if the value is not found (None).
My current code looks like this:
Input is a URL: /x/:xID/y/:yID/z/:zID
foo(xID) match {
case None => Left(s"$xID is not a valid id")
case Some(x) =>
bar(yID) match {
case None => Left(s"$yID is not a valid id")
case Some(y) =>
baz(zID) match {
case None => Left(s"$zID is not a valid id")
case Some(z) => Right(process(x, y, z))
}
}
}
As can be seen, the code is badly nested.
If instead, I use a for comprehension, I cannot give specific error messages, because I do not know which step failed:
(for {
x <- foo(xID)
y <- bar(yID)
z <- baz(zID)
} yield {
Right(process(x, y, z))
}).getOrElse(Left("One of the IDs was invalid, but we do not know which one"))
If I use map and getOrElse, I end up with code almost as nested as the first example.
Is these some better way to structure this to avoid the nesting while allowing specific error messages?
You can get your for loop working by using right projections.
def ckErr[A](id: String, f: String => Option[A]) = (f(id) match {
case None => Left(s"$id is not a valid id")
case Some(a) => Right(a)
}).right
for {
x <- ckErr(xID, foo)
y <- ckErr(yID, bar)
z <- ckErr(zID, baz)
} yield process(x,y,z)
This is still a little clumsy, but it has the advantage of being part of the standard library.
Exceptions are another way to go, but they slow things down a lot if the failure cases are common. I'd only use that if failure was truly exceptional.
It's also possible to use non-local returns, but it's kind of awkward for this particular setup. I think right projections of Either are the way to go. If you really like working this way but dislike putting .right all over the place, there are various places you can find a "right-biased Either" which will act like the right projection by default (e.g. ScalaUtils, Scalaz, etc.).
Instead of using an Option I would instead use a Try. That way you have the Monadic composition that you'd like mixed with the ability to retain the error.
def myDBAccess(..args..) =
thingThatDoesStuff(args) match{
case Some(x) => Success(x)
case None => Failure(new IdError(args))
}
I'm assuming in the above that you don't actually control the functions and can't refactor them to give you a non-Option. If you did, then simply substitute Try.
I know this question was answered some time back, but I wanted to give an alternative to the accepted answer.
Given that, in your example, the three Options are independent, you can treat them as Applicative Functors and use ValidatedNel from Cats to simplify and aggregate the handling of the unhappy path.
Given the code:
import cats.data.Validated.{invalidNel, valid}
def checkOption[B, T](t : Option[T])(ifNone : => B) : ValidatedNel[B, T] = t match {
case None => invalidNel(ifNone)
case Some(x) => valid(x)
def processUnwrappedData(a : Int, b : String, c : Boolean) : String = ???
val o1 : Option[Int] = ???
val o2 : Option[String] = ???
val o3 : Option[Boolean] = ???
You can then replicate obtain what you want with:
//import cats.syntax.cartesian._
(
checkOption(o1)(s"First option is not None") |#|
checkOption(o2)(s"Second option is not None") |#|
checkOption(o3)(s"Third option is not None")
) map (processUnwrappedData)
This approach will allow you to aggregate failures, which was not possible in your solution (as using for-comprehensions enforces sequential evaluation). More examples and documentation can be found here and here.
Finally this solution uses Cats Validated but could easily be translated to Scalaz Validation
I came up with this solution (based on #Rex's solution and his comments):
def ifTrue[A](boolean: Boolean)(isFalse: => A): RightProjection[A, Unit.type] =
Either.cond(boolean, Unit, isFalse).right
def none[A](option: Option[_])(isSome: => A): RightProjection[A, Unit.type] =
Either.cond(option.isEmpty, Unit, isSome).right
def some[A, B](option: Option[A])(ifNone: => B): RightProjection[B, A] =
option.toRight(ifNone).right
They do the following:
ifTrue is used when a function returns a Boolean, with true being the "success" case (e.g.: isAllowed(userId)). It actually returns Unit so should be used as _ <- ifTrue(...) { error } in a for comprehension.
none is used when a function returns an Option with None being the "success" case (e.g.: findUser(email) for creating accounts with unique email addresses). It actually returns Unit so should be used as _ <- none(...) { error } in a for comprehension.
some is used when a function returns an Option with Some() being the "success" case (e.g.: findUser(userId) for a GET /users/userId). It returns the contents of the Some: user <- some(findUser(userId)) { s"user $userId not found" }.
They are used in a for comprehension:
for {
x <- some(foo(xID)) { s"$xID is not a valid id" }
y <- some(bar(yID)) { s"$yID is not a valid id" }
z <- some(baz(zID)) { s"$zID is not a valid id" }
} yield {
process(x, y, z)
}
This returns an Either[String, X] where the String is an error message and the X is the result of calling process.
I'm trying to learn Futures and ReactiveMongo.
In my case I have a couple of invite objects and want to filter out the ones that already exist in the db. I do not want to update or upsert the ones already in the db. Therefore I have created a filter method:
filter method:
def isAllowedToReview(invite: Invite): Future[Boolean] = {
ReviewDAO.findById(invite.recoId, invite.invitedUserId).map {
maybeReview => {
maybeReview match {
case Some(review) => false
case None => true
}
}
}
}
DAO:
def findById(rId: Long, userId: Long): Future[Option[Review]] = findOne(Json.obj("rId" -> recoId, "userId" -> userId))
def findOne(query: JsObject)(implicit reader: Reads[T]): Future[Option[T]] = {
collection.find(query).one[T]
}
and then call:
val futureOptionSet: Set[Future[Option[Invite]]] = smsSet.filter(isAllowedToReview)
save the filtered set somehow...
this doesn't work since filter expects in this case Invite => Boolean but I'm sending Invite => Future(Boolean). How would you filter and save this?
smsSet.map(sms => isAllowedToReview(sms).map(b => sms -> b)) will have type Set[Future[(Invite, Boolean)]]. You should be able to call Future.sequence to turn it into a Future[Set[(Invite, Boolean)]]. Then you can collect the results .map(_.collect{ case (sms, true) => sms}).
So putting everything together a solution may look like this:
val futures = smsSet.map(sms => isAllowedToReview(sms).map(b => sms -> b))
val future = Future.sequence(futures)
val result = future.map(_.collect{ case (sms, true) => sms})
When you see map and sequence you may be able to refactor to:
val filteredSet = Future.traverse(smsSet){ sms =>
isAllowedToReview(sms).map(b => sms -> b)
}.map(_.collect{ case (sms, true) => sms})
Note that instead of returning the set, you may just want to save your sms there. But the way I wrote this, all will be wrapped in a Future and you can still compose with other operations.
You could try something like this:
val revsFut = Future.sequence(smsSet.map(invite => ReviewDAO.findById(invite.recoId, invite.invitedUserId)))
val toSave = for(revs <- revsFut) yield {
val flatRevs = revs.flatten
smsSet.filter{ invite =>
flatRevs.find(review => /*Add filter code here */).isDefined
}
}
What I'm doing here is first fetching the Set of reviews matching the the invites by mapping over the smsSet, fetching each individually and then sequencing that into one singe Future. Then, in the for-comprehension I flatten the Set of Option[Review] and then filter down the smsSet based on what's in that flatRevs Set. Since I don't know your object model, I had to leave the impl of the flatRevs.find up to you, but it should be pretty easy as that point.
Option monad is a great expressive way to deal with something-or-nothing things in Scala. But what if one needs to log a message when "nothing" occurs? According to the Scala API documentation,
The Either type is often used as an
alternative to scala.Option where Left
represents failure (by convention) and
Right is akin to Some.
However, I had no luck to find best practices using Either or good real-world examples involving Either for processing failures. Finally I've come up with the following code for my own project:
def logs: Array[String] = {
def props: Option[Map[String, Any]] = configAdmin.map{ ca =>
val config = ca.getConfiguration(PID, null)
config.properties getOrElse immutable.Map.empty
}
def checkType(any: Any): Option[Array[String]] = any match {
case a: Array[String] => Some(a)
case _ => None
}
def lookup: Either[(Symbol, String), Array[String]] =
for {val properties <- props.toRight('warning -> "ConfigurationAdmin service not bound").right
val logsParam <- properties.get("logs").toRight('debug -> "'logs' not defined in the configuration").right
val array <- checkType(logsParam).toRight('warning -> "unknown type of 'logs' confguration parameter").right}
yield array
lookup.fold(failure => { failure match {
case ('warning, msg) => log(LogService.WARNING, msg)
case ('debug, msg) => log(LogService.DEBUG, msg)
case _ =>
}; new Array[String](0) }, success => success)
}
(Please note this is a snippet from a real project, so it will not compile on its own)
I'd be grateful to know how you are using Either in your code and/or better ideas on refactoring the above code.
Either is used to return one of possible two meaningful results, unlike Option which is used to return a single meaningful result or nothing.
An easy to understand example is given below (circulated on the Scala mailing list a while back):
def throwableToLeft[T](block: => T): Either[java.lang.Throwable, T] =
try {
Right(block)
} catch {
case ex => Left(ex)
}
As the function name implies, if the execution of "block" is successful, it will return "Right(<result>)". Otherwise, if a Throwable is thrown, it will return "Left(<throwable>)". Use pattern matching to process the result:
var s = "hello"
throwableToLeft { s.toUpperCase } match {
case Right(s) => println(s)
case Left(e) => e.printStackTrace
}
// prints "HELLO"
s = null
throwableToLeft { s.toUpperCase } match {
case Right(s) => println(s)
case Left(e) => e.printStackTrace
}
// prints NullPointerException stack trace
Hope that helps.
Scalaz library has something alike Either named Validation. It is more idiomatic than Either for use as "get either a valid result or a failure".
Validation also allows to accumulate errors.
Edit: "alike" Either is complettly false, because Validation is an applicative functor, and scalaz Either, named \/ (pronounced "disjonction" or "either"), is a monad.
The fact that Validation can accumalate errors is because of that nature. On the other hand, / has a "stop early" nature, stopping at the first -\/ (read it "left", or "error") it encounters. There is a perfect explanation here: http://typelevel.org/blog/2014/02/21/error-handling.html
See: http://scalaz.googlecode.com/svn/continuous/latest/browse.sxr/scalaz/example/ExampleValidation.scala.html
As requested by the comment, copy/paste of the above link (some lines removed):
// Extracting success or failure values
val s: Validation[String, Int] = 1.success
val f: Validation[String, Int] = "error".fail
// It is recommended to use fold rather than pattern matching:
val result: String = s.fold(e => "got error: " + e, s => "got success: " + s.toString)
s match {
case Success(a) => "success"
case Failure(e) => "fail"
}
// Validation is a Monad, and can be used in for comprehensions.
val k1 = for {
i <- s
j <- s
} yield i + j
k1.toOption assert_≟ Some(2)
// The first failing sub-computation fails the entire computation.
val k2 = for {
i <- f
j <- f
} yield i + j
k2.fail.toOption assert_≟ Some("error")
// Validation is also an Applicative Functor, if the type of the error side of the validation is a Semigroup.
// A number of computations are tried. If the all success, a function can combine them into a Success. If any
// of them fails, the individual errors are accumulated.
// Use the NonEmptyList semigroup to accumulate errors using the Validation Applicative Functor.
val k4 = (fNel <**> fNel){ _ + _ }
k4.fail.toOption assert_≟ some(nel1("error", "error"))
The snippet you posted seems very contrived. You use Either in a situation where:
It's not enough to just know the data isn't available.
You need to return one of two distinct types.
Turning an exception into a Left is, indeed, a common use case. Over try/catch, it has the advantage of keeping the code together, which makes sense if the exception is an expected result. The most common way of handling Either is pattern matching:
result match {
case Right(res) => ...
case Left(res) => ...
}
Another interesting way of handling Either is when it appears in a collection. When doing a map over a collection, throwing an exception might not be viable, and you may want to return some information other than "not possible". Using an Either enables you to do that without overburdening the algorithm:
val list = (
library
\\ "books"
map (book =>
if (book \ "author" isEmpty)
Left(book)
else
Right((book \ "author" toList) map (_ text))
)
)
Here we get a list of all authors in the library, plus a list of books without an author. So we can then further process it accordingly:
val authorCount = (
(Map[String,Int]() /: (list filter (_ isRight) map (_.right.get)))
((map, author) => map + (author -> (map.getOrElse(author, 0) + 1)))
toList
)
val problemBooks = list flatMap (_.left.toSeq) // thanks to Azarov for this variation
So, basic Either usage goes like that. It's not a particularly useful class, but if it were you'd have seen it before. On the other hand, it's not useless either.
Cats has a nice way to create an Either from exception-throwing code:
val either: Either[NumberFormatException, Int] =
Either.catchOnly[NumberFormatException]("abc".toInt)
// either: Either[NumberFormatException,Int] = Left(java.lang.NumberFormatException: For input string: "abc")
in https://typelevel.org/cats/datatypes/either.html#working-with-exception-y-code