Some APIs offers paginated data.
I want to call the API and retrieve the data of all pages.
I was thinking an implementation of it using recursion.
I will call it to retrieve the first page data and the url of next page. If next page is empty it means there's no more data to retrieve.
I have this code, which is not really calling an API but it is useful to implement the recursive function I need. The issue I'm facing is that I can't add the tailrec annotation, I can't transform it to be tail recursive.
How can I achieve it?
Code for simulating the API which return results in pages:
final case class Response(data: List[Int], nextUrl: Option[String])
object Api {
def getNumbers(url: String): Future[Response] = {
url match {
case "1" => Future.successful(Response(List(1, 2), Option("2")))
case "2" => Future.successful(Response(List(3, 4), Option("3")))
case "3" => Future.successful(Response(List(5, 6), None))
case _ => Future.failed(new RuntimeException("Error!"))
}
}
}
Function/method to call the API, which I want to convert it to tail recursion:
def getPaginatedData(url: String): Future[Seq[Int]] = {
// #tailrec
def loop(url: String, acc: Seq[Int]): Future[Seq[Int]] = {
Api.getNumbers(url).flatMap { response =>
response.nextUrl match {
case None => Future.successful(acc ++ response.data)
case Some(link) => loop(link, acc ++ response.data)
}
}
}
loop(url, List.empty)
}
To run the example:
def main(args: Array[String]): Unit = {
SomeService.getPaginatedData("1").onComplete {
case Failure(e) => println(s"Error! $e")
case Success(value) => println(value)
}
}
Related
There is simple abstract look up service with possibility to retrieve value by key:
trait LookUp[F[_]] {
def read(key: String): F[Option[String]]
}
There is use case of this service, idea is to give implemented storage and accumulator with starting key, then ask for value from db if the result is None then stop and Return None, if the value is found then add it to accumulator list and look up for next value as key from previous call result. Execution stops when retrieved value already is found before or None is retrieved. Then a string of all acc elements is returned as result.
Tried like this:
def readWhileFound[F[_]: Monad](db: LookUp[F], acc: List[String]): F[Option[String]] = {
for{ result <- db.read(acc.head)} yield result match {
case Some(value) if(!acc.contains(value)) => readWhileFound(db, value::acc)
case _ => acc.mkstring("")
}
}
But I'm not able to get types right getting mismatch errors like:
found : F[cats.data.OptionT[[_]F[_],String]]
required: cats.data.OptionT[F,String]
Approach number 2:
def readWhileFound[F[_]: Monad](key: String, db: LookUp[F])(implicit m: Monad[F]): F[Option[String]] = {
m.tailRecM((Option(key), List.empty[String])) { case (currentK, accum) =>
currentK match {
case Some(value) if(!accum.contains(value)) => m.pure(Left((db.read(value), value :: accum)))
case _ => m.pure(Right(Some(accum.mkString(""))))
}
}
}
Getting compiler error:
(Found) F[Option[String]]
required: Option[String]
case Some(value) if(!accum.contains(value)) => m.pure(Left((db.read(value), value :: accum)))
Looks like db.read(value) somehow should be unwrapped out of F
This looks like a great use case for fs2:
You should be able to do something like this:
import fs2.Stream
def readWhileFound[F[_]: Concurrent](db: LookUp[F])(initialKey: String): F[List[String] =
Stream.unfoldEval(initialKey) { currentKey =>
db.read(key = currentKey).map(k => (k, k))
}.compile.toList
You are match-ing on the wrong expression in your first implementation. You should match on result, not on the entire for-comprehension. The implementation below should do what you're after.
def readWhileFound[F[_]: Monad](db: LookUp[F], startKey: String): F[Option[String]] = {
def loop(currKey: String, seenKeys: Set[String]): F[Option[String]] = {
db.read(currKey).flatMap {
case Some(nextKey) if !seenKeys.contains(nextKey) =>
loop(nextKey, seenKeys + nextKey)
case _ if seenKeys.nonEmpty => seenKeys.mkString("").some.pure[F]
case _ => none[String].pure[F]
}
}
loop(startKey, Set.empty)
}
I've replaced List with Set for the accumulated values because its contains method is more efficient, but if you care about the order in the returned result then you'll have to either go back to List (less efficient) or use two accumulators (one Set, the other List).
I want to rewrite console IO application (sum counting) to messenger-bot. The StdIn.readLine() let me get the next input number in recursion.
object HelloCountBot extends TelegramBot with Polling with Commands {
def go(number: Long, chatId: Long): IO[Long] =
for {
input <- ??? /*here I need get the next number*/
acc <- input.toLong match {
case 0L => sendMessageMethodIO(chatId, "The sum is:") *> IO(0L)
case _ => go(number + 1, chatId)
}
acc <- IO(acc + input.toLong)
} yield acc
/*input point for every new message*/
override def onMessage(message: Message) = message.text match {
case Some(text) if text == "start" => go(1, message.chat.id)
.unsafeRunSync
}
def main(args: Array[String]): Unit = HelloCountBot.run()
}
How to organize code to get the next message inside recursion from method onMessage with Unit return.
I didn't find in https://github.com/bot4s/telegram any methods for receiving messages in the way that you want. So I think the best option is to create stateful bot as shown in this example: https://github.com/bot4s/telegram/blob/master/examples/src/StatefulBot.scala.
So if I understand your code correctly, it can be reorganized in the following way (trait PerChatState is taken from the link above):
object HelloCountBot
extends TelegramBot
with Polling
with Commands
with PerChatState[Long] {
override def onMessage(message: Message): Future[Unit] = {
implicit val msg = message
message.text match {
case Some(text) if text == "start" =>
Future {
setChatState(0L)
}
case Some(value) if value == "0" =>
withChatState {
sum =>
reply(s"The sum is ${sum.getOrElse(0L)}")
.map(_ => clearChatState)
}
case Some(value) =>
withChatState {
mayBeSum =>
Future {
mayBeSum.foreach(
sum => setChatState(sum + value.toLong)
)
}
}
}
}
def main(args: Array[String]): Unit = HelloCountBot.run()
}
It uses Futures, but you can rewrite it to IO, if you prefer.
I have async Play Action, that retrieves data from the datbase, using Slick. And Slick, obviously, uses Futures to avoid blocking:
def show(id: Long) = Action.async {
db.run(entities.filter(_.id === id).result.headOption).map {
case None => templateFor("NEW_OBJECT")
case Some(x) => Ok(x)
}
def templateFor(code: String): Future[Result] = {
db.run(templates.filter(_.code === code).result.headOption).map {
case None => InternalServerError("No template")
case Some(x) => Ok(x)
}
}
The problem is that call to templateFor() returns Future, so the whole Action returns Future[Future[Result]] which is not what expected by Play. So, i would like to get rid of that nested Future. The simple way to do it is to Await for it's completion, but i would like to avoid unnecessary blocking. It would be nice if i would be able to take Future[Result] produced by templateFor() function and return it intact from my Action, thus replacing the outer Future with it.
You can use flatMap for that,
For any monandic strutcture such as Future[T], flatMap takes a function of type T => SomeOtherMonad[K], applies that function on all elements if monad and then flattens them to gives you Future[K].
def show(id: Long) = Action.async {
db.run(entities.filter(_.id === id).result.headOption).flatMap {
case None => templateFor("NEW_OBJECT")
case Some(x) => Future(Ok(x))
}
def templateFor(code: String): Future[Result] =
db.run(templates.filter(_.code === code).result.headOption).map {
case None => InternalServerError("No template")
case Some(x) => Ok(x)
}
}
My old code looks something like below, where all db calls blocking.
I need help converting this over to using Futures.
def getUserPoints(username: String): Option[Long]
db.getUserPoints(username) match {
case Some(userPoints) => Some(userPoints.total)
case None => {
if (db.getSomething("abc").isEmpty) {
db.somethingElse("asdf") match {
case Some(pointId) => {
db.setPoints(pointId, username)
db.findPointsForUser(username)
}
case _ => None
}
} else {
db.findPointsForUser(username)
}
}
}
}
My new API is below where I am returning Futures.
db.getUserPoints(username: String): Future[Option[UserPoints]]
db.getSomething(s: String): Future[Option[Long]]
db.setPoints(pointId, username): Future[Unit]
db.findPointsForUser(username): Future[Option[Long]]
How can I go about converting the above to use my new API that uses futures.
I tried using a for-compr but started to get wierd errors like Future[Nothing].
var userPointsFut: Future[Long] = for {
userPointsOpt <- db.getUserPoints(username)
userPoints <- userPointsOpt
} yield userPoints.total
But it gets a bit tricky with all the branching and if clauses and trying to convert it over to futures.
I would argue that the first issue with this design is that the port of the blocking call to a Future should not wrap the Option type:
The blocking call:
def giveMeSomethingBlocking(for:Id): Option[T]
Should become:
def giveMeSomethingBlocking(for:Id): Future[T]
And not:
def giveMeSomethingBlocking(for:Id): Future[Option[T]]
The blocking call give either a value Some(value) or None, the non-blocking Future version gives either a Success(value) or Failure(exception) which fully preserves the Option semantics in a non-blocking fashion.
With that in mind, we can model the process in question using combinators on Future. Let's see how:
First, lets refactor the API to something we can work with:
type UserPoints = Long
object db {
def getUserPoints(username: String): Future[UserPoints] = ???
def getSomething(s: String): Future[UserPoints] = ???
def setPoints(pointId:UserPoints, username: String): Future[Unit] = ???
def findPointsForUser(username: String): Future[UserPoints] = ???
}
class PointsNotFound extends Exception("bonk")
class StuffNotFound extends Exception("sthing not found")
Then, the process would look like:
def getUserPoints(username:String): Future[UserPoints] = {
db.getUserPoints(username)
.map(userPoints => userPoints /*.total*/)
.recoverWith{
case ex:PointsNotFound =>
(for {
sthingElse <- db.getSomething("abc")
_ <- db.setPoints(sthingElse, username)
points <- db.findPointsForUser(username)
} yield (points))
.recoverWith{
case ex: StuffNotFound => db.findPointsForUser(username)
}
}
}
Which type-checks correctly.
Edit
Given that the API is set in stone, a way to deal with nested monadic types is to define a MonadTransformer. In simple words, let's make Future[Option[T]] a new monad, let's call it FutureO that can be composed with other of its kind. [1]
case class FutureO[+A](future: Future[Option[A]]) {
def flatMap[B](f: A => FutureO[B])(implicit ec: ExecutionContext): FutureO[B] = {
val newFuture = future.flatMap{
case Some(a) => f(a).future
case None => Future.successful(None)
}
FutureO(newFuture)
}
def map[B](f: A => B)(implicit ec: ExecutionContext): FutureO[B] = {
FutureO(future.map(option => option map f))
}
def recoverWith[U >: A](pf: PartialFunction[Throwable, FutureO[U]])(implicit executor: ExecutionContext): FutureO[U] = {
val futOtoFut: FutureO[U] => Future[Option[U]] = _.future
FutureO(future.recoverWith(pf andThen futOtoFut))
}
def orElse[U >: A](other: => FutureO[U])(implicit executor: ExecutionContext): FutureO[U] = {
FutureO(future.flatMap{
case None => other.future
case _ => this.future
})
}
}
And now we can re-write our process preserving the same structure as the future-based composition.
type UserPoints = Long
object db {
def getUserPoints(username: String): Future[Option[UserPoints]] = ???
def getSomething(s: String): Future[Option[Long]] = ???
def setPoints(pointId: UserPoints, username:String): Future[Unit] = ???
def findPointsForUser(username: String): Future[Option[Long]] = ???
}
class PointsNotFound extends Exception("bonk")
class StuffNotFound extends Exception("sthing not found")
def getUserPoints2(username:String): Future[Option[UserPoints]] = {
val futureOpt = FutureO(db.getUserPoints(username))
.map(userPoints => userPoints /*.total*/)
.orElse{
(for {
sthingElse <- FutureO(db.getSomething("abc"))
_ <- FutureO(db.setPoints(sthingElse, username).map(_ => Some(())))
points <- FutureO(db.findPointsForUser(username))
} yield (points))
.orElse{
FutureO(db.findPointsForUser(username))
}
}
futureOpt.future
}
[1] with acknowledgements to http://loicdescotte.github.io/posts/scala-compose-option-future/
There're map/flatMap methods, there're also recover/recoverWith methods in the Scala Future standard API.
Why there's no collectWith ?
The code of the collect method is pretty simple :
def collect[S](pf: PartialFunction[T, S])(implicit executor: ExecutionContext): Future[S] =
map {
r => pf.applyOrElse(r, (t: T) => throw new NoSuchElementException("Future.collect partial function is not defined at: " + t))
}
The code of the collectWith method is then easy to imagine :
def collectWith[S](pf: PartialFunction[T, Future[S]])(implicit executor: ExecutionContext): Future[S] =
flatMap {
r => pf.applyOrElse(r, (t: T) => throw new NoSuchElementException("Future.collect partial function is not defined at: " + t))
}
I know that I can implement it and "extend" the Future standard API easily thanks to this article : http://debasishg.blogspot.fr/2008/02/why-i-like-scalas-lexically-scoped-open.html
I done that in my project :
class RichFuture[T](future: Future[T]) {
def collectWith[S](pf: PartialFunction[T, Future[S]])(implicit executor: ExecutionContext): Future[S] =
future.flatMap {
r => pf.applyOrElse(r, (t: T) => throw new NoSuchElementException("Future.collect partial function is not defined at: " + t))
}
}
trait WithRichFuture {
implicit def enrichFuture[T](person: Future[T]): RichFuture[T] = new RichFuture(person)
}
Maybe my needs for that does not justify to implement it in the standard API ?
Here is why I need this method in my Play2 project :
def createCar(key: String, eligibleCars: List[Car]): Future[Car] = {
def handleResponse: PartialFunction[WSResponse, Future[Car]] = {
case response: WSResponse if response.status == Status.CREATED => Future.successful(response.json.as[Car])
case response: WSResponse
if response.status == Status.BAD_REQUEST && response.json.as[Error].error == "not_the_good_one" =>
createCar(key, eligibleCars.tail)
}
// The "carApiClient.createCar" method just returns the result of the WS API call.
carApiClient.createCar(key, eligibleCars.head).collectWith(handleResponse)
}
I don't know how to do that without my collectWith method.
Maybe it's not the right way to do something like this ?
Do you know a better way ?
EDIT:
I have maybe a better solution for the createCar method that does not requires the collectWith method :
def createCar(key: String, eligibleCars: List[Car]): Future[Car] = {
for {
mayCar: Option[Car] <- Future.successful(eligibleCars.headOption)
r: WSResponse <- carApiClient.createCar(key, mayCar.get) if mayCar.nonEmpty
createdCar: Car <- Future.successful(r.json.as[Car]) if r.status == Status.CREATED
createdCar: Car <- createCar(key, eligibleCars.tail) if r.status == Status.BAD_REQUEST && r.json.as[Error].error == "not_the_good_one"
} yield createdCar
}
What do you think about this second solution ?
Second edit:
Just for information, here is my final solution thanks to #Dylan answer :
def createCar(key: String, eligibleCars: List[Car]): Future[Car] = {
def doCall(head: Car, tail: List[Car]): Future[Car] = {
carApiClient
.createCar(key, head)
.flatMap( response =>
response.status match {
case Status.CREATED => Future.successful(response.json.as[Car])
case Status.BAD_REQUEST if response.json.as[Error].error == "not_the_good_one" =>
createCar(key, tail)
}
)
}
eligibleCars match {
case head :: tail => doCall(head, tail)
case Nil => Future.failed(new RuntimeException)
}
}
Jules
How about:
def createCar(key: String, eligibleCars: List[Car]): Future[Car] = {
def handleResponse(response: WSResponse): Future[Car] = response.status match {
case Status.Created =>
Future.successful(response.json.as[Car])
case Status.BAD_REQUEST if response.json.as[Error].error == "not_the_good_one" =>
createCar(key, eligibleCars.tail)
case _ =>
// just fallback to a failed future rather than having a `collectWith`
Future.failed(new NoSuchElementException("your error here"))
}
// using flatMap since `handleResponse` is now a regular function
carApiClient.createCar(key, eligibleCars.head).flatMap(handleResponse)
}
Two changes:
handleResponse is no longer a partial function. The case _ returns a failed future, which is essentially what you were doing in your custom collectWith implementation.
use flatMap instead of collectWith, since handleResponse now suits that method signature
edit for extra info
If you really need to support the PartialFunction approach, you could always convert a PartialFunction[A, Future[B]] to a Function[A, Future[B]] by calling orElse on the partial function, e.g.
val handleResponsePF: PartialFunction[WSResponse, Future[Car]] = {
case ....
}
val handleResponse: Function[WSResponse, Future[Car]] = handleResponsePF orElse {
case _ => Future.failed(new NoSucheElementException("default case"))
}
Doing so would allow you to adapt an existing partial function to fit into a flatMap call.
(okay, technically, it already does, but you'd be throwing MatchErrors rather than your own custom exceptions)