Making multiple API calls in a functional way - scala

What would it be the best approach to solve this problem in the most functional (algebraic) way by using Scala and Cats (or maybe another library focused on Category Theory and/or functional programming)?
Resources
Provided we have the following methods which perform REST API calls to retrieve single pieces of information?
type FutureApiCallResult[A] = Future[Either[String, Option[A]]]
def getNameApiCall(id: Int): FutureApiCallResult[String]
def getAgeApiCall(id: Int): FutureApiCallResult[Int]
def getEmailApiCall(id: Int): FutureApiCallResult[String]
As you can see they produce asynchronous results. The Either monad is used to return possible errors during API calls and Option is used to return None whenever the resource is not found by the API (this case is not an error but a possible and desired result type).
Method to implement in a functional way
case class Person(name: String, age: Int, email: String)
def getPerson(id: Int): Future[Option[Person]] = ???
This method should used the three API calls methods defined above to asynchronously compose and return a Person or None if either any of the API calls failed or any of the API calls return None (the whole Person entity cannot be composed)
Requirements
For performance reasons all the API calls must be done in a parallel way
My guess
I think the best option would be to use the Cats Semigroupal Validated but I get lost when trying to deal with Future and so many nested Monads :S
Can anyone tell me how would you implement this (even if changing method signature or main concept) or point me to the right resources? Im quite new to Cats and Algebra in coding but I would like to learn how to handle this kind of situations so that I can use it at work.

The key requirement here is that it has to be done in parallel. It means that the obvious solution using a monad is out, because monadic bind is blocking (it needs the result in case it has to branch on it). So the best option is to use applicative.
I'm not a Scala programmer, so I can't show you the code, but the idea is that an applicative functor can lift functions of multiple arguments (a regular functor lifts functions of single argument using map). Here, you would use something like map3 to lift the three-argument constructor of Person to work on three FutureResults. A search for "applicative future in Scala" returns a few hits. There are also applicative instances for Either and Option and, unlike monads, applicatives can be composed together easily. Hope this helps.

You can make use of the cats.Parallel type class. This enables some really neat combinators with EitherT which when run in parallel will accumulate errors. So the easiest and most concise solution would be this:
type FutureResult[A] = EitherT[Future, NonEmptyList[String], Option[A]]
def getPerson(id: Int): FutureResult[Person] =
(getNameApiCall(id), getAgeApiCall(id), getEmailApiCall(id))
.parMapN((name, age, email) => (name, age, email).mapN(Person))
For more information on Parallel visit the cats documentation.
Edit: Here's another way without the inner Option:
type FutureResult[A] = EitherT[Future, NonEmptyList[String], A]
def getPerson(id: Int): FutureResult[Person] =
(getNameApiCall(id), getAgeApiCall(id), getEmailApiCall(id))
.parMapN(Person)

this is the only solution i came across with but still not satisfied because i have the feeling it could be done in a cleaner way
import cats.data.NonEmptyList
import cats.implicits._
import scala.concurrent.Future
case class Person(name: String, age: Int, email: String)
type FutureResult[A] = Future[Either[NonEmptyList[String], Option[A]]]
def getNameApiCall(id: Int): FutureResult[String] = ???
def getAgeApiCall(id: Int): FutureResult[Int] = ???
def getEmailApiCall(id: Int): FutureResult[String] = ???
def getPerson(id: Int): FutureResult[Person] =
(
getNameApiCall(id).map(_.toValidated),
getAgeApiCall(id).map(_.toValidated),
getEmailApiCall(id).map(_.toValidated)
).tupled // combine three futures
.map {
case (nameV, ageV, emailV) =>
(nameV, ageV, emailV).tupled // combine three Validated
.map(_.tupled) // combine three Options
.map(_.map { case (name, age, email) => Person(name, age, email) }) // wrap final result
}.map(_.toEither)

Personally I prefer to collapse all non-success conditions into the Future's failure. That really simplifies the error handling, like:
val futurePerson = for {
name <- getNameApiCall(id)
age <- getAgeApiCall(id)
email <- getEmailApiCall(id)
} yield Person(name, age, email)
futurePerson.recover {
case e: SomeKindOfError => ???
case e: AnotherKindOfError => ???
}
Note that this won't run the requests in parallel, to do so you'd need to move the future's creation outside of the for comprehension, like:
val futureName = getNameApiCall(id)
val futureAge = getAgeApiCall(id)
val futureEmail = getEmailApiCall(id)
val futurePerson = for {
name <- futureName
age <- futureAge
email <- futureEmail
} yield Person(name, age, email)

Related

Dynamically checking subclass relationship in Scala 3

I am trying to port a solution for DomainEventHandlers and -Dispatcher from PHP8 to Scala 3. Handlers should specify a list of events they can handle (in a type-safe way, preferably, by their classes). Handlers are dynamically registered with the Dispatcher, which should aggregate a map from the elements of the lists from each Handler to a List of Handlers for those events.
When an event is raised with the Dispatcher, it should check the class of the current event against the keys from the map, and pass the event to each Handler in each list of Handlers for a key if and only if the event's class is identical to or a subclass of the class specified by the key.
In dynamically typed OOP languages like PHP8, this is easy - a Handler stores a list of class-names, which can be reified simply by [ClassName]::class, then the Dispatcher gets the event's class via $event::class and performs an is_a-check for each HashMap-key, which checks both exact match and subclass-relationship.
In Scala 3, I can't seem to find a good way to do this. Working with underlying Java-reflections via getClass or Class[?] produces problems due to the mismatch between the Scala and Java type-systems (specifically, trailing $ being either present or not). In Scala 2, Tags would probably have been the way to go - but Scala 3 reflection is a different beast, and I have not found a way to utilize it to implement the above, and would appreciate advice.
Concretely, let's say we have
trait DomainEvent[D1 <: Serializable, D2 <: Serializable, A <: Aggregate]
extends Event[D1, D2]:
type AggregateType = A
val aggregateIdentifier: (String, UUID)
def applyAsPatch(aggregate: AggregateType): AggregateType
trait DomainEventHandler:
val handles: List[???]
def handle(event: DomainEvent[?, ?, ?]): ZIO[Any, Throwable, Unit]
object DomainEventDispatcher:
val registeredHandlers: scala.collection.mutable.Map[???, List[DomainEventHandler]] =
scala.collection.mutable.Map()
def registerHandler(handler: DomainEventHandler): Unit = ???
def raiseEvent(event: DomainEvent[?, ?, ?]): ZIO[Any, Throwable, Unit] = ???
I am unsure what to use in place of ??? in the DomainEventHandler's List and the Dispatcher's Map - the registerHandler and raiseEvent-implementations will follow from that.
Well, if your concrete event classes that you match aren't parametrized, it's pretty simple:
trait Event[A]
case class IntEvent(x: Int) extends Event[Int]
case class StringEvent(x: String) extends Event[String]
object Dispatcher {
var handlers = List.empty[PartialFunction[Event[_], String]]
def register(h: PartialFunction[Event[_], String]): Unit = { handlers = h :: handlers }
def dispatch(event: Event[_]) = handlers.flatMap { _.lift(event) }
}
Dispatcher.register { case e: IntEvent => s"Handled $e" }
Dispatcher.register {
case e: IntEvent => s"Handled ${e.x}"
case e: StringEvent => s"Handled ${e.x}"
}
Dispatcher.dispatch(new IntEvent()) // List(Handled 1, Handled IntEvent(1))
Dispatcher.dispatch(new StringEvent("foo")) // List(Handled foo)
But if you want to match on things like Event[Int], that makes things significantly more difficult. I wasn't able to find a good way to do it (though, I am by no means an expert in scala 3 features).
Not sure why they dropped ClassTag support ... I am taking it as a sign that matching on type parameters like this is no longer considered a good practice, and the "proper" solution to your problem is now naming all classes you want to match without type parameters.

Complex monad transformer for IO monad

I am trying to write a Cats MTL version of a function that would save an entity to a database. I want this function to read some SaveOperation[F[_]] from environment, execute it and handle possible failure. So far I came up with 2 version of this function: save is the more polymorphic MTL version and save2 uses exact monads in its signature, meaning that I confine myself to use of IO.
type SaveOperation[F[_]] = Employee => F[Int]
def save[F[_] : Monad](employee: Employee)(implicit
A: Ask[F, SaveOperation[F]],
R: Raise[F, AppError]): F[Unit] =
for {
s <- A.ask
rows <- s(employee)
res <- if rows != 1 then R.raise(FailedInsertion)
else ().pure[F]
} yield res
def save2(employee: Employee): Kleisli[IO, SaveOperation[IO], Either[AppError, Unit]] =
Kleisli((saveOperation) => saveOperation(employee)
.handleErrorWith(err => IO.pure(Left(PersistenceError(err))))
.map(rows =>
if rows != 1 then Left(FailedInsertion)
else Right(())
)
)
I can later call those like this:
val repo = new DoobieEmployeeRepository(xa)
val employee = Employee("john", "doe", Set())
type E[A] = Kleisli[IO, SaveOperation[IO], Either[AppError, A]]
println(EmployeeService.save[E](employee).run(repo.save).unsafeRunSync())
println(EmployeeService.save2(employee).run(repo.save).unsafeRunSync())
The problem is that for the call of save I get the following error:
Could not find an instance of Monad for E.
I found:
cats.data.Kleisli.catsDataMonadErrorForKleisli[F, A, E]
But method catsDataMonadErrorForKleisli in class KleisliInstances0_5 does not match type cats.Monad[E].
This error doesn't seem to make sense to me as effectively signatures are exactly the same for both function, so the monad should be there. I suspect the problem is with Ask[F, SaveOperation[F]] parameter as here F is not IO, while SaveOperation needs the IO.
Why can't I use the Kleisli monad for save call?
Update:
If I modify the type to type E[A] = EitherT[[X] =>> Kleisli[IO, SaveOperation[IO], X], AppError, A], I get a new error:
Could not find an implicit instance of Ask[E, SaveOperation[E]]
The right generic type for SaveOperation is supposed to be IO I guess, but I can't figure how to properly provide it through an instance of Ask
I hope you don't mind if I use this opportunity to do a quick tutorial on how to improve your question. It not only increases the chances of someone answering, but also might help you to find the solution yourself.
There are a couple of problems with the code you submitted, and I mean problems in terms of being a question on SO. Perhaps someone might have a ready answer just by looking at it, but let's say they don't, and they want to try it out in a worksheet. Turns out, your code has a lot of unnecessary stuff and doesn't compile.
Here are some steps you could take to make it better:
Strip away the unnecessary custom dependencies like Employee, DoobieEmployeeRepository, error types etc. and replace them with vanilla Scala types like String or Throwable.
Strip away any remaining code as long as you can still reproduce the problem. For example, the implementations of save and save2 are not needed, and neither are Ask and Raise.
Make sure that the code compiles. This includes adding the necessary imports.
By following these guidelines, we arrive at something like this:
import cats._
import cats.data.Kleisli
import cats.effect.IO
type SaveOperation[F[_]] = String => F[Int]
def save[F[_] : Monad](s: String)(): F[Unit] = ???
def save2(s: String): Kleisli[IO, SaveOperation[IO], Either[Throwable, Unit]] = ???
type E[A] = Kleisli[IO, SaveOperation[IO], Either[Throwable, A]]
println(save[E]("Foo")) // problem!
println(save2("Bar"))
That's already much better, because
a) it allows people to quickly try out your code, and
b) less code means less cognitive load and less space for problems.
Now, to check what's happening here, let's go to some docs:
https://typelevel.org/cats/datatypes/kleisli.html#type-class-instances
It has a Monad instance as long as the chosen F[_] does.
That's interesting, so let's try to further reduce our code:
type E[A] = Kleisli[IO, String, Either[Throwable, A]]
implicitly[Monad[E]] // Monad[E] doesn't exist
OK, but what about:
type E[A] = Kleisli[IO, String, A]
implicitly[Monad[E]] // Monad[E] exists!
This is the key finding. And the reason that Monad[E] doesn't exist in the first case is:
Monad[F[_]] expects a type constructor; F[_] is short for A => F[A] (note that this is actually Kleisli[F, A, A] :)). But if we try to "fix" the value type in Kleisli to Either[Throwable, A] or Option[A] or anything like that, then Monad instance doesn't exist any more. The contract was that we would provide the Monad typeclass with some type A => F[A], but now we're actually providing A => F[Either[Throwable, A]]. Monads don't compose so easily, which is why we have monad transformers.
EDIT:
After a bit of clarification, I think I know what you're going after now. Please check this code:
case class Employee(s: String, s2: String)
case class AppError(msg: String)
type SaveOperation[F[_]] = Employee => F[Int]
def save[F[_] : Monad](employee: Employee)(implicit
A: Ask[F, SaveOperation[F]],
R: Raise[F, AppError]): F[Unit] = for {
s <- A.ask
rows <- s(employee)
res <- if (rows != 1) R.raise(AppError("boom"))
else ().pure[F]
} yield res
implicit val askSaveOp = new Ask[IO, SaveOperation[IO]] {
override def applicative: Applicative[IO] =
implicitly[Applicative[IO]]
override def ask[E2 >: SaveOperation[IO]]: IO[E2] = {
val fun = (e: Employee) => IO({println(s"Saved $e!"); 1})
IO(fun)
}
}
implicit val raiseAppErr = new Raise[IO, AppError] {
override def functor: Functor[IO] =
implicitly[Functor[IO]]
override def raise[E2 <: AppError, A](e: E2): IO[A] =
IO.raiseError(new Throwable(e.msg))
}
save[IO](Employee("john", "doe")).unsafeRunSync() // Saved Employee(john,doe)!
I'm not sure why you expected Ask and Raise to already exist, they are referring to custom types Employee and AppError. Perhaps I'm missing something. So what I did here was that I implemented them myself, and I also got rid of your convoluted type E[A] because what you really want as F[_] is simply IO. There is not much point of having a Raise if you also want to have an Either. I think it makes sense to just have the code based around F monad, with Ask and Raise instances that can store the employee and raise error (in my example, error is raised if you return something other than 1 in the implementation of Ask).
Can you check if this is what you're trying to achieve? We're getting close. Perhaps you wanted to have a generic Ask defined for any kind of SaveOperation input, not just Employee? For what it's worth, I've worked with codebases like this and they can quickly blow up into code that's hard to read and maintain. MTL is fine, but I wouldn't want to go more generic than this. I might even prefer to pass the save function as a parameter rather than via Ask instance, but that's a personal preference.

In Scala Cats Validated, how can combine ordered validations

In https://gist.github.com/satyagraha/897e427bfb5ed203e9d3054ac6705704 I have posted a Scala Cats validation scenario which seems reasonable, but I haven't found a very neat solution.
Essentially, there is a two-stage validation, where individual fields are validated, then a class constructor is called which may throw due to internal checks (in general this may not be under my control to change, hence the exception handling code). We wish to not to call the constructor if any field validation fails, but also combine any constructor failure into the final result. "Fail-fast" is definitely right here for the two-phase check.
This is a kind of flatMap problem, which the cats.data.Validated framework appears to handle via the cats.data.Validated#andThenoperation. However I couldn't find a particularly neat solution to the problem as you can see in the code. There are quite a limited number of operations available on a cats.syntax.CartesianBuilder and is wasn't clear to me how to link it with the andThen operation.
Any ideas welcome! Note there is a Cats issue https://github.com/typelevel/cats/issues/1343 which possibly is related, not sure.
For fail fast, chained validation it is easier to use Either than Validated. You can easily switch from Either to Validated or vice versa depending if you want error accumulation.
A possible solution to your problem would be to create a smart constructor for User which returns an Either[Message, User] and use this with Validated[Message, (Name, Date)].
import cats.implicits._
import cats.data.Validated
def user(name: Name, date: Date): Either[Message, User] =
Either.catchNonFatal(User(name, date)).leftMap(Message.toMessage)
// error accumulation -> Validated
val valids: Validated[Message, (Name, Date)] =
(validateName(nameRepr) |#| validateDate(dateDepr)).tupled
// error short circuiting -> either
val userOrMessage: Either[Message, User] =
valids.toEither.flatMap((user _).tupled)
// Either[Message,User] = Right(User(Name(joe),Date(now)))
I would make a helper second-order function to wrap the exception-throwing ones:
def attempt[A, B](f: A => B): A => Validated[Message, B] = a => tryNonFatal(f(a))
Also, default companions of case classes extend the FunctionN trait, so there's no need to do (User.apply _).tupled, it can be shortened to User.tupled (on custom companions, you need to write extends ((...) => ...)) but apply override will be autogenerated)
So we end up with that using andThen:
val valids = validateName(nameRepr) |#| validateDate(dateDepr)
val res: Validated[Message, User] = valids.tupled andThen attempt(User.tupled)

why we need Free monad to interpret Action to Future

I wrote one example to use scalaz.Free to to map Action to Future, it looks pretty cool. However, I am trying to understand the benefits of it. I hope I can get the answer here. Here is my code snippet
Firstly, I create an Action, which is AST.
trait Action[A]
case class GetNumberAction(x: Int) extends Action[Int]
case class GetStringAction(x: String) extends Action[String]
case class ConvertToIntAction(x: String) extends Action[Int]
case class AddAction(x: Int, y: Int) extends Action[Int]
Then, I create a class to map Action to ASTMonad by using Scalaz Free and Coyonda.
type Functor[A] = Coyoneda[Action, A]
type ASTMonad[A]= Free[Functor, A]
def toMonad[A](action: Action[A]): ASTMonad[A] = Free.liftFC[Action, A](action)
object ADTMonad {
def getNumber(x: Int): ASTMonad[Int] = toMonad(GetNumberAction(x))
def getString(x: String): ASTMonad[String] = toMonad(GetStringAction(x))
def converToInt(x: String): ASTMonad[Int] = toMonad(ConvertToIntAction(x))
def add(x: Int, y: Int): ASTMonad[Int] = toMonad(AddAction(x, y))
}
At last, I create an Interpreter to interpret Action to Future
object Interpreter extends (Action ~> Future) {
def apply[A](action: Action[A]): Future[A] = {
action match {
case GetNumberAction(x) => Future(x)
case GetStringAction(x) => Future(x)
case ConvertToIntAction(x) => Future(x.toInt)
case AddAction(x, y) => Future(x + y)
}
}
}
When I run it, I can use
val chain = for {
number <- ASTMonad.getNumber(x)
str <- ASTMonad.getString(y)
convertedNumber <- ASTMonad.converToInt(str)
total <- ASTMonad.add(number, convertedNumber)
} yield total
chain.runWith(Interpreter)
It seems to work and I think I understand this monad and interpreter things. However, I am thinking what is the benefits comparing to the solution if I am using Future.flatmap and map directly ?
for {
number <- Future(x)
str <- Future(y)
convertedNumber <- Future(str.toInt)
total <- Future(number + convertedNumber)
} yield total
The code of using Future flatmap and map looks simpler to me. So back to my questions, do we need to use Free monad to interpret the business logic to Future, since Future has already provided flatMap and map. If it does, can someone give me more concrete example, so I can see the benefits ?
Thanks in advance
A good and motivated example for using free applicative are command-line parsers, let's call the type CLI[A].
A value of type CLI[A] means you will get an A if you provide command-line arguments (Array[String]) and they can be parsed successfully. Now this functionality is isomorphic to Array[String] -> Either[String,A] when using Either for error handling.
Because you made CLI applicative, you can map and apply (combine) values. You can for example create a Int argument count, another Int argument count2, and combine them to a final sum: CLI[Int] that holds their sum.
Suppose you apply the computation directly, this yields something that is "only" equivalent to Array[String] -> Either[String,Int]. But if you want to create a help text you have to know both initial arguments, and this information is lost.
Free to the rescue. With Free you can retain the computation graph, which you can use to extract all initial CLI values that are directly parsed from the arguments. You can then later run the computation which yields the final value of sum by providing the parse results for all initial arguments.
Of course you could implement a special CLI that keeps track of all the initial values over computations, but Free let's you avoid this extra work.

How can I combine fluent interfaces with a functional style in Scala?

I've been reading about the OO 'fluent interface' approach in Java, JavaScript and Scala and I like the look of it, but have been struggling to see how to reconcile it with a more type-based/functional approach in Scala.
To give a very specific example of what I mean: I've written an API client which can be invoked like this:
val response = MyTargetApi.get("orders", 24)
The return value from get() is a Tuple3 type called RestfulResponse, as defined in my package object:
// 1. Return code
// 2. Response headers
// 2. Response body (Option)
type RestfulResponse = (Int, List[String], Option[String])
This works fine - and I don't really want to sacrifice the functional simplicity of a tuple return value - but I would like to extend the library with various 'fluent' method calls, perhaps something like this:
val response = MyTargetApi.get("customers", 55).throwIfError()
// Or perhaps:
MyTargetApi.get("orders", 24).debugPrint(verbose=true)
How can I combine the functional simplicity of get() returning a typed tuple (or similar) with the ability to add more 'fluent' capabilities to my API?
It seems you are dealing with a client side API of a rest style communication. Your get method seems to be what triggers the actual request/response cycle. It looks like you'd have to deal with this:
properties of the transport (like credentials, debug level, error handling)
providing data for the input (your id and type of record (order or customer)
doing something with the results
I think for the properties of the transport, you can put some of it into the constructor of the MyTargetApi object, but you can also create a query object that will store those for a single query and can be set in a fluent way using a query() method:
MyTargetApi.query().debugPrint(verbose=true).throwIfError()
This would return some stateful Query object that stores the value for log level, error handling. For providing the data for the input, you can also use the query object to set those values but instead of returning your response return a QueryResult:
class Query {
def debugPrint(verbose: Boolean): this.type = { _verbose = verbose; this }
def throwIfError(): this.type = { ... }
def get(tpe: String, id: Int): QueryResult[RestfulResponse] =
new QueryResult[RestfulResponse] {
def run(): RestfulResponse = // code to make rest call goes here
}
}
trait QueryResult[A] { self =>
def map[B](f: (A) => B): QueryResult[B] = new QueryResult[B] {
def run(): B = f(self.run())
}
def flatMap[B](f: (A) => QueryResult[B]) = new QueryResult[B] {
def run(): B = f(self.run()).run()
}
def run(): A
}
Then to eventually get the results you call run. So at the end of the day you can call it like this:
MyTargetApi.query()
.debugPrint(verbose=true)
.throwIfError()
.get("customers", 22)
.map(resp => resp._3.map(_.length)) // body
.run()
Which should be a verbose request that will error out on issue, retrieve the customers with id 22, keep the body and get its length as an Option[Int].
The idea is that you can use map to define computations on a result you do not yet have. If we add flatMap to it, then you could also combine two computations from two different queries.
To be honest, I think it sounds like you need to feel your way around a little more because the example is not obviously functional, nor particularly fluent. It seems you might be mixing up fluency with not-idempotent in the sense that your debugPrint method is presumably performing I/O and the throwIfError is throwing exceptions. Is that what you mean?
If you are referring to whether a stateful builder is functional, the answer is "not in the purest sense". However, note that a builder does not have to be stateful.
case class Person(name: String, age: Int)
Firstly; this can be created using named parameters:
Person(name="Oxbow", age=36)
Or, a stateless builder:
object Person {
def withName(name: String)
= new { def andAge(age: Int) = new Person(name, age) }
}
Hey presto:
scala> Person withName "Oxbow" andAge 36
As to your use of untyped strings to define the query you are making; this is poor form in a statically-typed language. What is more, there is no need:
sealed trait Query
case object orders extends Query
def get(query: Query): Result
Hey presto:
api get orders
Although, I think this is a bad idea - you shouldn't have a single method which can give you back notionally completely different types of results
To conclude: I personally think there is no reason whatsoever that fluency and functional cannot mix, since functional just indicates the lack of mutable state and the strong preference for idempotent functions to perform your logic in.
Here's one for you:
args.map(_.toInt)
args map toInt
I would argue that the second is more fluent. It's possible if you define:
val toInt = (_ : String).toInt
That is; if you define a function. I find functions and fluency mix very well in Scala.
You could try having get() return a wrapper object that might look something like this
type RestfulResponse = (Int, List[String], Option[String])
class ResponseWrapper(private rr: RestfulResponse /* and maybe some flags as additional arguments, or something? */) {
def get : RestfulResponse = rr
def throwIfError : RestfulResponse = {
// Throw your exception if you detect an error
rr // And return the response if you didn't detect an error
}
def debugPrint(verbose: Boolean, /* whatever other parameters you had in mind */) {
// All of your debugging printing logic
}
// Any and all other methods that you want this API response to be able to execute
}
Basically, this allows you to put your response into a contain that has all of these nice methods that you want, and, if you simply want to get the wrapped response, you can just call the wrapper's get() method.
Of course, the downside of this is that you will need to change your API a bit, if that's worrisome to you at all. Well... you could probably avoid needing to change your API, actually, if you, instead, created an implicit conversion from RestfulResponse to ResponseWrapper and vice versa. That's something worth considering.