I'm trying to figure out how to write this piece of code in an elegant pure-functional style using scalaz7 IO and monad transformers but just can't get my head around it.
Just imagine I have this simple API:
def findUuid(request: Request): Option[String] = ???
def findProfile(uuid: String): Future[Option[Profile]] = redisClient.get[Profile](uuid)
Using this API I can easily write impure function with OptionT transformer like this:
val profileT = for {
uuid <- OptionT(Future.successful(findUuid(request)))
profile <- OptionT(findProfile(uuid))
} yield profile
val profile: Future[Option[Profile]] = profileT.run
As you have noticed - this function contains findProfile() with a side-effect. I want to isolate this effect inside of the IO monad and interpret outside of the pure function but don't know how to combine it all together.
def findProfileIO(uuid: String): IO[Future[Option[Profile]]] = IO(findProfile(uuid))
val profileT = for {
uuid <- OptionT(Future.successful(findUuid(request)))
profile <- OptionT(findProfileIO(uuid)) //??? how to put Option inside of the IO[Future[Option]]
} yield profile
val profile = profileT.run //how to run transformer and interpret IO with the unsafePerformIO()???
Any peaces of advice on how it might be done?
IO is meant more for synchronous effects. Task is more what you want!
See this question and answer: What's the difference between Task and IO in Scalaz?
You can convert your Future to Task and then have an API like this:
def findUuid(request: Request): Option[String] = ???
def findProfile(uuid: String): Task[Option[Profile]] = ???
This works because Task can represent both synchronous and asynchronous operations, so findUuid can also be wrapped in Task instead of IO.
Then you can wrap these in OptionT:
val profileT = for {
uuid <- OptionT(Task.now(findUuid(request)))
profile <- OptionT(findProfileIO(uuid))
} yield profile
Then at the end somewhere you can run it:
profileT.run.attemptRun
Check out this link for converting Futures to Tasks and vice versa: Scalaz Task <-> Future
End up with this piece of code, thought it might be useful for someone (Play 2.6).
Controller's method is a pure function since Task evaluation takes place outside of the controller inside of PureAction ActionBuilder. Thanks to Luka's answer!
Still struggling with new paradigm of Action composition in Play 2.6 though, but this is another story.
FrontendController.scala:
def index = PureAction.pure { request =>
val profileOpt = (for {
uuid <- OptionT(Task.now(request.cookies.get("uuid").map(t => uuidKey(t.value))))
profile <- OptionT(redis.get[Profile](uuid).asTask)
} yield profile).run
profileOpt.map { profileOpt =>
Logger.info(profileOpt.map(p => s"User logged in - $p").getOrElse("New user, suggesting login"))
Ok(views.html.index(profileOpt))
}
}
Actions.scala
Convenient action with Task resolution at the end
class PureAction #Inject()(parser: BodyParsers.Default)(implicit ec: ExecutionContext) extends ActionBuilderImpl(parser) {
self =>
def pure(block: Request[AnyContent] => Task[Result]): Action[AnyContent] = composeAction(new Action[AnyContent] {
override def parser: BodyParser[AnyContent] = self.parser
override def executionContext: ExecutionContext = self.ec
override def apply(request: Request[AnyContent]): Future[Result] = {
val taskResult = block(request)
taskResult.asFuture //End of the world lives here
}
})
}
Converters.scala
Task->Future and Future->Task implicit converters
implicit class FuturePimped[+T](root: => Future[T]) {
import scalaz.Scalaz._
def asTask(implicit ec: ExecutionContext): Task[T] = {
Task.async { register =>
root.onComplete {
case Success(v) => register(v.right)
case Failure(ex) => register(ex.left)
}
}
}
}
implicit class TaskPimped[T](root: => Task[T]) {
import scalaz._
val p: Promise[T] = Promise()
def asFuture: Future[T] = {
root.unsafePerformAsync {
case -\/(ex) => p.failure(ex); ()
case \/-(r) => p.success(r); ()
}
p.future
}
}
Related
I'm trying to share a Ref[F, A] between 2 concurrent streams. Below is a simplified example of the actual scenario.
class Container[F[_]](implicit F: Sync[F]) {
private val counter = Ref[F].of(0)
def incrementBy2 = counter.flatMap(c => c.update(i => i + 2))
def printCounter = counter.flatMap(c => c.get.flatMap(i => F.delay(println(i))))
}
In the main function,
object MyApp extends IOApp {
def run(args: List[String]): IO[ExitCode] = {
val s = for {
container <- Ref[IO].of(new Container[IO]())
} yield {
val incrementBy2 = Stream.repeatEval(
container.get
.flatTap(c => c.incrementBy2)
.flatMap(c => container.update(_ => c))
)
.metered(2.second)
.interruptScope
val printStream = Stream.repeatEval(
container.get
.flatMap(_.printCounter)
)
.metered(1.seconds)
incrementBy2.concurrently(printStream)
}
Stream.eval(s)
.flatten
.compile
.drain
.as(ExitCode.Success)
}
}
The updates made by the incrementBy2 are not visible in printStream.
How can I fix this?
I would appreciate any help to understand the mistake in this code.
Thanks
Your code is broken since the class definition, you are not even updating the same Ref
Remember that the point of IO is to be a description of a computation, so Ref[F].of(0) returns a program that when evaluated will create a new Ref, calling multiple flatMaps on it will result in multiple Refs being created.
Also, your is not doing tagless final in the right way (and some may argue that even the right way is not worth it: https://alexn.org/blog/2022/04/18/scala-oop-design-sample/)
This is how I would write your code:
trait Counter {
def incrementBy2: IO[Unit]
def printCounter: IO[Unit]
}
object Counter {
val inMemory: IO[Counter] =
IO.ref(0).map { ref =>
new Counter {
override final val incrementBy2: IO[Unit] =
ref.update(c => c + 2)
override final val printCounter: IO[Unit] =
ref.get.flatMap(IO.println)
}
}
}
object Program {
def run(counter: Counter): Stream[IO, Unit] =
Stream
.repeatEval(counter.printCounter)
.metered(1.second)
.concurrently(
Stream.repeatEval(counter.incrementBy2).metered(2.seconds)
).interruptAfter(10.seconds)
}
object Main extends IOApp.Simple {
override final val run: IO[Unit] =
Stream
.eval(Counter.inMemory)
.flatMap(Program.run)
.compile
.drain
}
PS: I would actually not have printCounter but getCounter
and do the printing in the Program
You can see the code running here.
Just trying to propagate my tracing context inside Kleisli as it was done originally in the next tutorial.
object TraceLogger {
def log(msg: String): Kleisli[IO, UUID, Unit] = Kleisli { traceId => IO(println(s"[$traceId] $msg")) }
}
trait ServiceStub {
def request(arg: String): Kleisli[IO, UUID, _]
}
trait ClientStub {
def get(arg: String): Kleisli[IO, UUID, _]
}
case class FirstServiceExample(clientStub: ClientStub) extends ServiceStub {
override def request(arg: String): Kleisli[IO, UUID, _] = Kleisli { (context: UUID) =>
val requestComputation = clientStub.get("calling second service!")
TraceLogger.log(arg)
requestComputation(context)
}
}
case class FirstClientExample(service: FirstServiceExample) {
def request(): IO[_] = {
val traceId = UUID.randomUUID()
service.request("root!").run(traceId)
}
}
And now I need to run execution:
val exampleClientStub = new ClientStub() {
override def get(arg: String): Kleisli[IO, UUID, _] = Kleisli.ask
}
val exampleClientService = FirstServiceExample(exampleClientStub)
FirstClientExample(exampleClientService).request().unsafeRunSync()
But, unfortunately, I don't see any logs here. Would you kindly help me to find an issue?
TraceLogger.log(arg) This returns an IO which is just a description of computation; it is doing nothing.
And since you just leave that value alone it is equivalent to just having a 1 in the middle of your code, it is simply discarded.
You need to chain your IOs together to create new IOs that represent "do this and then do that", that is basically what the flatMap method does.
Kleisli { (context: UUID) =>
val requestComputation = clientStub.get("calling second service!")
TraceLogger.log(arg)(context) >> // >> is equivalent to flatMap(_ => )
requestComputation(context)
}
(There is probably a better way to write this, I am not used to Kliesli)
Fabio's series on "Programas as Values" may be very useful: https://systemfw.org/archive.html
My old code looks something like below, where all db calls blocking.
I need help converting this over to using Futures.
def getUserPoints(username: String): Option[Long]
db.getUserPoints(username) match {
case Some(userPoints) => Some(userPoints.total)
case None => {
if (db.getSomething("abc").isEmpty) {
db.somethingElse("asdf") match {
case Some(pointId) => {
db.setPoints(pointId, username)
db.findPointsForUser(username)
}
case _ => None
}
} else {
db.findPointsForUser(username)
}
}
}
}
My new API is below where I am returning Futures.
db.getUserPoints(username: String): Future[Option[UserPoints]]
db.getSomething(s: String): Future[Option[Long]]
db.setPoints(pointId, username): Future[Unit]
db.findPointsForUser(username): Future[Option[Long]]
How can I go about converting the above to use my new API that uses futures.
I tried using a for-compr but started to get wierd errors like Future[Nothing].
var userPointsFut: Future[Long] = for {
userPointsOpt <- db.getUserPoints(username)
userPoints <- userPointsOpt
} yield userPoints.total
But it gets a bit tricky with all the branching and if clauses and trying to convert it over to futures.
I would argue that the first issue with this design is that the port of the blocking call to a Future should not wrap the Option type:
The blocking call:
def giveMeSomethingBlocking(for:Id): Option[T]
Should become:
def giveMeSomethingBlocking(for:Id): Future[T]
And not:
def giveMeSomethingBlocking(for:Id): Future[Option[T]]
The blocking call give either a value Some(value) or None, the non-blocking Future version gives either a Success(value) or Failure(exception) which fully preserves the Option semantics in a non-blocking fashion.
With that in mind, we can model the process in question using combinators on Future. Let's see how:
First, lets refactor the API to something we can work with:
type UserPoints = Long
object db {
def getUserPoints(username: String): Future[UserPoints] = ???
def getSomething(s: String): Future[UserPoints] = ???
def setPoints(pointId:UserPoints, username: String): Future[Unit] = ???
def findPointsForUser(username: String): Future[UserPoints] = ???
}
class PointsNotFound extends Exception("bonk")
class StuffNotFound extends Exception("sthing not found")
Then, the process would look like:
def getUserPoints(username:String): Future[UserPoints] = {
db.getUserPoints(username)
.map(userPoints => userPoints /*.total*/)
.recoverWith{
case ex:PointsNotFound =>
(for {
sthingElse <- db.getSomething("abc")
_ <- db.setPoints(sthingElse, username)
points <- db.findPointsForUser(username)
} yield (points))
.recoverWith{
case ex: StuffNotFound => db.findPointsForUser(username)
}
}
}
Which type-checks correctly.
Edit
Given that the API is set in stone, a way to deal with nested monadic types is to define a MonadTransformer. In simple words, let's make Future[Option[T]] a new monad, let's call it FutureO that can be composed with other of its kind. [1]
case class FutureO[+A](future: Future[Option[A]]) {
def flatMap[B](f: A => FutureO[B])(implicit ec: ExecutionContext): FutureO[B] = {
val newFuture = future.flatMap{
case Some(a) => f(a).future
case None => Future.successful(None)
}
FutureO(newFuture)
}
def map[B](f: A => B)(implicit ec: ExecutionContext): FutureO[B] = {
FutureO(future.map(option => option map f))
}
def recoverWith[U >: A](pf: PartialFunction[Throwable, FutureO[U]])(implicit executor: ExecutionContext): FutureO[U] = {
val futOtoFut: FutureO[U] => Future[Option[U]] = _.future
FutureO(future.recoverWith(pf andThen futOtoFut))
}
def orElse[U >: A](other: => FutureO[U])(implicit executor: ExecutionContext): FutureO[U] = {
FutureO(future.flatMap{
case None => other.future
case _ => this.future
})
}
}
And now we can re-write our process preserving the same structure as the future-based composition.
type UserPoints = Long
object db {
def getUserPoints(username: String): Future[Option[UserPoints]] = ???
def getSomething(s: String): Future[Option[Long]] = ???
def setPoints(pointId: UserPoints, username:String): Future[Unit] = ???
def findPointsForUser(username: String): Future[Option[Long]] = ???
}
class PointsNotFound extends Exception("bonk")
class StuffNotFound extends Exception("sthing not found")
def getUserPoints2(username:String): Future[Option[UserPoints]] = {
val futureOpt = FutureO(db.getUserPoints(username))
.map(userPoints => userPoints /*.total*/)
.orElse{
(for {
sthingElse <- FutureO(db.getSomething("abc"))
_ <- FutureO(db.setPoints(sthingElse, username).map(_ => Some(())))
points <- FutureO(db.findPointsForUser(username))
} yield (points))
.orElse{
FutureO(db.findPointsForUser(username))
}
}
futureOpt.future
}
[1] with acknowledgements to http://loicdescotte.github.io/posts/scala-compose-option-future/
I need to create an akka.stream.scaladsl.Source[T, Unit] from a collection of Future[T].
E.g., having a collection of futures returning integers,
val f1: Future[Int] = ???
val f2: Future[Int] = ???
val fN: Future[Int] = ???
val futures = List(f1, f2, fN)
how to create a
val source: Source[Int, Unit] = ???
from it.
I cannot use Future.sequence combinator, since then I would wait for each future to complete before getting anything from the source. I want to get results in any order as soon as any future completes.
I understand that Source is a purely functional API and it should not run anything before somehow materializing it. So, my idea is to use an Iterator (which is lazy) to create a source:
Source { () =>
new Iterator[Future[Int]] {
override def hasNext: Boolean = ???
override def next(): Future[Int] = ???
}
}
But that would be a source of futures, not of actual values. I could also block on next using Await.result(future) but I'm not sure which tread pool's thread will be blocked. Also this will call futures sequentially, while I need parallel execution.
UPDATE 2: it turned out there was a much easier way to do it (thanks to Viktor Klang):
Source(futures).mapAsync(1)(identity)
UPDATE: here is what I've got based on #sschaef answer:
def futuresToSource[T](futures: Iterable[Future[T]])(implicit ec: ExecutionContext): Source[T, Unit] = {
def run(actor: ActorRef): Unit = {
futures.foreach { future =>
future.onComplete {
case Success(value) =>
actor ! value
case Failure(NonFatal(t)) =>
actor ! Status.Failure(t) // to signal error
}
}
Future.sequence(futures).onSuccess { case _ =>
actor ! Status.Success(()) // to signal stream's end
}
}
Source.actorRef[T](futures.size, OverflowStrategy.fail).mapMaterializedValue(run)
}
// ScalaTest tests follow
import scala.concurrent.ExecutionContext.Implicits.global
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
"futuresToSource" should "convert futures collection to akka-stream source" in {
val f1 = Future(1)
val f2 = Future(2)
val f3 = Future(3)
whenReady {
futuresToSource(List(f1, f2, f3)).runFold(Seq.empty[Int])(_ :+ _)
} { results =>
results should contain theSameElementsAs Seq(1, 2, 3)
}
}
it should "fail on future failure" in {
val f1 = Future(1)
val f2 = Future(2)
val f3 = Future.failed(new RuntimeException("future failed"))
whenReady {
futuresToSource(List(f1, f2, f3)).runWith(Sink.ignore).failed
} { t =>
t shouldBe a [RuntimeException]
t should have message "future failed"
}
}
Creating a source of Futures and then "flatten" it via mapAsync:
scala> Source(List(f1,f2,fN)).mapAsync(1)(identity)
res0: akka.stream.scaladsl.Source[Int,Unit] = akka.stream.scaladsl.Source#3e10d804
One of the easiest ways to feed a Source is through an Actor:
import scala.concurrent.Future
import akka.actor._
import akka.stream._
import akka.stream.scaladsl._
implicit val system = ActorSystem("MySystem")
def run(actor: ActorRef): Unit = {
import system.dispatcher
Future { Thread.sleep(100); actor ! 1 }
Future { Thread.sleep(200); actor ! 2 }
Future { Thread.sleep(300); actor ! 3 }
}
val source = Source
.actorRef[Int](0, OverflowStrategy.fail)
.mapMaterializedValue(ref ⇒ run(ref))
implicit val m = ActorMaterializer()
source runForeach { int ⇒
println(s"received: $int")
}
The Actor is created through the Source.actorRef method and made available through the mapMaterializedValue method. run simply takes the Actor and sends all the completed values to it, which can then be accessed through source. In the example above, the values are sent directly in the Future, but this can of course be done everywhere (for example in the onComplete call on the Future).
I have a list of string ids representing DB records. I'd like to load them from the DB asynchronously, then upload each record to a remote server asynchronously, then when all are done uploading, make a record of the ids of the records that were uploaded.
Since I'm on Scala 2.9.2, I'm using Twitter's core-util Future implementation, but it should work exactly like the 2.10 futures in terms of Monadic transformations.
The general concept is this:
def fetch(id: String): Future[Option[Record]]
def upload(record: Record): Future[String]
def notifyUploaded(ids: Seq[String]): Unit
val ids: Seq[String] = ....
I'm trying to do this via a for comprehension but the fact that fetch returns a Future of Option makes it obscure and the code doesn't compile:
for {
id <- ids
maybeRecord <- fetch(id)
record <- maybeRecord
uploadedId <- upload(record)
} yield uploadedId
Compiling this results in the following error:
scala: type mismatch;
found : com.twitter.util.Future[String]
required: Option[?]
uploadedId <- upload(record)
^
What am I missing? why does the compiler expect uploadedId to be an Option? is there any pretty way I could work around this?
Consider the signature of the flatMap (or bind) function:
trait Monad[M[_]] {
def flatMap[A](a : M[A], f : A => M[B]) : M[B]
....
In your case, you're trying to use flatMap on an Option, giving it an f that generates a Future. But as in the signature above, f should be generating something in the same monad that it's been called on.
Scala isn't necessarily terribly helpful in this regard, since it's pretty good at converting things around (to Seqs, for example) in such a way that you get the impression that you can chain arbitrary flatMap calls together, regardless of the container.
What you possibly want is a 'Monad transformer', which gives you some ability to compose monads. Debasish Ghosh has a post on using Scalaz monad transformers here.
You cannot mix all different types in one for comprehension, I figured out that you might mix Seq and Option and result would be either Seq or Option depending on what is first. It is not possible to mix Future and Seq or Option. If you want to use for-comprehension you would have to cascade them few. In such cases it might be nicer with map/flatMap. I implemented your question in both ways and added types to few intermediate results so that you see the mess that is being created while working with all that different types.
object TestClass {
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent._
import scala.concurrent.duration._
case class Record(id: String)
def fetch(id: String): Future[Option[Record]] = Future {
Thread.sleep(1000);
Some(Record(id))
}
def upload(record: Record): Future[String] = Future {
Thread.sleep(3000);
record.id + "_uploaded"
}
def notifyUploaded(ids: Seq[String]): Unit = println("notified" + ids)
val ids: Seq[String] = Seq("a", "b", "c")
def main(args: Array[String]): Unit = {
forComprehensionImpl()
mapAndFlatMapImpl()
}
def forComprehensionImpl() = {
val result: Seq[Future[Option[Future[String]]]] = for {
id <- ids
} yield {
for {
maybeRecord <- fetch(id)
} yield {
for {
record <- maybeRecord
} yield {
for {
uploadedId <- upload(record)
} yield uploadedId
}
}
}
val result2: Future[Seq[Option[Future[String]]]] = Future.sequence(result)
val result3: Future[Unit] = result2.flatMap { x: Seq[Option[Future[String]]] =>
Future.sequence(x.flatten).map(notifyUploaded)
}
Await.result(result3, Duration.Inf)
}
def mapAndFlatMapImpl() = {
val res: Seq[Future[Iterable[String]]] = ids.map { id =>
fetch(id).flatMap { maybeRecord =>
val res1: Option[Future[Seq[String]]] = maybeRecord.map { record =>
upload(record) map (Seq(_))
}
res1 match {
case Some(a) => a
case None => Future(Seq())
}
}
}
val res3: Future[Unit] = Future.sequence(res) map (a => notifyUploaded(a.flatten))
Await.result(res3, Duration.Inf)
}
}