I want to execute List of functions which returns futures sequentially.
What could be the possible implementations for following method:
def runSequentially[A, B](lazyFutures: List[A ⇒ Future[B]])(input: A)
(implicit ec: ExecututionContext): Future[List[B]]
Test
test("execute futures sequentially") {
import scala.concurrent.ExecutionContext.Implicits.global
def runSequentially[A, B](lazyFutures: List[A ⇒ Future[B]])(input: A): Future[List[B]] = ???
val l: List[Unit ⇒ Future[Unit]] = List(
_ ⇒ Future { Thread.sleep(1000); println(1) },
_ ⇒ Future { Thread.sleep(5000); println(2) },
_ ⇒ Future { Thread.sleep(500); println(3) }
)
Await.result(runSequentially(l)(5), 20.seconds)
}
This should print:
1
2
3
Try creating a pool with one thread and use Future.traverse like so
implicit val singleThreadEc: ExecutionContext =
ExecutionContext.fromExecutor(Executors.newSingleThreadExecutor())
def runSequentially[A, B](lazyFutures: List[A ⇒ Future[B]])(input: A): Future[List[B]] =
Future.traverse(lazyFutures)(f => f(input))
def runSequentially[A, B](lazyFutures: Seq[A => Future[B]])(input: A)(implicit ctx: ExecutionContext): Future[List[B]] =
lazyFutures.foldLeft(Future.successful(List.empty[B])) { (futAcc, f) =>
futAcc.flatMap { acc =>
f(input).flatMap { result =>
result :: acc
}
}
}.map(_.reverse)
Should do the trick (haven't tested it). By prepending to the list and reversing, the complexity is O(n), while appending is O(n^2), since appending is O(n) and the append is done n times.
You can also use cats and Kleisli
import scala.concurrent.ExecutionContext.Implicits.global
import cats.implicits._
import cats.data.Kleisli
def runSequentially[A, B](lazyFutures: List[A ⇒ Future[B]])(input: A) : Future[List[B]] = lazyFutures
.map(Kleisli.apply) //List[Kleisli[Future, A, B]]
.sequence //Kleisli[Future, A, List[B]]
.run(input)
This is what I came up with which does work
def runSequentially[A, B](lazyFutures: List[A ⇒ Future[B]])(input: A): Future[List[B]] = {
lazyFutures.foldLeft(Future.successful(List.empty[B])) { (acc, curr) ⇒
for {
a ← acc
c ← curr(input)
} yield c :: a
}
}
Related
I have function that splits List[B] in half and returns tuple.
I need to release combinators for executing them in parallel.
If I do something like fl.map(x => x.map(...)) it is not parallel.
object Parallel {
implicit class ParList[A](fl: Future[List[A]]) {
private def split[B](l: List[B]): (List[B], List[B]) = l.splitAt(l.length / 2)
def paralFilter(f: A => Boolean): Future[List[A]] = ???
def paralFold(init: A)(f: (A, A) => A): Future[A] = ???
}
I understand that I must split list elementwise via split[B] and execute combinators with elements in parallel. How can I do it?
To achieve this use method par of scala.collection.Parallelizable :
object Parallel {
implicit class ParList[A](fl: Future[List[A]]) {
private def split[B](l: List[B]): (List[B], List[B]) = l.splitAt(l.length / 2)
def paralFilter(f: A => Boolean)(implicit ec:ExecutionContext): Future[List[A]] =
fl.map(_.par.filter(f).toList)
def paralFold(init: A)(f: (A, A) => A)(implicit ec:ExecutionContext): Future[A] =
fl.map(_.par.fold(init)(f))
}
}
I am trying to use parMapN function and I am not able to compile the code. If my type is IO then there is not problem, but when I use types on my functions, then I cannot manage to make it work.
In the snippet below, there is randomMessage which compiles and runs correctly but calling to randomMessageF does not compiles since there is not a NonEmptyParallel implicit on the scope. But then, which implicit is using randomMessage? Passing contextShift doest not work either.
import cats.NonEmptyParallel
import cats.effect._
import cats.syntax.all._
import fs2._
import scala.util.Random
object Test extends IOApp {
def randomMessageF[F[_], A, B, C](toA: => F[A],
toB: => F[B],
toC: (A, B) => C)(implicit nep: NonEmptyParallel[F, F]): Stream[F, C] = Stream.eval {
val funcA = toA
val funcB = toB
(funcA, funcB).parMapN {
case (a, b) =>
toC(a, b)
}
}
def randomMessage[A, B, C](toA: => IO[A],
toB: => IO[B],
toC: (A, B) => C): Stream[IO, C] = Stream.eval {
val funcA = toA
val funcB = toB
(funcA, funcB).parMapN {
case (a, b) =>
toC(a, b)
}
}
def run(args: List[String]): IO[ExitCode] = {
println(
randomMessage(
IO(Random.nextInt(1000).toString),
IO(Random.nextString(10)),
(k: String, v: String) => s"$k:$v"
).compile.toList.unsafeRunSync().head)
println(
randomMessageF[IO, String, String, String](
IO(Random.nextInt(1000).toString),
IO(Random.nextString(10)),
(k, v) => s"$k:$v"
)(???).compile.toList.unsafeRunSync().head)
IO(ExitCode(0))
}
}
Try
def randomMessageF[M[_], F[_], A, B, C](toA: => M[A],
toB: => M[B],
toC: (A, B) => C)(implicit
nep: NonEmptyParallel[M, F]): Stream[M, C] = Stream.eval {
val funcA = toA
val funcB = toB
(funcA, funcB).parMapN {
case (a, b) =>
toC(a, b)
}
}
println(
randomMessageF/*[IO, IO.Par, String, String, String]*/(
IO(Random.nextInt(1000).toString),
IO(Random.nextString(10)),
(k: String, v: String) => s"$k:$v"
).compile.toList.unsafeRunSync().head)
In randomMessage the implicit used is NonEmptyParallel[IO, IO.Par].
https://github.com/typelevel/cats-effect/blob/master/core/shared/src/main/scala/cats/effect/IO.scala#L834
I am looking at this question from Scala and Scalaz angles. OptionT works for Future but not Try. What is the reason there is no OptionT for Try where there is a usecase a function i.e. def foo(i: Int): Try[Option[Int]] = ... may or not return a value and occasionally network exception will happen? Thanks
The reason is Try isn't a valid functor.
You'll need to use scalaz-outlaws or write your own Try instances. Here's a working example using scalaz-outlaws' Try instances:
import scala.util.{Try,Success,Failure}
import scalaz._
import Scalaz._
implicit val tryOutlawInstances = new Traverse[Try] with Monad[Try] with Plus[Try]{
def point[A](a: ⇒ A): Try[A] = Success(a)
override def map[A,B](fa: Try[A])(f: A ⇒ B) = fa map f
def bind[A,B](fa: Try[A])(f: A ⇒ Try[B]) = fa flatMap f
def traverseImpl[F[_], A, B](fa: Try[A])(f: A ⇒ F[B])(implicit F: Applicative[F]) : F[Try[B]] = fa match {
case Success(a) ⇒ F.map(f(a))(Success.apply)
case Failure(f) ⇒ F.point(Failure(f))
}
def plus[A](a: Try[A], b: ⇒ Try[A]) = a orElse b
}
val foo = Try("foo".some)
val result = OptionT(foo).map(x => x.toUpperCase).run
I've used cats for the first time to solve day 1 of advent of code and I'm wondering if it's possible to improve things.
Given a method update with the following signature
def update(i: Instruction): PosAndDir => PosAndDir
I've come up with :
val state: State[PosAndDir, List[Unit]] = instructions.map(i => State.modify(update(i))).toList.sequenceU
val finalState = state.runS(PosAndDir(Pos(0, 0), North)).value
And also
def update2(i: Instruction): State[PosAndDir, Option[Pos]] =
State.modify(update(i)).inspect(pad => if (i == Walk) Some(pad.pos) else None)
…
val state = instructions.map(update2).toList.sequenceU
val positions = state.runA(PosAndDir(Pos(0, 0), North)).value.flatten
More precisely, questions are :
why do we need to call .value (with scalaz, it's transparent) ?
is there a way to write update2 with a for comprehension to improve readability ?
is there an Applicative instance for Seq in cats (I know there is not in scalaz). ?
any idea to improve the code ?
cats defines State[S, A] as an alias for stack-safe StateT[Eval, S , A] which is StateT[Trampoline, S, A] in scalaz terms, so runS returns Eval[A], where value will be run without stackoverflow even for very long flatMap sequences.
Using some more additional imports
import cats.data.{State, StateT}
import cats.MonadState
import cats.syntax.functorFilter._
import cats.instances.option._
and some preparations
type Walk[x] = StateT[Option, PosAndDir, x]
val stateMonad = MonadState[Walk, PosAndDir]
import stateMonad._
you can make your function look like this
def update2(i: Instruction): StateT[Option, PosAndDir, Pos] =
for (pad ← get if i == Walk) yield pad.pos
not that this solution will not work in 2.12 due to this improvement, you can make it work with this workaround
implicit class FunctorWithFilter[F[_] : FunctorFilter, A](fa: F[A]) {
def withFilter(f: A ⇒ Boolean) = fa.filter(f)
}
There is no instances for Seq, this answer describes why. While there are some non-orthodox instances in the alleycats project. I'm not really sure if you need for Applicative[Seq], from your code you are rather have need for Traverse[Seq], or if you replace your sequence with sequence_ even Foldable[Seq].
Good news there is Foldable[Iterable] in the alleycats, and here is my attempt to define something lookalike for Seq instance
implicit val seqInstance = new MonadFilter[Seq] with Traverse[Seq] {
def traverse[G[_] : Applicative, A, B](fa: Seq[A])(f: (A) ⇒ G[B]): G[Seq[B]] =
fa match {
case head +: tail ⇒ f(head).map2(traverse(tail)(f))(_ +: _)
case _empty ⇒ Seq.empty[B].pure[G]
}
def foldLeft[A, B](fa: Seq[A], b: B)(f: (B, A) ⇒ B): B = fa.foldLeft(b)(f)
def foldRight[A, B](fa: Seq[A], lb: Eval[B])(f: (A, Eval[B]) ⇒ Eval[B]): Eval[B] =
fa match {
case head +: tail ⇒ f(head, foldRight(tail, lb)(f))
case _empty ⇒ lb
}
def pure[A](x: A): Seq[A] = Seq(x)
def empty[A]: Seq[A] = Seq.empty[A]
def flatMap[A, B](fa: Seq[A])(f: (A) ⇒ Seq[B]): Seq[B] = fa.flatMap(f)
def tailRecM[A, B](a: A)(f: (A) ⇒ Seq[Either[A, B]]): Seq[B] = {
#tailrec def go(seq: Seq[Either[A, B]]): Seq[B] =
if (seq.contains((_: Either[A, B]).isLeft))
go(seq.flatMap {
case Left(a) ⇒ f(a)
case b ⇒ Seq(b)
}) else seq.collect { case Right(b) ⇒ b }
go(Seq(Left(a)))
}
override def mapFilter[A, B](fa: Seq[A])(f: (A) ⇒ Option[B]): Seq[B] =
fa.flatMap(f(_).toSeq)
}
didn't spent much time but here is my attempt to simplifying some parts via the Monocle library:
import cats.{MonadState, Foldable, Functor}
import cats.instances.option._
import cats.syntax.foldable._
import cats.syntax.functor._
import cats.syntax.functorFilter._
import monocle.macros.Lenses
#Lenses
case class Pos(x: Int, y: Int)
sealed abstract class Dir(val cmd: Pos ⇒ Pos)
case object South extends Dir(Pos.y.modify(_ - 1))
case object North extends Dir(Pos.y.modify(_ + 1))
case object East extends Dir(Pos.x.modify(_ + 1))
case object West extends Dir(Pos.x.modify(_ - 1))
#Lenses
case class PosAndDir(pos: Pos, dir: Dir)
val clockwise = Vector(North, East, South, West)
val right: Map[Dir, Dir] = clockwise.zip(clockwise.tail :+ clockwise.head).toMap
val left: Map[Dir, Dir] = right.map(_.swap)
sealed abstract class Instruction(val cmd: PosAndDir ⇒ PosAndDir)
case object TurnLeft extends Instruction(PosAndDir.dir.modify(left))
case object TurnRight extends Instruction(PosAndDir.dir.modify(right))
case object Walk extends Instruction(pd ⇒ PosAndDir.pos.modify(pd.dir.cmd)(pd))
def runInstructions[F[_] : Foldable : Functor](instructions: F[Instruction])(start: PosAndDir): PosAndDir =
instructions.map(i => State.modify(i.cmd)).sequence_.runS(start).value
I occasionally hit code like this:
val things : List[A \/ B] = ???
val (as, bs) : (List[A], List[B]) = ??? //insert something to do this
or in my current case I want Map[A, B \/ C] => (Map[A, B], Map[A, C])
Is there a nice way to do this in the general case F[A \/ B] with appropriate restrictions on F? It looks vaguely like a variation on the theme of Unzip.
Here's how we deal with this for / but also Either and Validation, and not just for Lists, but other Foldable.
object Uncozip {
implicit val wtf = language.higherKinds
// Typeclass which covers sum types such as \/, Either, Validation
trait Sum2[F[_, _]] {
def cata[A, B, X](a: A ⇒ X, b: B ⇒ X)(fab: F[A, B]): X
}
implicit val sumEither: Sum2[Either] = new Sum2[Either] {
def cata[A, B, X](a: A ⇒ X, b: B ⇒ X)(fab: Either[A, B]): X = {
fab match {
case Left(l) ⇒ a(l)
case Right(r) ⇒ b(r)
}
}
}
implicit val sumEitherz: Sum2[\/] = new Sum2[\/] {
def cata[A, B, X](a: A ⇒ X, b: B ⇒ X)(fab: A \/ B): X = {
fab.fold(a(_), b(_))
}
}
implicit val sumValidation: Sum2[Validation] = new Sum2[Validation] {
def cata[A, B, X](a: A ⇒ X, b: B ⇒ X)(fab: A Validation B): X = {
fab.fold(a(_), b(_))
}
}
abstract class Uncozips[F[_], G[_, _], A, B](fab: F[G[A, B]]) {
def uncozip: (F[A], F[B])
}
implicit def uncozip[F[_]: Foldable, G[_, _], A, B](fab: F[G[A, B]])(implicit g: Sum2[G], mfa: ApplicativePlus[F], mfb: ApplicativePlus[F]): Uncozips[F, G, A, B] = new Uncozips[F, G, A, B](fab) {
def uncozip = {
implicitly[Foldable[F]].foldRight[G[A, B], (F[A], F[B])](fab, (mfa.empty, mfb.empty)) { (l, r) ⇒
g.cata[A, B, (F[A], F[B])]({ (a: A) ⇒ (mfa.plus(mfa.point(a), r._1), r._2) },
{ (b: B) ⇒ (r._1, mfa.plus(mfa.point(b), r._2)) })(l)
}
}
}
}
You can map things in to a list of (Option[A], Option[B]), unzip that list in to two lists, and then unite the resulting lists:
import scalaz._
import Scalaz._
val things: List[String \/ Int] = List("foo".left, 42.right)
val (strs, ints): (List[String], List[Int]) = things.
map { d => (d.swap.toOption, d.toOption) }. // List[(Option[String], Option[Int])]
unzip. // (List[Option[String]], List[Option[Int]])
bimap(_.unite, _.unite) // (List[String], List[Int])
This isn't particularly efficient due to traversing the list three times.
Here is one way (for lists):
val things : List[A \/ B] = ???
val (as, bs) = (things.map(_.swap.toList).join, things.map(_.toList).join)
And for a map:
val things: Map[String, String \/ Int] = ???
val (as, bs) = (things.mapValues(_.swap.toList).filterNot(e => e._2.isEmpty),
things.mapValues(_.toList).filterNot(e => e._2.isEmpty))
I'm having a hard time coming up with a way to generalize this over any F (I believe you would need instances of Monoid and Applicative for F).