How to make sure a given future completes first in tests? - scala

I'm writing tests for function bar:
def bar(fut1: Future[Int],
fut2: Future[Int],
fut3: Future[Int]): Future[Result] = ???
bar returns Result like this:
case class Result(
x: Int, // fut1 value
oy: Option[Int], // if fut2 is complete then Some of fut2 value else None
oz: Option[Int] // if fut3 is complete then Some of fut3 value else None
)
I want to write tests for all test cases:
fut1 completed, fut2 and fut3 did not complete
fut1 completed, fut2 completed, fut3 did not complete
etc.
So I am writing a fake implementation of functions foo1, foo2, and foo3 for these tests.
def foo1(x: Int): Future[Int] = ???
def foo2(x: Int): Future[Int] = ???
def foo3(x: Int): Future[Int] = ???
Test #1 invokes all these functions, checks if fut1 completes first, and invokes bar
val fut1 = foo1(0)
val fut2 = foo2(0)
val fut3 = foo3(0)
// make sure `fut1` completes first
Test #2 invokes all these functions, makes sure that fut2 completes first, and invokes bar.
Test #3 invokes all these functions, makes sure that fut3 completes first, and invokes bar.
My question is how to implement the functions foo1, foo2, and foo3 and the tests.

You can try attach completeness timestamp to each future via map, like:
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent._
import scala.concurrent.duration._
import scala.language.postfixOps
def foo1(x: Int): Future[Int] = Future {Thread.sleep(200); 1}
def foo2(x: Int): Future[Int] = Future {Thread.sleep(500); 2}
def foo3(x: Int): Future[Int] = Future {Thread.sleep(500); 3}
def completeTs[T](future: Future[T]): Future[(T, Long)] = future.map(v => v -> System.currentTimeMillis())
val resutls = Await.result(Future.sequence(
List(completeTs(foo1(1)), completeTs(foo2(1)), completeTs(foo3(1)))
), 2 seconds)
val firstCompleteTs = resutls.map(_._2).min
val firstCompleteIndex = resutls.indexWhere(_._2 == firstCompleteTs)
assert(firstCompleteIndex == 0)
Scatie: https://scastie.scala-lang.org/L9g78DSNQIm2K1jGlQzXBg

You could repurpose firstCompletedOf to verify whether the future of a given index in the futures list is the first completed one:
import java.util.concurrent.atomic.AtomicReference
import scala.concurrent.{ExecutionContext, Future, Promise}
import scala.util.Try
def isFirstCompleted[T](idx: Int)(futures: List[Future[T]])(
implicit ec: ExecutionContext): Future[Boolean] = {
val promise = Promise[(T, Int)]()
val pRef = new AtomicReference[Promise[(T, Int)]](promise)
futures.zipWithIndex foreach { case (f, i) => f onComplete { case tt: Try[T] =>
val p = pRef.getAndSet(null)
if (p != null) p tryComplete tt.map((_, i))
}
}
promise.future.map{ case (t, i) => i == idx }
}
Test running:
import scala.concurrent.ExecutionContext.Implicits.global
val futures = List(
Future{Thread.sleep(100); 1},
Future{Thread.sleep(150); throw new Exception("oops!")},
Future{Thread.sleep(50); 3}
)
isFirstCompleted(0)(futures) // Future(Success(false))
isFirstCompleted(2)(futures) // Future(Success(true))
For writing test cases, consider using ScalaTest AsyncFlatSpec.

It is unclear what it is exactly you are trying to test.
If you simply use futures that have already completed, you will get the behavior you describe:
def f1 = Future.successful(1)
def f2 = Future.successful(2)
def f3 = Future.successful(3)
eventually {
Future.firstCompletedOf(Seq(f1, f2, f3)).value shouldBe Some(1)
}
(note, that you cannot compare directly with fut1 like you did in the question, that'll always be false, because .firstCompletedOf returns a new future).
You can also make only one future complete, and leave the others alone:
val promise = Promise[Int].future
def f1 = promise.future // or just Future.successful(1) ... or Future(1)
def f2 = Future.never
def f3 = Future.never
result = Future.firstCompletedOf(Seq(f1, f2, f3))
promise.complete(Success(1))
eventually {
result.value shouldBe 1
}
Etc ... Can make the other futures be backed by their own promise too for example, if you want them all to complete eventually (not sure what it'll gain you, but then again, I am not sure what you are testing here to begin with).
Another possibility is make them depend on each other:
val promise = Promise[Int]
def f1 = promise.future
def f2 = promise.future.map(_ + 1)
def f3 = promise.future.map(_ + 2)
...
promise.complete(Success(1))

Related

Does cats mapN run all the futures in parallel?

As mentioned in the jump-start guide, mapN will run all the futures in parallel, so I created the below simple Scala program, but a sample run shows diff to be 9187 ms and diffN to be 9106 ms. So it looks like that the mapN is also running the futures sequentially, isn't it? Please let me know if I am missing something?
package example
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
import java.time.LocalDateTime
import java.time.Duration
import scala.util.Failure
import scala.util.Success
import java.time.ZoneOffset
import cats.instances.future._
import cats.syntax.apply._
object FutureEx extends App {
val before = LocalDateTime.now()
val sum = for {
a <- getA
b <- getB
c <- getC
} yield (a + b + c)
sum onComplete {
case Failure(ex) => println(s"Error: ${ex.getMessage()}")
case Success(value) =>
val after = LocalDateTime.now()
println(s"before: $before")
println(s"after: $after")
val diff = getDiff(before, after)
println(s"diff: $diff")
println(s"sum: $value")
}
// let the above finish
Thread.sleep(20000)
val beforeN = LocalDateTime.now()
val usingMapN = (getA, getB, getC).mapN(add)
usingMapN onComplete {
case Failure(ex) => println(s"Error: ${ex.getMessage()}")
case Success(value) =>
val afterN = LocalDateTime.now()
println(s"beforeN: $beforeN")
println(s"afterN: $afterN")
val diff = getDiff(beforeN, afterN)
println(s"diffN: $diff")
println(s"sum: $value")
}
def getA: Future[Int] = {
println("inside A")
Thread.sleep(3000)
Future.successful(2)
}
def getB: Future[Int] = {
println("inside B")
Thread.sleep(3000)
Future.successful(3)
}
def getC: Future[Int] = {
println("inside C")
Thread.sleep(3000)
Future.successful(4)
}
def add(a: Int, b: Int, c: Int) = a + b + c
def getDiff(before: LocalDateTime, after: LocalDateTime): Long = {
Duration.between(before.toInstant(ZoneOffset.UTC), after.toInstant(ZoneOffset.UTC)).toMillis()
}
}
Because you have sleep outside Future it should be like:
def getA: Future[Int] = Future {
println("inside A")
Thread.sleep(3000)
2
}
So you start async Future with apply - Future.successful on the other hand returns pure value, meaning you execute sleep in same thread.
The time is going before mapN is ever called.
(getA, getB, getC).mapN(add)
This expression is creating a tuple (sequentially) and then calling mapN on it. So it is calling getA then getB then getC and since each of them has a 3 second delay, it takes 9 seconds.

Specifying an execution context for Monad[Future] when using EitherT in Scalaz 7

I've been trying to tidy up some code which uses multiple functions that all return the type Future[Either[String, A]].
These functions don't compose neatly in a for comprehension because of the problem of having to peak inside the Future and then inside the Either to get the value. After using an EitherT monad transformer I found a solution I liked using EitherT, even though having to add eitherT and having to have the extra step of calling 'run' when you get the final result is not ideal.
My solution is below, but there's one thing I'm not happy with is that you need to create a Monad[Future] in order for eitherT to work, and this needs an execution context. There's no obvious way to do that. What I've done is to have an implicit execution context in scope of my code and create a Future Monad that I pass the same execution context to so that it both pieces of code use the same one. This seems a little messy, error prone.
Please let me know if there's a better way.
/*
Example of EitherT in ScalaZ
val scalaZVersion = "7.2.8"
"org.scalaz" %% "scalaz-core" % scalaZVersion,
"org.scalaz" %% "scalaz-effect" % scalaZVersion,
*/
import java.util.concurrent.Executors
import scala.concurrent.duration._
import org.scalatest._
import scala.concurrent.{Await, ExecutionContext, Future}
import scalaz.{-\/, Monad, \/, \/-}
import scalaz.EitherT.eitherT
object MonadFutureUtil {
// a Future Monad with a specific instance of an EC
case class MonadWithExecutionContext()(implicit ec : ExecutionContext) extends Monad[Future] {
def point[A](a: => A): Future[A] = Future(a)
def bind[A, B](fa: Future[A])(f: (A) => Future[B]): Future[B] = fa flatMap f
}
}
class TestFutureUtil extends FlatSpec with Matchers with OptionValues with Inside with Inspectors {
implicit val ec = new ExecutionContext {
implicit val threadPool = Executors.newFixedThreadPool(8)
def execute(runnable: Runnable) {
threadPool.submit(runnable)
}
def reportFailure(t: Throwable): Unit = {
println(s"oh no! ${t.getMessage}")
}
}
implicit val monadicEC = MonadFutureUtil.MonadWithExecutionContext()(ec)
// halves the input if it is even else fails
def dummyFunction1(n: Int)(implicit ec : ExecutionContext) : Future[\/[String, Int]] = {
Future.successful(
if(n % 2 == 0)
\/-(n / 2)
else
-\/("An odd number")
)
}
// appends a suffix to the input after converting to a string
// it doesn't like numbers divisible by 3 and 7 though
def dummyFunction2(n: Int)(implicit ec : ExecutionContext) : Future[\/[String, String]] = {
Future.successful(
if(n % 3 != 0 && n % 7 != 0)
\/-(n.toString + " lah!")
else
-\/(s"I don't like the number $n")
)
}
"EitherFuture" should "add the results of two dummyFunction1 calls" in {
val r = for (
rb1 <- eitherT(dummyFunction1(8));
rb2 <- eitherT(dummyFunction1(12))
) yield (rb1 + rb2)
r.run.map {
_ shouldBe \/-(11)
}
}
it should "handle functions with different type" in {
val r = for (
rb1 <- eitherT(dummyFunction1(14));
rb2 <- eitherT(dummyFunction1(12));
rb3 <- eitherT(dummyFunction2(rb2 + rb1))
) yield rb3
val r2 = Await.result(r.run.map {
case \/-(s) =>
(s == "13 lah!")
case -\/(e) =>
false
}, 5 seconds)
assert(r2)
}
it should "doesn't like divisible by 7" in {
val r = for (
rb1 <- eitherT(dummyFunction1(14));
rb2 <- eitherT(dummyFunction1(14));
rb3 <- eitherT(dummyFunction2(rb1 + rb2))
) yield rb3
val r2 = Await.result(r.run.map {
case \/-(s) =>
false
case -\/(e) =>
true
}, 5 seconds)
assert(r2)
}
}
I would suggest trying the following instead of a case class:
implicit def MWEC(implicit ec: ExecutionContext): Monad[Future] = ???
This way it should be harder to mix up execution contexts. The proper way would be to use a pure IO abstraction, one that doesn't require a execution context to be mapped/flatmapped over...

Left flatMap on EitherT

Let's say I have functions which return Future[Either[_, _] and I want to apply some of these functions in case of failures, that means apply them only to left side. The simplified example is:
def operation1: Future[Either[String, Int]] = Future.successful(Right(5))
def operation2: Future[Either[String, Int]] = Future.successful(Left("error"))
def operation2FallBackWork = Future.successful{
println("Doing some revert stuff")
Left("Error happened, but reverting was successful")
}
val res = for {
res1 <- EitherT.fromEither(operation1)
res2 <- EitherT.fromEither(operation2)//.leftFlatMap(operation2FallBackWork) -????
} yield res1 + res2
Await.result(res.toEither, 5 seconds)
How to achieve that?
The closest thing to a leftFlatMap is MonadError's handleError, which has exactly the signature you'd expect from something called leftFlatMap (although note that you'll need to change the fallback operation to an EitherT and provide a constant function instead of passing it as-is). You can use the EitherT instance directly like this:
import scala.concurrent.{ Await, Future }
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import scalaz._, Scalaz._
def operation1: Future[Either[String, Int]] = Future.successful(Right(5))
def operation2: Future[Either[String, Int]] = Future.successful(Left("error"))
def operation2FallBack: EitherT[Future, String, Int] = EitherT(
Future.successful {
println("Doing some revert stuff")
"Error happened, but reverting was successful".left
}
)
val E: MonadError[({ type L[x] = EitherT[Future, String, x] })#L, String] =
implicitly
val res = for {
a <- EitherT.fromEither(operation1)
b <- E.handleError(EitherT.fromEither(operation2))(_ => operation2FallBack)
} yield a + b
Await.result(res.toEither, 5.seconds)
You can also use the syntax provided by MonadError to make it look like EitherT has a handleError method, although it takes a bit more ceremony to get the Scala compiler to recognize that your operations have the right shape:
import scala.concurrent.{ Await, Future }
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import scalaz._, Scalaz._
type FE[x] = EitherT[Future, String, x]
def operation1: FE[Int] = EitherT(Future.successful(5.right))
def operation2: FE[Int] = EitherT(Future.successful("error".left))
def operation2FallBack: FE[Int] = EitherT(
Future.successful {
println("Doing some revert stuff")
"Error happened, but reverting was successful".left
}
)
val res = for {
a <- operation1
b <- operation2.handleError(_ => operation2FallBack)
} yield a + b
Await.result(res.toEither, 5.seconds)
I'd prefer this second version, but it's a matter of style and taste.

In akka-stream how to create a unordered Source from a futures collection

I need to create an akka.stream.scaladsl.Source[T, Unit] from a collection of Future[T].
E.g., having a collection of futures returning integers,
val f1: Future[Int] = ???
val f2: Future[Int] = ???
val fN: Future[Int] = ???
val futures = List(f1, f2, fN)
how to create a
val source: Source[Int, Unit] = ???
from it.
I cannot use Future.sequence combinator, since then I would wait for each future to complete before getting anything from the source. I want to get results in any order as soon as any future completes.
I understand that Source is a purely functional API and it should not run anything before somehow materializing it. So, my idea is to use an Iterator (which is lazy) to create a source:
Source { () =>
new Iterator[Future[Int]] {
override def hasNext: Boolean = ???
override def next(): Future[Int] = ???
}
}
But that would be a source of futures, not of actual values. I could also block on next using Await.result(future) but I'm not sure which tread pool's thread will be blocked. Also this will call futures sequentially, while I need parallel execution.
UPDATE 2: it turned out there was a much easier way to do it (thanks to Viktor Klang):
Source(futures).mapAsync(1)(identity)
UPDATE: here is what I've got based on #sschaef answer:
def futuresToSource[T](futures: Iterable[Future[T]])(implicit ec: ExecutionContext): Source[T, Unit] = {
def run(actor: ActorRef): Unit = {
futures.foreach { future =>
future.onComplete {
case Success(value) =>
actor ! value
case Failure(NonFatal(t)) =>
actor ! Status.Failure(t) // to signal error
}
}
Future.sequence(futures).onSuccess { case _ =>
actor ! Status.Success(()) // to signal stream's end
}
}
Source.actorRef[T](futures.size, OverflowStrategy.fail).mapMaterializedValue(run)
}
// ScalaTest tests follow
import scala.concurrent.ExecutionContext.Implicits.global
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
"futuresToSource" should "convert futures collection to akka-stream source" in {
val f1 = Future(1)
val f2 = Future(2)
val f3 = Future(3)
whenReady {
futuresToSource(List(f1, f2, f3)).runFold(Seq.empty[Int])(_ :+ _)
} { results =>
results should contain theSameElementsAs Seq(1, 2, 3)
}
}
it should "fail on future failure" in {
val f1 = Future(1)
val f2 = Future(2)
val f3 = Future.failed(new RuntimeException("future failed"))
whenReady {
futuresToSource(List(f1, f2, f3)).runWith(Sink.ignore).failed
} { t =>
t shouldBe a [RuntimeException]
t should have message "future failed"
}
}
Creating a source of Futures and then "flatten" it via mapAsync:
scala> Source(List(f1,f2,fN)).mapAsync(1)(identity)
res0: akka.stream.scaladsl.Source[Int,Unit] = akka.stream.scaladsl.Source#3e10d804
One of the easiest ways to feed a Source is through an Actor:
import scala.concurrent.Future
import akka.actor._
import akka.stream._
import akka.stream.scaladsl._
implicit val system = ActorSystem("MySystem")
def run(actor: ActorRef): Unit = {
import system.dispatcher
Future { Thread.sleep(100); actor ! 1 }
Future { Thread.sleep(200); actor ! 2 }
Future { Thread.sleep(300); actor ! 3 }
}
val source = Source
.actorRef[Int](0, OverflowStrategy.fail)
.mapMaterializedValue(ref ⇒ run(ref))
implicit val m = ActorMaterializer()
source runForeach { int ⇒
println(s"received: $int")
}
The Actor is created through the Source.actorRef method and made available through the mapMaterializedValue method. run simply takes the Actor and sends all the completed values to it, which can then be accessed through source. In the example above, the values are sent directly in the Future, but this can of course be done everywhere (for example in the onComplete call on the Future).

for comprehension with futures in scala translation to flatMap

I have been looking at this How are Scala Futures chained together with flatMap? and the corresponding article as well on translating for comprehension. I am slowly adding stuff to my for comprehension and am stuck as I guess the code I thought would translate to is not correct.
Here I have a runProgram and runProgram2 which I thought would be equivalent and are not because runProgram2 does not compile. Can someone explain the equiavalent of this for comprehension...
NOTE: yes I know that future.flatMap is typically for collapsing Future[Future[String]] but this is a trimmed down version of my file(perhaps I trimmed it down too far).
def main(args: Array[String]) = {
val future1: Future[String] = runMyProgram()
//val future2: Future[String] = runMyProgram2()
}
def runMyProgram() : Future[String] = {
val future = serviceCall()
future.flatMap(processAllReturnCodes)
}
// def runMyProgram2() : Future[String] = {
// val future = serviceCall()
// for {
// result <- future
// } yield processAllReturnCodes(result)
// }
def processAllReturnCodes(count: Int) : Future[String] = {
val promise = Promise.successful("done")
promise.future
}
def serviceCall() : Future[Int] = {
val promise = Promise.successful(5)
promise.future
}
def serviceCall2() : Future[String] = {
val promise = Promise.successful("hithere")
promise.future
}
This for comprehension:
for {
result <- future
} yield processAllReturnCodes(result)
Is being translated to this:
val t: Future[Future[String]] = future.map(result => processAllReturnCodes(result))
Comprehension is really only syntactic sugar for map and flatMap, with flatMap you can flatten the future nesting:
val u: Future[String] = future.flatMap(result => processAllReturnCodes(result))
The difference lies in the signatures:
def map[S](f: T => S)(implicit executor: ExecutionContext): Future[S]
So map takes a function form T to S and wraps the S in a future, the problem is that here your S is a Future[String] which is wrapped in another future giving Future[Future[String]], flatMap instead:
def flatMap[S](f: T => Future[S])(implicit executor: ExecutionContext): Future[S]
Takes a function from T to Future[S] and returns that future, in your case your method already returns a future and it's a valid parameter for flatMap.