I'd like to measure elapsed time inside IO container. It's relatively easy to do with plain calls or futures (e.g. something like the code below)
class MonitoringComponentSpec extends FunSuite with Matchers with ScalaFutures {
import scala.concurrent.ExecutionContext.Implicits.global
def meter[T](future: Future[T]): Future[T] = {
val start = System.currentTimeMillis()
future.onComplete(_ => println(s"Elapsed ${System.currentTimeMillis() - start}ms"))
future
}
def call(): Future[String] = Future {
Thread.sleep(500)
"hello"
}
test("metered call") {
whenReady(meter(call()), timeout(Span(550, Milliseconds))) { s =>
s should be("hello")
}
}
}
But not sure how to wrap IO call
def io_meter[T](effect: IO[T]): IO[T] = {
val start = System.currentTimeMillis()
???
}
def io_call(): IO[String] = IO.pure {
Thread.sleep(500)
"hello"
}
test("metered io call") {
whenReady(meter(call()), timeout(Span(550, Milliseconds))) { s =>
s should be("hello")
}
}
Thank you!
Cats-effect has a Clock implementation that allows pure time measurement as well as injecting your own implementations for testing when you just want to simulate the passing of time. The example from their documentation is:
def measure[F[_], A](fa: F[A])
(implicit F: Sync[F], clock: Clock[F]): F[(A, Long)] = {
for {
start <- clock.monotonic(MILLISECONDS)
result <- fa
finish <- clock.monotonic(MILLISECONDS)
} yield (result, finish - start)
}
In cats effect 3, you can use .timed. Like,
import cats.effect.IO
import cats.effect.unsafe.implicits.global
import cats.implicits._
import scala.concurrent.duration._
val twoSecondsLater = IO.sleep(2.seconds) *> IO.println("Hi")
val elapsedTime = twoSecondsLater.timed.map(_._1)
elapsedTime.unsafeRunSync()
// would give something like this.
Hi
res0: FiniteDuration = 2006997667 nanoseconds
Related
As mentioned in the jump-start guide, mapN will run all the futures in parallel, so I created the below simple Scala program, but a sample run shows diff to be 9187 ms and diffN to be 9106 ms. So it looks like that the mapN is also running the futures sequentially, isn't it? Please let me know if I am missing something?
package example
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
import java.time.LocalDateTime
import java.time.Duration
import scala.util.Failure
import scala.util.Success
import java.time.ZoneOffset
import cats.instances.future._
import cats.syntax.apply._
object FutureEx extends App {
val before = LocalDateTime.now()
val sum = for {
a <- getA
b <- getB
c <- getC
} yield (a + b + c)
sum onComplete {
case Failure(ex) => println(s"Error: ${ex.getMessage()}")
case Success(value) =>
val after = LocalDateTime.now()
println(s"before: $before")
println(s"after: $after")
val diff = getDiff(before, after)
println(s"diff: $diff")
println(s"sum: $value")
}
// let the above finish
Thread.sleep(20000)
val beforeN = LocalDateTime.now()
val usingMapN = (getA, getB, getC).mapN(add)
usingMapN onComplete {
case Failure(ex) => println(s"Error: ${ex.getMessage()}")
case Success(value) =>
val afterN = LocalDateTime.now()
println(s"beforeN: $beforeN")
println(s"afterN: $afterN")
val diff = getDiff(beforeN, afterN)
println(s"diffN: $diff")
println(s"sum: $value")
}
def getA: Future[Int] = {
println("inside A")
Thread.sleep(3000)
Future.successful(2)
}
def getB: Future[Int] = {
println("inside B")
Thread.sleep(3000)
Future.successful(3)
}
def getC: Future[Int] = {
println("inside C")
Thread.sleep(3000)
Future.successful(4)
}
def add(a: Int, b: Int, c: Int) = a + b + c
def getDiff(before: LocalDateTime, after: LocalDateTime): Long = {
Duration.between(before.toInstant(ZoneOffset.UTC), after.toInstant(ZoneOffset.UTC)).toMillis()
}
}
Because you have sleep outside Future it should be like:
def getA: Future[Int] = Future {
println("inside A")
Thread.sleep(3000)
2
}
So you start async Future with apply - Future.successful on the other hand returns pure value, meaning you execute sleep in same thread.
The time is going before mapN is ever called.
(getA, getB, getC).mapN(add)
This expression is creating a tuple (sequentially) and then calling mapN on it. So it is calling getA then getB then getC and since each of them has a 3 second delay, it takes 9 seconds.
I'm exploring fs2 library and for testing in I have a Stream[IO, Int] that are pushed into a Queue[IO,(Double, String, String, String)]. I need to identify the entry time to the queue as well as the exit time.
The problem here is that the processing time may vary among elements. For this, I thought of using .metered((Random.between(1,10)).seconds)
The issue is that I can not figure out a way in order to store entry time at the time of enqueuing and the exitTime after a certain amount of time given with the .metered((Random.between(1,10)).seconds)
Here is what I have tried :
import cats.effect.{ExitCode, IO, IOApp, Timer}
import fs2._
import fs2.concurrent.Queue
import scala.concurrent.duration._
import scala.util.Random
class StreamTypeIntToDouble(q: Queue[IO, (Double, String, String, String)])(
implicit timer: Timer[IO]
) {
import core.Processing._
def storeInQueue: Stream[IO, Unit] = {
Stream(1, 2, 3)
.covary[IO]
.evalTap(n => IO.delay(println(s"Pushing $n to Queue")))
.map { n =>
val entryTime = currentTimeNow
val exitTime = currentTimeNow
(n.toDouble, "Service", entryTime, exitTime)
}
.metered(Random.between(1, 20).seconds)
.through(q.enqueue)
//.metered((Random.between(1, 10).seconds))
}
def getFromQueue: Stream[IO, Unit] = {
q.dequeue
.evalMap(n => IO.delay(println(s"Pulling from queue $n")))
}
}
object Five extends IOApp {
override def run(args: List[String]): IO[ExitCode] = {
val program = for {
q <- Queue.bounded[IO, (Double, String, String, String)](10)
b = new StreamTypeIntToDouble(q)
_ <- b.storeInQueue.compile.drain.start
_ <- b.getFromQueue.compile.drain
} yield ()
program.as(ExitCode.Success)
}
}
The method used for identifying the current time is implemented as follows
def currentTimeNow: String = {
val format = new SimpleDateFormat("dd-MM-yy hh:mm:ss")
format.format(Calendar.getInstance().getTime())
}
Question : How can I keep track of the entry and exit time to construct the new output stream?
I have an external future operation which I can override it's onOperation complete method . I want to wrap it and complete by closure of a Promise. however I couldn't complete that future within other future. for example :
import scala.concurrent.{Future, Promise}
import scala.util.{Failure, Success}
import scala.concurrent.ExecutionContext.Implicits.global
def foo():Future[String] = {
val p = Promise[String]()
channel.addListener(new ChannelFutureListener[IoReadFuture] {
override def operationComplete(future: IoReadFuture): Unit = {
p.success("hello")}
}
p.future
}
val x = foo()
x: scala.concurrent.Future[String] = List()
x.onComplete{
case Success(msg) => println(s"$msg world")
case Failure(e) => println(e.getMessage)
}
res1: Unit = ()
is there idiomatic way to do that without blocking?
I think you are trying to create a function which takes a delay as input and returns a delayed future.
If you want to do that, you can do it in a non-sleeping way using Await and Promise.
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.{Future, Promise}
import scala.concurrent.duration._
// returns a future delayed by `delay` seconds
def getDelayedFutureOfStringValue(delay: Int, value: String): Future[String] = {
// we will use this promise for wait only
val waitProxyPromise = Promise[Int]()
val delayedFuture = Await.ready(waitProxyPromise.future, delay.second).map({
case _ => value
})
delayedFuture
}
val helloFuture = getDelayedFutureOfStringValue(2, "hello")
The above amy seem like a decent implementation but it is actually not. Await actually blocks the thread. Sadly... there is not easy way to get a delayedFutue in a totally non-blocking way in idiomatic Scala.
You can get a nice non-blocking and non-sleeping delayed future using Timer utility from Java,
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.{Future, Promise}
import java.util.{Timer, TimerTask}
// delay is number of seconds
def getDelayedFutureOfStringValue(delay: Int, value: String): Future[String] = {
val promise = Promise[String]()
val timer = new Timer()
val timerTask = new TimerTask {
override def run(): Unit = promise.success(value)
}
timer.schedule(timerTask, delay * 1000)
promise.future
}
val niceHelloFuture = getDelayedFutureOfStringValue(2, "hello")
And in case you already have a future with you and you want to use that to compose a dependant future, then its pretty easy.
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.{Future, Promise}
import scala.util.{Failure, Success}
// I assume you IoReadFuture is either similar to Future or wraps the actual Future
def foo(value: String):Future[String] = {
val p = Promise[String]()
channel.addListener(new ChannelFutureListener[IoReadFuture] {
override def operationComplete(ioFuture: IoReadFuture): Unit = {
ioFuture.future.onComplete(_ => p.success(value))
}
}
p.future
}
scheduleOne of the actor scheduler is non blocking way to wait for some code to execute
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext
import ExecutionContext.Implicits.global
val p = Promise(doSomething())
val system = akka.actor.ActorSystem("system")
system.scheduler.scheduleOne(deplay seconds)(p.future)
val f = p.future
f.flatMap { println(s"${_}") }
I want to add an after(d: FiniteDuration)(callback: => Unit) util to Scala Futures that would enable me to do this:
val f = Future(someTask)
f.after(30.seconds) {
println("f has not completed in 30 seconds!")
}
f.after(60.seconds) {
println("f has not completed in 60 seconds!")
}
How can I do this?
Usually I use a thread pool executor and promises:
import scala.concurrent.duration._
import java.util.concurrent.{Executors, ScheduledThreadPoolExecutor}
import scala.concurrent.{Future, Promise}
val f: Future[Int] = ???
val executor = new ScheduledThreadPoolExecutor(2, Executors.defaultThreadFactory(), AbortPolicy)
def withDelay[T](operation: ⇒ T)(by: FiniteDuration): Future[T] = {
val promise = Promise[T]()
executor.schedule(new Runnable {
override def run() = {
promise.complete(Try(operation))
}
}, by.length, by.unit)
promise.future
}
Future.firstCompletedOf(Seq(f, withDelay(println("still going"))(30 seconds)))
Future.firstCompletedOf(Seq(f, withDelay(println("still still going"))(60 seconds)))
One way is to use Future.firstCompletedOf (see this blogpost):
val timeoutFuture = Future { Thread.sleep(500); throw new TimeoutException }
val f = Future.firstCompletedOf(List(f, timeoutFuture))
f.map { case e: TimeoutException => println("f has not completed in 0.5 seconds!") }
where TimeoutException is some exception or type.
Use import akka.pattern.after. If you want to implement it without akka here is the source code. The other (java) example is TimeoutFuture in com.google.common.util.concurrent.
Something like this, perhaps:
object PimpMyFuture {
implicit class PimpedFuture[T](val f: Future[T]) extends AnyVal {
def after(delay: FiniteDuration)(callback: => Unit): Future[T] = {
Future {
blocking { Await.ready(f, delay) }
} recover { case _: TimeoutException => callback }
f
}
}
}
import PimpMyFuture._
Future { Thread.sleep(10000); println ("Done") }
.after(5.seconds) { println("Still going") }
This implementation is simple, but it basically doubles the number of threads you need - each active future effectively occupies two threads - which is a bit wasteful. Alternatively, you could use scheduled tasks to make your waits non-blocking. I don't know of a "standard" scheduler in scala (each lib has their own), but for a simple task like this you can use java's TimerTask directly:
object PimpMyFutureNonBlocking {
val timer = new java.util.Timer
implicit class PimpedFuture[T](val f: Future[T]) extends AnyVal {
def after(delay: FiniteDuration)(callback: => Unit): Future[T] = {
val task = new java.util.TimerTask {
def run() { if(!f.isCompleted) callback }
}
timer.schedule(task, delay.toMillis)
f.onComplete { _ => task.cancel }
f
}
}
}
I have a couple of futures. campaignFuture returns a List[BigInt] and I want to be able to call the second future profileFuture for each of the values in the list returned from the first one. The second future can only be called when the first one is complete. How do I achieve this in Scala?
campaignFuture(1923).flatMap?? (May be?)
def campaignFuture(advertiserId: Int): Future[List[BigInt]] = Future {
val campaignHttpResponse = getCampaigns(advertiserId.intValue())
parseProfileIds(campaignHttpResponse.entity.asString)
}
def profileFuture(profileId: Int): Future[List[String]] = Future {
val profileHttpResponse = getProfiles(profileId.intValue())
parseSegmentNames(profileHttpResponse.entity.asString)
}
A for comprehension is here not applicable because we have a mix of List's and Future's.
So, your friends are map and flatMap:
To react on Future result
import scala.concurrent.{Future, Promise, Await}
import scala.concurrent.duration.Duration
import scala.concurrent.ExecutionContext.Implicits.global
def campaignFuture(advertiserId: Int): Future[List[BigInt]] = Future {
List(1, 2, 3)
}
def profileFuture(profileId: Int): Future[List[String]] = {
// delayed Future
val p = Promise[List[String]]
Future {
val delay: Int = (math.random * 5).toInt
Thread.sleep(delay * 1000)
p.success(List(s"profile-for:$profileId", s"delayed:$delay sec"))
}
p.future
}
// Future[List[Future[List[String]]]
val listOfProfileFuturesFuture = campaignFuture(1).map { campaign =>
campaign.map(id => profileFuture(id.toInt))
}
// React on Futures which are done
listOfProfileFuturesFuture foreach { campaignFutureRes =>
campaignFutureRes.foreach { profileFutureRes =>
profileFutureRes.foreach(profileListEntry => println(s"${new Date} done: $profileListEntry"))
}
}
// !!ONLY FOR TESTING PURPOSE - THIS CODE BLOCKS AND EXITS THE VM WHEN THE FUTURES ARE DONE!!
println(s"${new Date} waiting for futures")
listOfProfileFuturesFuture.foreach{listOfFut =>
Await.ready(Future.sequence(listOfFut), Duration.Inf)
println(s"${new Date} all futures done")
System.exit(0)
}
scala.io.StdIn.readLine()
To get the result of all Futures at once
import scala.concurrent.{Future, Await}
import scala.concurrent.duration.Duration
import scala.concurrent.ExecutionContext.Implicits.global
def campaignFuture(advertiserId: Int): Future[List[BigInt]] = Future {
List(1, 2, 3)
}
def profileFuture(profileId: Int): Future[List[String]] = Future {
List(s"profile-for:$profileId")
}
// type: Future[List[Future[List[String]]]]
val listOfProfileFutures = campaignFuture(1).map { campaign =>
campaign.map(id => profileFuture(id.toInt))
}
// type: Future[List[List[String]]]
val listOfProfileFuture = listOfProfileFutures.flatMap(s => Future.sequence(s))
// print the result
//listOfProfileFuture.foreach(println)
//scala.io.StdIn.readLine()
// wait for the result (THIS BLOCKS INFINITY!)
Await.result(listOfProfileFuture, Duration.Inf)
we use Future.sequence to convert a List[Future[T]] to Future[List[T]].
flatMap to get a Future[T] from Future[Future[T]]
if you need wait for the result (BLOCKING!) use Await to wait for the result