suppose I have a sleep function:
def sleep(delay:Int) : Unit #suspendable = {
....
}
is it possible to have a function future that creates an async version of the sleep function that can be awaited on synchronously.
def future(targetFunc: (Int => Unit #suspendable)) : (Int => Future) = {
....
}
class Future {
def await : Unit #suspendable = {
....
}
}
you should be able to do something like this:
reset {
val sleepAsync = future(sleep)
val future1 = sleepAsync(2000)
val future2 = sleepAsync(3000)
future1.await
future2.await
/* finishes after a delay of 3000 */
}
the two calls to sleepAsync should appear to return straight away and the two calls to Future#await should appear to block. of course they all really fall off the end of reset and the code after is responsible for calling the continuation after the delay.
otherwise is there an alternative method to running two #suspendable functions in parallel and wait for both of them to complete?
I have a compilable gist with a skeleton of what i want to do: https://gist.github.com/1191381
object Forks {
import scala.util.continuations._
case class Forker(forks: Vector[() => Unit #suspendable]) {
def ~(block: => Unit #suspendable): Forker = Forker(forks :+ (() => block))
def joinIf(pred: Int => Boolean): Unit #suspendable = shift { k: (Unit => Unit) =>
val counter = new java.util.concurrent.atomic.AtomicInteger(forks.size)
forks foreach { f =>
reset {
f()
if (pred(counter.decrementAndGet)) k()
}
}
}
def joinAll() = joinIf(_ == 0)
def joinAny() = joinIf(_ == forks.size - 1)
}
def fork(block: => Unit #suspendable): Forker = Forker(Vector(() => block))
}
using fork(), we can now wait many "suspendables". use ~() to chain together suspendables. use joinAll() to wait for all suspendables and joinAny() to wait for just one. use joinIf() to customize join strategy.
object Tests extends App {
import java.util.{Timer, TimerTask}
import scala.util.continuations._
implicit val timer = new Timer
def sleep(ms: Int)(implicit timer: Timer): Unit #suspendable = {
shift { k: (Unit => Unit) =>
timer.schedule(new TimerTask {
def run = k()
}, ms)
}
}
import Forks._
reset {
fork {
println("sleeping for 2000 ms")
sleep(2000)
println("slept for 2000 ms")
} ~ {
println("sleeping for 4000 ms")
sleep(4000)
println("slept for 4000 ms")
} joinAll()
println("and we are done")
}
println("outside reset")
readLine
timer.cancel
}
and this is the output. program starts at time T:
sleeping for 2000 ms
sleeping for 4000 ms
outside reset <<<<<< T + 0 second
slept for 2000 ms <<<<<< T + 2 seconds
slept for 4000 ms <<<<<< T + 4 seconds
and we are done <<<<<< T + 4 seconds
I'm not sure that I completely understand the question, but here's a try:
import scala.util.continuations._
class Future(thread: Thread) {
def await = thread.join
}
object Future {
def sleep(delay: Long) = Thread.sleep(delay)
def future[A,B](f: A => B) = (a: A) => shift { k: (Future => Unit) =>
val thread = new Thread { override def run() { f(a) } }
thread.start()
k(new Future(thread))
}
def main(args:Array[String]) = reset {
val sleepAsync = future(sleep)
val future1 = sleepAsync(2000) // returns right away
val future2 = sleepAsync(3000) // returns right away
future1.await // returns after two seconds
future2.await // returns after an additional one second
// finished after a total delay of three seconds
}
}
Here, a Future instance is nothing more than a handle on a Thread, so you can use its join method to block until it finishes.
The future function takes a function of type A => B, and returns a function which, when supplied with an A will kick off a thread to run the "futured" function, and wrap it up in a Future, which is injected back into the continuation, thereby assigning it to val future1.
Is this anywhere close to what you were going for?
Related
// I have hundreds of tasks converting inputs into outputs, which should be persisted.
case class Op(i: Int)
case class Output(i: Int)
val inputs: Seq[Op] = ??? // Number of inputs is huge
def executeLongRunning(op: Op): Output = {
Thread.sleep(Random.nextInt(1000) + 1000) // I cannot predict which tasks will finish first
println("<==", op)
Output(op.i)
}
def executeSingleThreadedSave(outputs: Seq[Output]): Unit = {
synchronized { // Problem is, persisting output is itself a long-running process,
// which cannot be parallelized (internally uses blocking queue).
Thread.sleep(5000) // persist time is independent of outputs.size
println("==>", outputs) // Order of persisted records does not matter
}
}
// TODO: this needs to be implemented
def magicSaver(eventualOutputs: Seq[Future[Output]], saver: Seq[Output] => Unit): Unit = ???
val eventualOutputs: Seq[Future[Output]] = inputs.map((input: Op) => Future(executeLongRunning(input)))
magicSaver(eventualOutputs, executeSingleThreadedSave)
I could implement magicSaver to be:
def magicSaver(eventualOutputs: Seq[Future[Output]], saver: Seq[Output] => Unit): Unit = {
saver(Await.result(Future.sequence(eventualOutputs), Duration.Inf))
}
But this has major drawback that we're waiting for all inputs to get processed before we're starting to persisting outputs, which is not ideal from fault-tolerance standpoint.
Another implementation is:
def magicSaver(eventualOutputs: Seq[Future[Output]], saver: Seq[Output] => Unit): Unit = {
eventualOutputs.foreach(_.onSuccess { case output: Output => saver(Seq(output)) })
}
but this blows up execution time to inputs.size * 5secs (because of synchronized nature of, which is not acceptable.
I want a way to batch together already completed futures, when number of such futures reached some trade-off size (100, for example), but I'm not sure how to do that in clean manner without explicitly coding polling logic:
def magicSaver(eventualOutputs: Seq[Future[Output]], saver: Seq[Output] => Unit): Unit = {
def waitFor100CompletedFutures(eventualOutputs: Seq[Future[Output]]): (Seq[Output], Seq[Future[Output]]) = {
var completedCount: Int = 0
do {
completedCount = eventualOutputs.count(_.isCompleted)
Thread.sleep(100)
} while ((completedCount < 100) && (completedCount != eventualOutputs.size))
val (completed: Seq[Future[Output]], remaining: Seq[Future[Output]]) = eventualOutputs.partition(_.isCompleted)
(Await.result(Future.sequence(completed), Duration.Inf), remaining)
}
var completed: Seq[Output] = null
var remaining: Seq[Future[Output]] = eventualOutputs
do {
(completed: Seq[Output], remaining: Seq[Future[Output]]) = waitFor100CompletedFutures(remaining)
saver(completed)
} while (remaining.nonEmpty)
}
Any elegant solution I'm missing here?
I'm posting my solution here, for reference. It has the benefit that it avoids batching altogether, and invokes processOutput as soon as output becomes available, which is the best situation under constraints I've described.
def magicSaver[T, R](eventualOutputs: Seq[Future[T]],
processOutput: Seq[T] => R)(implicit ec: ExecutionContext): Seq[R] = {
logInfo(s"Size of outputs to save: ${eventualOutputs.size}")
var remaining: Seq[Future[T]] = eventualOutputs
val processorOutput: mutable.ListBuffer[R] = new mutable.ListBuffer[R]
do {
val (currentCompleted: Seq[Future[T]], currentRemaining: Seq[Future[T]]) = remaining.partition(_.isCompleted)
if (remaining.size == currentRemaining.size) {
Thread.sleep(100)
} else {
logInfo(s"Got ${currentCompleted.size} completed records, remaining ${currentRemaining.size}")
val completed = currentCompleted.map(Await.result(_, Duration.Zero))
processorOutput.append(processOutput(completed))
}
remaining = currentRemaining
} while (remaining.nonEmpty)
processorOutput
}
I have two Scala functions that are expensive to run. Each one is like below, they start improving the value of a variable and I'd like to run them simultaneously and after 5 minutes (or some other time). I'd like to terminate the two functions and take their latest value up to that time.
def func1(n: Int): Double = {
var a = 0.0D
while (not terminated) {
/// improve value of 'a' with algorithm 1
}
}
def func2(n: Int): Double = {
var a = 0.0D
while (not terminated) {
/// improve value of 'a' with algorithm 2
}
}
I would like to know how I should structure my code for doing that and what is the best practice here? I was thinking about running them in two different threads with a timeout and return their latest value at time out. But it seems there can be other ways for doing that. I am new to Scala so any insight would be tremendously helpful.
It is not hard. Here is one way of doing it:
#volatile var terminated = false
def func1(n: Int): Double = {
var a = 0.0D
while (!terminated) {
a = 0.0001 + a * 0.99999; //some useless formula1
}
a
}
def func2(n: Int): Double = {
var a = 0.0D
while (!terminated) {
a += 0.0001 //much simpler formula2, just for testing
}
a
}
def main(args: Array[String]): Unit = {
val f1 = Future { func1(1) } //work starts here
val f2 = Future { func2(2) } //and here
//aggregate results into one common future
val aggregatedFuture = for{
f1Result <- f1
f2Result <- f2
} yield (f1Result, f2Result)
Thread.sleep(500) //wait here for some calculations in ms
terminated = true //this is where we actually command to stop
//since looping to while() takes time, we need to wait for results
val res = Await.result(aggregatedFuture, 50.millis)
//just a printout
println("results:" + res)
}
But, of course, you would want to maybe look at your while loops and create a more manageable and chainable calculations.
Output: results:(9.999999999933387,31206.34691883926)
I am not 100% sure if this is something you would want to do, but here is one approach (not for 5 minutes, but you can change that) :
object s
{
def main(args: Array[String]): Unit = println(run())
def run(): (Int, Int) =
{
val (s, numNanoSec, seedVal) = (System.nanoTime, 500000000L, 0)
Seq(f1 _, f2 _).par.map(f =>
{
var (i, id) = f(seedVal)
while (System.nanoTime - s < numNanoSec)
{
i = f(i)._1
}
(i, id)
}).seq.maxBy(_._1)
}
def f1(a: Int): (Int, Int) = (a + 1, 1)
def f2(a: Int): (Int, Int) = (a + 2, 2)
}
Output:
me#ideapad:~/junk> scala s.scala
(34722678,2)
me#ideapad:~/junk> scala s.scala
(30065688,2)
me#ideapad:~/junk> scala s.scala
(34650716,2)
Of course this all assumes you have at least two threads available to distribute tasks to.
You can use Future with Await result to do that:
def fun2(): Double = {
var a = 0.0f
val f = Future {
// improve a with algorithm 2
a
}
try {
Await.result(f, 5 minutes)
} catch {
case e: TimeoutException => a
}
}
use the Await.result to wait algorithm with timeout, when we met this timeout, we return the a directly
I am doing Exercises from Learning Concurrent Programming in Scala.
For an exercise question in code comment.
Program prints proper output of HTML contents for proper URL and timeout sufficiently enough.
Program prints "Error occured" for proper URL and low timeout.
However for invalid URL "Error occured" is not printed. What is the problem with the code below?
/*
* Implement a command-line program that asks the user to input a URL of some website,
* and displays the HTML of that website. Between the time that the user hits ENTER and
* the time that the HTML is retrieved, the program should repetitively print a . to the
* standard output every 50 milliseconds, with a two seconds timeout. Use only futures
* and promises, and avoid the synchronization primitives from the previous chapters.
* You may reuse the timeout method defined in this chapter.
*/
object Excersices extends App {
val timer = new Timer()
def timeout(t: Long = 1000): Future[Unit] = {
val p = Promise[Unit]
val timer = new Timer(true)
timer.schedule(new TimerTask() {
override def run() = {
p success ()
timer cancel()
}
}, t)
p future
}
def printDot = println(".")
val taskOfPrintingDot = new TimerTask {
override def run() = printDot
}
println("Enter a URL")
val lines = io.Source.stdin.getLines()
val url = if (lines hasNext) Some(lines next) else None
timer.schedule(taskOfPrintingDot, 0L, 50.millisecond.toMillis)
val timeOut2Sec = timeout(2.second.toMillis)
val htmlContents = Future {
url map { x =>
blocking {
Source fromURL (x) mkString
}
}
}
Future.firstCompletedOf(Seq(timeOut2Sec, htmlContents)) map { x =>
timer cancel ()
x match {
case Some(x) =>
println(x)
case _ =>
println("Error occured")
}
}
Thread sleep 5000
}
As #Gábor Bakos said exception produces Failure which doesn't handled by map:
val fut = Future { Some(Source fromURL ("hhhttp://google.com")) }
scala> fut map { x => println(x) } //nothing printed
res12: scala.concurrent.Future[Unit] = scala.concurrent.impl.Promise$DefaultPromise#5e025724
To process failure - use recover method :
scala> fut recover { case failure => None } map { x => println(x) }
None
res13: scala.concurrent.Future[Unit] = scala.concurrent.impl.Promise$DefaultPromise#578afc83
In your context it's something like:
Future.firstCompletedOf(Seq(timeOut2Sec, htmlContents)) recover {case x => println("Error:" + x); None} map { x => ...}
The Complete Code after using recover as advised by #dk14:
object Exercises extends App {
val timer = new Timer()
def timeout(t: Long = 1000): Future[Unit] = {
val p = Promise[Unit]
val timer = new Timer(true)
timer.schedule(new TimerTask() {
override def run() = {
p success ()
timer cancel ()
}
}, t)
p future
}
def printDot = println(".")
val taskOfPrintingDot = new TimerTask {
override def run() = {
printDot
}
}
println("Enter a URL")
val lines = io.Source.stdin.getLines()
val url = if (lines hasNext) Some(lines next) else None
timer.schedule(taskOfPrintingDot, 0L, 50.millisecond.toMillis)
val timeOut2Sec = timeout(2.second.toMillis)
val htmlContents = Future {
url map { x =>
blocking {
Source fromURL (x) mkString
}
}
}
Future.firstCompletedOf(Seq(timeOut2Sec, htmlContents))
.recover { case x => println("Error:" + x); None }
.map { x =>
timer cancel ()
x match {
case Some(x) =>
println(x)
case _ =>
println("Timeout occurred")
}
}
Thread sleep 5000
}
I have a class with some methods in Scala, each method has an execution time, for example methodA takes 10 seconds and methodB takes 5 seconds. and each method calls asynchronously.when I call methodB, it should cancel the thread that is running another methods. I first call methodA, and 2 seconds later I call methodB. What is the best solution for this problem?
def methodA()={
async{
// a job that takes 10 seconds
}
}
def methodB()={
async{
// other methods should stop their job
// a job that takes 5 second
}
}
def async[T](fn: => Unit): Unit = scala.actors.Actor.actor {
fn
}
.....
methodA()
methodB()
Here is an idea, based on the assumption that your method is actively checking whether it should still run or cancel:
import concurrent.{ExecutionContext, Future, Promise, blocking, future, promise}
case class Cancelled() extends RuntimeException
object Job {
def apply[A](fun: (() => Boolean) => A)(implicit ctx: ExecutionContext): Job[A] =
new Job[A] {
private val p = promise[A]
def result = p.future
def cancel(): Unit = p.tryFailure(Cancelled())
p tryCompleteWith future {
fun(() => !p.isCompleted)
}
}
}
trait Job[A] {
def result: Future[A]
def cancel(): Unit
}
So Job embodies a future along with a cancel() method. Your example could be similar to this:
import ExecutionContext.Implicits.global
val a = Job { active =>
for (i <- 1 to 100 if active()) {
blocking {
Thread.sleep(1000) // doing actual heavy work here
println(s"A $i")
}
}
}
val b = Job { active =>
for (i <- 1 to 20 if active()) {
blocking {
Thread.sleep(1000) // doing actual heavy work here
println(s"B $i")
}
}
println("Goodbye A...")
a.cancel()
}
I have also seen a rather harsh variant that calls Thread.interrupt.
This is a followup to my previous question.
Suppose I have a task, which executes an interruptible blocking call. I would like to run it as a Future and cancel it with failure method of Promise.
I would like the cancel to work as follows:
If one cancels the task before it finished I would like the task to finish "immediately", interrupting the blocking call if it has already started and I would like the Future to invoke onFailure.
If one cancels the task after the task finished I would like to get a status saying that the cancel failed since the task already finished.
Does it make sense? Is it possible to implement in Scala? Are there any examples of such implementations?
scala.concurrent.Future is read-only, so one reader cannot mess things up for the other readers.
It seems like you should be able to implement what you want as follows:
def cancellableFuture[T](fun: Future[T] => T)(implicit ex: ExecutionContext): (Future[T], () => Boolean) = {
val p = Promise[T]()
val f = p.future
p tryCompleteWith Future(fun(f))
(f, () => p.tryFailure(new CancellationException))
}
val (f, cancel) = cancellableFuture( future => {
while(!future.isCompleted) continueCalculation // isCompleted acts as our interrupted-flag
result // when we're done, return some result
})
val wasCancelled = cancel() // cancels the Future (sets its result to be a CancellationException conditionally)
Here is the interruptable version of Victor's code per his comments (Victor, please correct me if I misinterpreted).
object CancellableFuture extends App {
def interruptableFuture[T](fun: () => T)(implicit ex: ExecutionContext): (Future[T], () => Boolean) = {
val p = Promise[T]()
val f = p.future
val aref = new AtomicReference[Thread](null)
p tryCompleteWith Future {
val thread = Thread.currentThread
aref.synchronized { aref.set(thread) }
try fun() finally {
val wasInterrupted = (aref.synchronized { aref getAndSet null }) ne thread
//Deal with interrupted flag of this thread in desired
}
}
(f, () => {
aref.synchronized { Option(aref getAndSet null) foreach { _.interrupt() } }
p.tryFailure(new CancellationException)
})
}
val (f, cancel) = interruptableFuture[Int] { () =>
val latch = new CountDownLatch(1)
latch.await(5, TimeUnit.SECONDS) // Blocks for 5 sec, is interruptable
println("latch timed out")
42 // Completed
}
f.onFailure { case ex => println(ex.getClass) }
f.onSuccess { case i => println(i) }
Thread.sleep(6000) // Set to less than 5000 to cancel
val wasCancelled = cancel()
println("wasCancelled: " + wasCancelled)
}
With Thread.sleep(6000) the output is:
latch timed out
42
wasCancelled: false
With Thread.sleep(1000) the output is:
wasCancelled: true
class java.util.concurrent.CancellationException
Twitter's futures implement cancellation. Have a look here:
https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Future.scala
Line 563 shows the abstract method responsible for this. Scala's futures currently do not support cancellation.
You can use Monix library instead of Future
https://monix.io