In the below code, first of all we can create the something with calling create method, and for destroying it we can call destroy method, each method at first changes the status of the object, then asynchronously wait until the time of the operation is up. assume that time of creating is 10 seconds and time for destroying is 5 second. when I call destroy method, it changes the destroying boolean to true, then another threads that is waiting for the creation duration will break the loop in the waitUntil method. but it does not work properly. when I call create method, it changes the status and then creates a thread to wait until the time of the creation duration goes up. after calling create method, I call destroy method as below:
map(0, 0).create(....)
map(0, 0).destroy(....)
But changing the destroying variable in destroy method did not act the another thread that is in the creating block to break and it continue until the end of duration.
#volatile
protected var status: UnitStatus = UnitStatus.NEED_TO_CREATE
#volatile
protected var destroying = false
def destroy(f: Int => Unit): Unit = status match {
case UnitStatus.DESTROYED => {
throw new UnitAlreadyDestroyedException
}
case UnitStatus.DESTROYING =>
case UnitStatus.PREPARED_TO_DESTROY => {
destroying = true
status = UnitStatus.DESTROYING
async {
waitUntil(destroyDuration, UnitStatus.DESTROYED) {
f
}
}
}
}
def create(f: Int => Unit): Unit = status match {
case UnitStatus.DESTROYED => throw new UnitAlreadyDestroyedException
case UnitStatus.NEED_TO_CREATE => {
ResourcesContainer -= creationCost
status = UnitStatus.CONSTRUCTION
async {
waitUntil(creationDuration, UnitStatus.READY) {
f
}
}
}
case _ => throw new UnitAlreadyCreatedException
}
def waitUntil(seconds: Int, finalState: UnitStatus)(f: Int => Unit): Unit = {
var timeElapse = 0
var percent: Int = 0
breakable {
while (timeElapse < seconds) {
val destroyState = isInDestroyState
if (destroying && !destroyState) {
break() // **program does not enter to this part of code**
}
timeElapse += 1
percent = ((timeElapse.asInstanceOf[Float] / seconds) * 100).toInt
f(percent)
Thread.sleep(1000)
}
if (status != null) {
status = finalState
}
}
}
def async[T](fn: => Unit): Unit = scala.actors.Actor.actor {
fn
}
def isInDestroyState: Boolean = {
status == UnitStatus.DESTROYED ||
status == UnitStatus.DESTROYING ||
status == UnitStatus.PREPARED_TO_DESTROY
}
Related
I need functionality like monix.Observable.bufferTimedAndCounted but with custom "weither". I found bufferTimedWithPressure operator which allow use item's weith:
val subj = PublishSubject[String]()
subj
.bufferTimedWithPressure(1.seconds, 5, _ => 3)
.subscribe(s => {
println(s)
Future.successful(Ack.Continue)
})
for (i <- 1 to 60) {
Thread.sleep(100)
subj.onNext(i.toString)
}
But emission happens every specified duration. I need behavior like bufferTimedAndCounted, so emission happens when buffer full. How to achive that?
I copied BufferTimedObservable from monix sources and slightly change it, add weight function (Note - i'm not tested it in all cases):
import java.util.concurrent.TimeUnit
import monix.execution.Ack.{Continue, Stop}
import monix.execution.cancelables.{CompositeCancelable, MultiAssignCancelable}
import monix.execution.{Ack, Cancelable}
import monix.reactive.Observable
import monix.reactive.observers.Subscriber
import scala.collection.mutable.ListBuffer
import scala.concurrent.Future
import scala.concurrent.duration.{Duration, FiniteDuration, MILLISECONDS}
/**
* Copied from monix sources, adopted to size instead count
*
*/
final class BufferTimedWithWeigthObservable[+A](source: Observable[A], timespan: FiniteDuration, maxSize: Int, sizeOf: A => Int)
extends Observable[Seq[A]] {
require(timespan > Duration.Zero, "timespan must be strictly positive")
require(maxSize >= 0, "maxSize must be positive")
def unsafeSubscribeFn(out: Subscriber[Seq[A]]): Cancelable = {
val periodicTask = MultiAssignCancelable()
val connection = source.unsafeSubscribeFn(new Subscriber[A] with Runnable {
self =>
implicit val scheduler = out.scheduler
private[this] val timespanMillis = timespan.toMillis
// MUST BE synchronized by `self`
private[this] var ack: Future[Ack] = Continue
// MUST BE synchronized by `self`
private[this] var buffer = ListBuffer.empty[A]
// MUST BE synchronized by `self`
private[this] var currentSize = 0
private[this] var sizeOfLast = 0
private[this] var expiresAt = scheduler.clockMonotonic(MILLISECONDS) + timespanMillis
locally {
// Scheduling the first tick, in the constructor
periodicTask := out.scheduler.scheduleOnce(timespanMillis, TimeUnit.MILLISECONDS, self)
}
// Runs periodically, every `timespan`
def run(): Unit = self.synchronized {
val now = scheduler.clockMonotonic(MILLISECONDS)
// Do we still have time remaining?
if (now < expiresAt) {
// If we still have time remaining, it's either a scheduler
// problem, or we rushed to signaling the bundle upon reaching
// the maximum size in onNext. So we sleep some more.
val remaining = expiresAt - now
periodicTask := scheduler.scheduleOnce(remaining, TimeUnit.MILLISECONDS, self)
} else if (buffer != null) {
// The timespan has passed since the last signal so we need
// to send the current bundle
sendNextAndReset(now, byPeriod = true).syncOnContinue(
// Schedule the next tick, but only after we are done
// sending the bundle
run())
}
}
// Must be synchronized by `self`
private def sendNextAndReset(now: Long, byPeriod: Boolean = false): Future[Ack] = {
val prepare = if (byPeriod) buffer else buffer.dropRight(1)
// Reset
if (byPeriod) {
buffer = ListBuffer.empty[A]
currentSize = 0
sizeOfLast = 0
} else {
buffer = buffer.takeRight(1)
currentSize = sizeOfLast
}
// Setting the time of the next scheduled tick
expiresAt = now + timespanMillis
ack = ack.syncTryFlatten.syncFlatMap {
case Continue => out.onNext(prepare)
case Stop => Stop
}
ack
}
def onNext(elem: A): Future[Ack] = self.synchronized {
val now = scheduler.clockMonotonic(MILLISECONDS)
buffer.append(elem)
sizeOfLast = sizeOf(elem)
currentSize = currentSize + sizeOfLast
// 9 and 9 true
//10 and 9
if (expiresAt <= now || (maxSize > 0 && maxSize < currentSize)) {
sendNextAndReset(now)
}
else {
Continue
}
}
def onError(ex: Throwable): Unit = self.synchronized {
periodicTask.cancel()
ack = Stop
buffer = null
out.onError(ex)
}
def onComplete(): Unit = self.synchronized {
periodicTask.cancel()
if (buffer.nonEmpty) {
val bundleToSend = buffer.toList
// In case the last onNext isn't finished, then
// we need to apply back-pressure, otherwise this
// onNext will break the contract.
ack.syncOnContinue {
out.onNext(bundleToSend)
out.onComplete()
}
} else {
// We can just stream directly
out.onComplete()
}
// GC relief
buffer = null
// Ensuring that nothing else happens
ack = Stop
}
})
CompositeCancelable(connection, periodicTask)
}
}
How use it:
object MonixImplicits {
implicit class RichObservable[+A](source: Observable[A]) {
def bufferTimedAndSized(timespan: FiniteDuration, maxSize: Int, sizeOf: A => Int): Observable[Seq[A]] = {
new BufferTimedWithWeigthObservable(source, timespan, maxSize, sizeOf)
}
}
}
import MonixImplicits._
someObservable.bufferTimedAndSized(1.seconds, 5, item => item.size)
private class FutMemorizer[T](valid: T => Boolean)(f: () => Future[T]) {
private val ref = new AtomicReference[Promise[T]]
#scala.annotation.tailrec
final def get(): Future[T] = {
val nullableRef = ref.get()
val valid = checkPromise(ref.get())
if(valid) {
nullableRef.future
} else {
val p = Promise[T]
val success = ref.compareAndSet(nullableRef, p)
if(success) {
p.completeWith(f())
p.future
} else {
get()
}
}
}
private def checkPromise(nullable: Promise[T]) = {
nullable != null && {
nullable.future.value match {
case None => true // future is not complete all caller should wait
case Some(Success(v)) => valid(v)
case _ => false
}
}
}
}
I am implementing an Future memorizer that only cache a valid future value.
It must meet following requirements
Futures created by f never executed paralleled
get never return a invalid value (once invalid recall f() to reload)
Is my implementation correct ?
Is there a more functional or simpler way to do this (because I hardly prove correntness of mime)?
As far as I understand this is wrong:
p.completeWith(f())
The caller gets a future the value of which is (or will be sometimes) that of the future returned by f(), but it's not checked anywhere that this value satisfies or will satisfy valid(...); same for other callers that came while the resulting future returned by f() is in progress if it takes time. It's only when result of f() completes will the next caller probably start "fixing" it.
I would probably go about fixing this problem the following way (see the fixed method), with some stylistic changes:
class FutMemorizer[T](valid: T => Boolean)(f: () => Future[T]) {
private val ref = new AtomicReference[Future[T]]
#tailrec
final def get: Future[T] = {
val current = ref.get
if (current != null && isValid(current)) current
else {
val p = Promise[T]
val pf = p.future
if (ref.compareAndSet(current, pf)) {
p.completeWith(fixed(f))
pf
} else get
}
}
private def fixed(f: () => Future[T]): Future[T] =
f() flatMap { t =>
if (valid(t)) Future.successful(t) else fixed(f)
}
private def isValid(future: Future[T]) =
future.value match {
case None => true // future is not complete all caller should wait
case Some(Success(v)) => valid(v)
case _ => false
}
}
As for your question about a more functional way of doing it I guess f and valid having effects on the external state and basing their computations on it (which I guess is the point of having a memorizer with invalidation) would seriously hinder it.
Just find spray-cache already have this feature
I am doing Exercises from Learning Concurrent Programming in Scala.
For an exercise question in code comment.
Program prints proper output of HTML contents for proper URL and timeout sufficiently enough.
Program prints "Error occured" for proper URL and low timeout.
However for invalid URL "Error occured" is not printed. What is the problem with the code below?
/*
* Implement a command-line program that asks the user to input a URL of some website,
* and displays the HTML of that website. Between the time that the user hits ENTER and
* the time that the HTML is retrieved, the program should repetitively print a . to the
* standard output every 50 milliseconds, with a two seconds timeout. Use only futures
* and promises, and avoid the synchronization primitives from the previous chapters.
* You may reuse the timeout method defined in this chapter.
*/
object Excersices extends App {
val timer = new Timer()
def timeout(t: Long = 1000): Future[Unit] = {
val p = Promise[Unit]
val timer = new Timer(true)
timer.schedule(new TimerTask() {
override def run() = {
p success ()
timer cancel()
}
}, t)
p future
}
def printDot = println(".")
val taskOfPrintingDot = new TimerTask {
override def run() = printDot
}
println("Enter a URL")
val lines = io.Source.stdin.getLines()
val url = if (lines hasNext) Some(lines next) else None
timer.schedule(taskOfPrintingDot, 0L, 50.millisecond.toMillis)
val timeOut2Sec = timeout(2.second.toMillis)
val htmlContents = Future {
url map { x =>
blocking {
Source fromURL (x) mkString
}
}
}
Future.firstCompletedOf(Seq(timeOut2Sec, htmlContents)) map { x =>
timer cancel ()
x match {
case Some(x) =>
println(x)
case _ =>
println("Error occured")
}
}
Thread sleep 5000
}
As #Gábor Bakos said exception produces Failure which doesn't handled by map:
val fut = Future { Some(Source fromURL ("hhhttp://google.com")) }
scala> fut map { x => println(x) } //nothing printed
res12: scala.concurrent.Future[Unit] = scala.concurrent.impl.Promise$DefaultPromise#5e025724
To process failure - use recover method :
scala> fut recover { case failure => None } map { x => println(x) }
None
res13: scala.concurrent.Future[Unit] = scala.concurrent.impl.Promise$DefaultPromise#578afc83
In your context it's something like:
Future.firstCompletedOf(Seq(timeOut2Sec, htmlContents)) recover {case x => println("Error:" + x); None} map { x => ...}
The Complete Code after using recover as advised by #dk14:
object Exercises extends App {
val timer = new Timer()
def timeout(t: Long = 1000): Future[Unit] = {
val p = Promise[Unit]
val timer = new Timer(true)
timer.schedule(new TimerTask() {
override def run() = {
p success ()
timer cancel ()
}
}, t)
p future
}
def printDot = println(".")
val taskOfPrintingDot = new TimerTask {
override def run() = {
printDot
}
}
println("Enter a URL")
val lines = io.Source.stdin.getLines()
val url = if (lines hasNext) Some(lines next) else None
timer.schedule(taskOfPrintingDot, 0L, 50.millisecond.toMillis)
val timeOut2Sec = timeout(2.second.toMillis)
val htmlContents = Future {
url map { x =>
blocking {
Source fromURL (x) mkString
}
}
}
Future.firstCompletedOf(Seq(timeOut2Sec, htmlContents))
.recover { case x => println("Error:" + x); None }
.map { x =>
timer cancel ()
x match {
case Some(x) =>
println(x)
case _ =>
println("Timeout occurred")
}
}
Thread sleep 5000
}
I am spawning a small number of actors to fetch, process and save RSS feed items to a database. This is done through a main method of an object running on cron. I create these actors and dole out jobs to them as they complete the previous job assigned to them. My main class spawns a single actor, the one that doles out jobs to a pool of actors. Eventually the main method seems to hang. It doesn't exit, but execution halts on all the actors. My CTO believes the main is exiting before the actors complete their work and leaving them, but I am not convinced that's the case. I receive no success exit on main (no exit at all).
Essentially I'm wondering how to debug these actors, and what possible reason could cause this to happen. Will main exit before actors have completed their execution (and if it does, does that matter?) From what I can tell actors using receive are mapped 1-to-1 to threads, correct? Code is below. Please ask any follow-up questions, help is greatly appreciated. I know I may not have provided sufficient detail, I'm new to scala and actors and will update as needed.
object ActorTester {
val poolSize = 10
var pendingQueue :Set[RssFeed] = RssFeed.pendingQueue
def main(args :Array[String]) {
val manager = new SpinnerManager(poolSize, pendingQueue)
manager.start
}
}
case object Stop
class SpinnerManager(poolSize :Int = 1, var pendingQueue :Set[RssFeed]) extends Actor {
val pool = new Array[Spinner](poolSize)
override def start() :Actor = {
for (i <- 0 to (poolSize - 1)) {
val spinner = new Spinner(i)
spinner.start()
pool(i) = spinner
}
super.start
}
def act() {
for {
s <- pool
if (!pendingQueue.isEmpty)
} {
s ! pendingQueue.head
pendingQueue = pendingQueue.tail
}
while(true) {
receive {
case id :Int => {
if (!pendingQueue.isEmpty) {
pool(id) ! pendingQueue.head
pendingQueue = pendingQueue.tail
} else if ((true /: pool) { (done, s) => {
if (s.getState != Actor.State.Runnable) {
val exited = future {
s ! Stop
done && true
}
exited()
} else {
done && false
}
}}) {
exit
}
}
}
}
}
}
class Spinner(id :Int) extends Actor {
def act() {
while(true) {
receive {
case dbFeed :RssFeed => {
//process rss feed
//this has multiple network requests, to the original blogs, bing image api
//our instance of solr - some of these spawn their own actors
sender ! id
}
case Stop => exit
}
}
}
}
For one thing you're making a tiny but important mistake when you're folding left in order to determine whether all Spinner actors have "terminated" or not. What you should do is evaluate to done && true resp. done && false at the end of the if cases, but currently you just say true resp. false without respect to done.
For example, imagine having 4 Spinner actors where the first and second ones were Runnable, the third one not, and the fourth one Runnable again. In that case the result of your foldleft would be true in spite of the fact that the third actor hasn't finished yet. If you were using a logical &&, you'd get the correct result.
This is possibily also what causes your application to hang.
EDIT: There also was an issue wrt a race condition. The following code works now, hope it helps. Anyway, I was wondering, doesn't Scala's actor implementation automatically make use of worker threads?
import actors.Actor
import scala.collection.mutable.Queue
case class RssFeed()
case class Stop()
class Spinner(id: Int) extends Actor {
def act() {
loop {
react {
case dbFeed: RssFeed => {
// Process RSS feed
sender ! id
}
case Stop => exit()
}
}
}
}
class SpinnerManager(poolSize: Int, pendingQueue: Queue[RssFeed]) extends Actor {
val pool = Array.tabulate(poolSize)(new Spinner(_).start())
def act() {
for (s <- pool; if (!pendingQueue.isEmpty)) {
pendingQueue.synchronized {
s ! pendingQueue.dequeue()
}
}
loop {
react {
case id: Int => pendingQueue.synchronized {
if (!pendingQueue.isEmpty) {
Console println id
pool(id) ! pendingQueue.dequeue()
} else {
if (pool forall (_.getState != Actor.State.Runnable)) {
pool foreach (_ ! Stop)
exit()
}
}
}
}
}
}
}
object ActorTester {
def main(args: Array[String]) {
val poolSize = 10
val pendingQueue: Queue[RssFeed] = Queue.tabulate(100)(_ => RssFeed())
new SpinnerManager(poolSize, pendingQueue).start()
}
}
So after several days of debugging I've solved this issue. fotNelton's code suggestions were very helpful in doing so, so I've given him a vote. However, they didn't address the problem itself. What I've found is that if you are running this in a main method then if the parent actors exit before their child actors then the program will hang forever and never exit, still holding all of its memory. In the process of handling the RSS feed, a Fetcher would spawn actors and send them messages to do things involving network requests. These actors need to complete their work before the parent actor quits. Fetcher wouldn't wait for these actors to finish though, once he sent the message he would just move on. So he would tell manager he was finished before his child actors had finished all their work. To deal with this, one option would be to use futures and wait until the actors are done (pretty slow). My solution was to create services accessible via URL (POST to a service that has an actor waiting to react). The service would respond right away, and send a message to its own actor. Thus the actors can quit once they send the request to the service, and don't need to spawn any other actors.
object FeedFetcher {
val poolSize = 10
var pendingQueue :Queue[RssFeed] = RssFeed.pendingQueue
def main(args :Array[String]) {
new FetcherManager(poolSize, pendingQueue).start
}
}
case object Stop
class FetcherManager(poolSize :Int = 1, var pendingQueue :Queue[RssFeed]) extends Actor {
val pool = new Array[Fetcher](poolSize)
var numberProcessed = 0
override def start() :Actor = {
for (i <- 0 to (poolSize - 1)) {
val fetcher = new Fetcher(i)
fetcher.start()
pool(i) = fetcher
}
super.start
}
def act() {
for {
f <- pool
if (!pendingQueue.isEmpty)
} {
pendingQueue.synchronized {
f ! pendingQueue.dequeue
}
}
loop {
reactWithin(10000L) {
case id :Int => pendingQueue.synchronized {
numberProcessed = numberProcessed + 1
if (!pendingQueue.isEmpty) {
pool(id) ! pendingQueue.dequeue
} else if ((true /: pool) { (done, f) => {
if (f.getState == Actor.State.Suspended) {
f ! Stop
done && true
} else if (f.getState == Actor.State.Terminated) {
done && true
} else {
false
}
}}) {
pool foreach { f => {
println(f.getState)
}}
println("Processed " + numberProcessed + " feeds total.")
exit
}
}
case TIMEOUT => {
if (pendingQueue.isEmpty) {
println("Manager just woke up from timeout with all feeds assigned.")
pool foreach { f => {
if (f.getState == Actor.State.Suspended) {
println("Sending Stop to Fetcher " + f.id)
f ! Stop
}
}}
println("Checking state of all Fetchers for termination.")
if ((true /: pool) { (done, f) => {
done && (f.getState == Actor.State.Terminated)
}}) {
exit
}
}
}
}
}
}
}
class Fetcher(val id :Int) extends Actor {
var feedsIveDone = 0
def act() {
loop {
react {
case dbFeed :RssFeed => {
println("Fetcher " + id + " starting feed")
//process rss feed here
feedsIveDone = feedsIveDone + 1
sender ! id
}
case Stop => {
println(id + " exiting")
println(feedsIveDone)
exit
}
}
}
}
suppose I have a sleep function:
def sleep(delay:Int) : Unit #suspendable = {
....
}
is it possible to have a function future that creates an async version of the sleep function that can be awaited on synchronously.
def future(targetFunc: (Int => Unit #suspendable)) : (Int => Future) = {
....
}
class Future {
def await : Unit #suspendable = {
....
}
}
you should be able to do something like this:
reset {
val sleepAsync = future(sleep)
val future1 = sleepAsync(2000)
val future2 = sleepAsync(3000)
future1.await
future2.await
/* finishes after a delay of 3000 */
}
the two calls to sleepAsync should appear to return straight away and the two calls to Future#await should appear to block. of course they all really fall off the end of reset and the code after is responsible for calling the continuation after the delay.
otherwise is there an alternative method to running two #suspendable functions in parallel and wait for both of them to complete?
I have a compilable gist with a skeleton of what i want to do: https://gist.github.com/1191381
object Forks {
import scala.util.continuations._
case class Forker(forks: Vector[() => Unit #suspendable]) {
def ~(block: => Unit #suspendable): Forker = Forker(forks :+ (() => block))
def joinIf(pred: Int => Boolean): Unit #suspendable = shift { k: (Unit => Unit) =>
val counter = new java.util.concurrent.atomic.AtomicInteger(forks.size)
forks foreach { f =>
reset {
f()
if (pred(counter.decrementAndGet)) k()
}
}
}
def joinAll() = joinIf(_ == 0)
def joinAny() = joinIf(_ == forks.size - 1)
}
def fork(block: => Unit #suspendable): Forker = Forker(Vector(() => block))
}
using fork(), we can now wait many "suspendables". use ~() to chain together suspendables. use joinAll() to wait for all suspendables and joinAny() to wait for just one. use joinIf() to customize join strategy.
object Tests extends App {
import java.util.{Timer, TimerTask}
import scala.util.continuations._
implicit val timer = new Timer
def sleep(ms: Int)(implicit timer: Timer): Unit #suspendable = {
shift { k: (Unit => Unit) =>
timer.schedule(new TimerTask {
def run = k()
}, ms)
}
}
import Forks._
reset {
fork {
println("sleeping for 2000 ms")
sleep(2000)
println("slept for 2000 ms")
} ~ {
println("sleeping for 4000 ms")
sleep(4000)
println("slept for 4000 ms")
} joinAll()
println("and we are done")
}
println("outside reset")
readLine
timer.cancel
}
and this is the output. program starts at time T:
sleeping for 2000 ms
sleeping for 4000 ms
outside reset <<<<<< T + 0 second
slept for 2000 ms <<<<<< T + 2 seconds
slept for 4000 ms <<<<<< T + 4 seconds
and we are done <<<<<< T + 4 seconds
I'm not sure that I completely understand the question, but here's a try:
import scala.util.continuations._
class Future(thread: Thread) {
def await = thread.join
}
object Future {
def sleep(delay: Long) = Thread.sleep(delay)
def future[A,B](f: A => B) = (a: A) => shift { k: (Future => Unit) =>
val thread = new Thread { override def run() { f(a) } }
thread.start()
k(new Future(thread))
}
def main(args:Array[String]) = reset {
val sleepAsync = future(sleep)
val future1 = sleepAsync(2000) // returns right away
val future2 = sleepAsync(3000) // returns right away
future1.await // returns after two seconds
future2.await // returns after an additional one second
// finished after a total delay of three seconds
}
}
Here, a Future instance is nothing more than a handle on a Thread, so you can use its join method to block until it finishes.
The future function takes a function of type A => B, and returns a function which, when supplied with an A will kick off a thread to run the "futured" function, and wrap it up in a Future, which is injected back into the continuation, thereby assigning it to val future1.
Is this anywhere close to what you were going for?