I am trying to use a TrampolineExecutionContext in ZIO in order to test background stream subscriptions on the same thread (so I can run effect in the order I would expect).
testM("Using trampoline execution context") {
(for {
queue <- Queue.unbounded[String]
_ <- ZStream
.fromQueue(queue)
.take(1)
.foreach(el => ZIO.effect(println(s"In Stream $el")))
.fork
_ <- queue.offer("Element")
_ <- ZIO.effect(println("Inside for comprehension")).on(trampolineExecutionContext)
} yield {
assert(1)(equalTo(1))
}).on(trampolineExecutionContext)
}
In this situation, I obtain what I would expect that is:
"In Stream Element", "Inside for comprehension"
If I remove the on(trampolineExecutionContext), I would obtain "Inside for comprehension" only because I am not joining the fiber (creating a sync point).
How can I set for the entire test the default context to be trampolineExecutionContext without repeating it every time in every call or in the important calls?
Maybe it's not exactly what you need, but you can try to override runner method of DefaultRunnableSpec and replace main context with TrampolineExecutionContext:
override def runner = {
super.runner.withPlatform( _.withExecutor(
Executor.fromExecutionContext(1)(
trampolineExecutionContext
)
))
}
In this case you will only need one on(trampolineExecutionContext) at the end of the test instead of two.
Related
I write a custom spark sink. In my addBatch method I use ForEachPartitionAsync which if I'm not wrong only makes the driver work asynchronously, returning a future.
val work: FutureAction[Unit] = rdd.foreachPartitionAsync { rows =>
val sourceInfo: StreamSourceInfo = serializeRowsAsInputStream(schema, rows)
val ackIngestion = Future {
ingestRows(sourceInfo) } andThen {
case Success(ingestion) => ackIngestionDone(partitionId, ingestion)
}
Await.result(ackIngestion, timeOut) // I would like to remove this line..
}
work onSuccess {
case _ => // move data from temporary table, report success of all workers
}
work onFailure{
//delete tmp data
case t => throw t.getCause
}
I can't find a way to run the worker nodes without blocking on the Await call, as if I remove them a success is reported to the work future object although the future didn't really finish.
Is there a way to report to the driver that all the workers finished
their asynchronous jobs?
Note: I looked at the foreachPartitionAsync function and it has only one implementation that expects a function that returns a Unit (i would've expected it to have another one returning a future or maybe a CountDownLatch..)
I have a method which connects to a websocket and gets stream messages from some really outside system.
The simplified version is:
def watchOrders(): Var[Option[Order]] = {
val value = Var[Option[Order]](None)
// onMessage( order => value.update(Some(order))
value
}
When I test it (with scalatest), I want to make it connect to the real outside system, and only check the first 4 orders:
test("watchOrders") {
var result = List.empty[Order]
val stream = client.watchOrders()
stream.foreach {
case Some(order) =>
result = depth :: result
if (result.size == 4) { // 1.
assert(orders should ...) // 2.
stream.kill() // 3.
}
case _ =>
}
Thread.sleep(10000) // 4.
}
I have 4 questions:
Is it the right way to check the first 4 orders? there is no take(4) method found in scala.rx
If the assert fails, the test still passes, how to fix it?
Is it the right way to stop the stream?
If the thread doesn't sleep here, the test will pass the code in case Some(order) never runs. Is there a better way to wait?
One approach you might consider to get a List out of a Var is to use the .fold combinator.
The other issue you have is dealing with the asynchronous nature of the data - assuming you really want to talk to this outside real world system in your test code (ie, this is closer to the integration test side of things), you are going to want to look at scalatest's support for async tests and will probably do something like construct a future out of a promise that you can complete when you accumulate the 4 elements in your list.
See: http://www.scalatest.org/user_guide/async_testing
In the following action it should return response immediately after hitting URL but instead it waits till all the Future blocks are started and then only sends response. It waits till "Starting for group 10" is logged in console even though "Returning from action" is logged immediately after hitting URL.
def test = Action { implicit request =>
Future.successful(0 to 150).foreach { items =>
items.grouped(15).zipWithIndex.foreach{ itemGroupWithIndex =>
val (itemGroup, index) = itemGroupWithIndex
Future {
logger.info("************** Starting for group " + index)
itemGroup.foreach { item =>
Thread.sleep(1000)
logger.info("Completed for item " + item)
}
}
}
}
logger.info("************** Returning from action **************")
Ok(views.html.test("test page"))
}
I am not able to understand reason behind this delay and how can i make this action send response immediately.
Play framework version 2.5.9
Your Action is not Async. You have a synchronous endpoint which is why you see the Returning from action printed immediately on the console. You should probably use the Action.async as your processing type. Using async Actions will drastically improve the overall performance of your application and is highly recommended when building high throughput and responsive web applications.
Two points in your code needs to change
Asynchronous Action: Because you are using Future, the action should be asynchronous: Action.async{...}.
No Blocking Code: The whole point of using Future and asynchronous programming is not to have a code that "blocks" the execution. So I suggest to remove the Thread.sleep(1000) part of the code.
Note that if you write your code non-blocking way; whenever the action method get the result; it will perform the required action(s), such as logging or providing the view.
This is because there are race conditions in your Futures.
You need to ensure that you are returning a single Future[T] and not a Future[Future[T]] (or any layered variances).
If the Futures are independent of each other, use Future.sequence
example:
def future: Future[String] = Future.successful("hi")
def action = Action.async { _ =>
val futures: Seq[Future[String]] = (1 to 50).map(_ => future()).toSeq
val oneFuture = Future.sequence(futures)
oneFuture
}
This will avoid race conditions
FYI, this has nothing to do with the Play framework. This is concurrent programming in scala.
From the well-written Akka Concurrency:
As I understand, the diagram points out, both numSummer and charConcat will run on the same thread.
Is it possible to run each Future in parallel, i.e. on separate threads?
The picture on the left is them running in parallel.
The point of the illustration is that the Future.apply method is what kicks off the execution, so if it doesn't happen until the first future's result is flatMaped (as in the picture on the right), then you don't get the parallel execution.
(Note that by "kicked off", i mean the relevant ExecutionContext is told about the job. How it parallelizes is a different question and may depend on things like the size of its thread pool.)
Equivalent code for the left:
val numSummer = Future { ... } // execution kicked off
val charConcat = Future { ... } // execution kicked off
numSummer.flatMap { numsum =>
charConcat.map { string =>
(numsum, string)
}
}
and for the right:
Future { ... } // execution kicked off
.flatMap { numsum =>
Future { ... } // execution kicked off (Note that this does not happen until
// the first future's result (`numsum`) is available.)
.map { string =>
(numsum, string)
}
}
Let's say you have a gui component and 10 threads all tell it to repaint at sufficiently the same time as they all arrive before a single paint operation takes place. Instead of naively wasting resources repainting 10 times, just merge/ignore all but the last one and repaint once (or more likely, twice--once for the first, and once for the last). My understanding is that the Swing repaint manager does this.
Is there a way to accomplish this same type of behavior in a Scala Actor? Is there a way to look at the queue and merge messages, or ignore all but the last of a certain type or something?
Something like this?:
act =
loop {
react {
case Repaint(a, b) => if (lastRepaint + minInterval < System.currentTimeMillis) {
lastRepaint = System.currentTimeMillis
repaint(a, b)
}
}
If you want to repaint whenever the actor's thread gets a chance, but no more, then:
(UPDATE: repainting using the last message arguments)
act =
loop {
react {
case r#Repaint(_, _) =>
var lastMsg = r
def findLast: Unit = {
reactWithin(0) {
case r#Repaint(_, _) =>
lastMsg = r
case TIMEOUT => repaint(lastMsg.a, lastMsg.b)
}
}
findLast
}
}