I expect the following code to result in syncResult containing "string two", but instead, I get the error java.lang.NoClassDefFoundError: Could not initialize class.
import scala.concurrent.{Await, Future}
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
def randomFunction1(): Future[String] = {
Future.successful("string one")
}
def randomFunction2(): Future[String] = {
Future.successful("string two")
}
val asyncResult: Future[String] = for {
r1 <- randomFunction1()
r2 <- randomFunction2()
} yield r2
val syncResult: String = Await.result(
asyncResult,
1.second
)
I get similar results with the following.
import scala.concurrent.{Await, Future}
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
def randomFunction1(): Future[String] = {
Future.successful("string one")
}
def randomFunction2(): Future[String] = {
Future.successful("string two")
}
val asyncResult: Future[String] = randomFunction1().flatMap(
r1 => {
randomFunction2()
}
)
val syncResult: String = Await.result(
asyncResult,
1.second
)
I'm using the Scala 2.12.2 interpreter to run this using :paste.
What's wrong with my code?
This seemed to be an issue with the Scala REPL.
Related
I use the below code from Databricks itself on how to run its notebook in parallel in Scala, https://docs.databricks.com/notebooks/notebook-workflows.html#run-multiple-notebooks-concurrently . I am trying to add retry feature where if one of the notebooks in the sequence failed, it will retry that notebook based on the retry value I passed to it.
Here is the parallel notebook code from Databricks:
//parallel notebook code
import scala.concurrent.{Future, Await}
import scala.concurrent.duration._
import scala.util.control.NonFatal
case class NotebookData(path: String, timeout: Int, parameters: Map[String, String] = Map.empty[String, String])
def parallelNotebooks(notebooks: Seq[NotebookData]): Future[Seq[String]] = {
import scala.concurrent.{Future, blocking, Await}
import java.util.concurrent.Executors
import scala.concurrent.ExecutionContext
import com.databricks.WorkflowException
val numNotebooksInParallel = 5
// If you create too many notebooks in parallel the driver may crash when you submit all of the jobs at once.
// This code limits the number of parallel notebooks.
implicit val ec = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(numNotebooksInParallel))
val ctx = dbutils.notebook.getContext()
Future.sequence(
notebooks.map { notebook =>
Future {
dbutils.notebook.setContext(ctx)
if (notebook.parameters.nonEmpty)
dbutils.notebook.run(notebook.path, notebook.timeout, notebook.parameters)
else
dbutils.notebook.run(notebook.path, notebook.timeout)
}
.recover {
case NonFatal(e) => s"ERROR: ${e.getMessage}"
}
}
)
}
This is an example of how I am calling the above code to run multiple examples notebooks:
import scala.concurrent.Await
import scala.concurrent.duration._
import scala.language.postfixOps
val notebooks = Seq(
NotebookData("Notebook1", 0, Map("client"->client)),
NotebookData("Notebook2", 0, Map("client"->client))
)
val res = parallelNotebooks(notebooks)
Await.result(res, 3000000 seconds) // this is a blocking call.
res.value
Here is one attempt. Since your code does not compile, I inserted a few dummy classes.
Also, you did not fully specify the desired behavior, so I made some assumptions. Only five retries will be made for each connection. If any of the Futures are still failing after five retries, then the entire Future is failed. Both of these behaviors can be changed, but since you did not specify, I am not sure what it is you want.
If you have questions or would like me to make an alteration to the program, let me know in the comments section.
object TestNotebookData extends App{
//parallel notebook code
import scala.concurrent.{Future, Await}
import scala.concurrent.duration._
import scala.util.control.NonFatal
case class NotebookData(path: String, timeout: Int, parameters: Map[String, String] = Map.empty[String, String])
case class Context()
case class Notebook(){
def getContext(): Context = Context()
def setContext(ctx: Context): Unit = ()
def run(path: String, timeout: Int, paramters: Map[String, String] = Map()): Seq[String] = Seq()
}
case class Dbutils(notebook: Notebook)
val dbutils = Dbutils(Notebook())
def parallelNotebooks(notebooks: Seq[NotebookData]): Future[Seq[Seq[String]]] = {
import scala.concurrent.{Future, blocking, Await}
import java.util.concurrent.Executors
import scala.concurrent.ExecutionContext
// This code limits the number of parallel notebooks.
implicit val ec = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(numNotebooksInParallel))
val ctx = dbutils.notebook.getContext()
val isRetryable = true
val retries = 5
def runNotebook(notebook: NotebookData): Future[Seq[String]] = {
def retryWrapper(retry: Boolean, current: Int, max: Int): Future[Seq[String]] = {
val fut = Future {runNotebookInner}
if (retry && current < max) fut.recoverWith{ _ => retryWrapper(retry, current + 1, max)}
else fut
}
def runNotebookInner() = {
dbutils.notebook.setContext(ctx)
if (notebook.parameters.nonEmpty)
dbutils.notebook.run(notebook.path, notebook.timeout, notebook.parameters)
else
dbutils.notebook.run(notebook.path, notebook.timeout)
}
retryWrapper(isRetryable, 0, retries)
}
Future.sequence(
notebooks.map { notebook =>
runNotebook(notebook)
}
)
}
val notebooks = Seq(
NotebookData("Notebook1", 0, Map("client"->"client")),
NotebookData("Notebook2", 0, Map("client"->"client"))
)
val res = parallelNotebooks(notebooks)
Await.result(res, 3000000 seconds) // this is a blocking call.
res.value
}
I found this to work:
import scala.util.{Try, Success, Failure}
def tryNotebookRun (path: String, timeout: Int, parameters: Map[String, String] = Map.empty[String, String]): Try[Any] = {
Try(
if (parameters.nonEmpty){
dbutils.notebook.run(path, timeout, parameters)
}
else{
dbutils.notebook.run(path, timeout)
}
)
}
//parallel notebook code
import scala.concurrent.{Future, Await}
import scala.concurrent.duration._
import scala.util.control.NonFatal
def runWithRetry(path: String, timeout: Int, parameters: Map[String, String] = Map.empty[String, String], maxRetries: Int = 2) = {
var numRetries = 0
while (numRetries < maxRetries){
tryNotebookRun(path, timeout, parameters) match {
case Success(_) => numRetries = maxRetries
case Failure(_) => numRetries = numRetries + 1
}
}
}
case class NotebookData(path: String, timeout: Int, parameters: Map[String, String] = Map.empty[String, String])
def parallelNotebooks(notebooks: Seq[NotebookData]): Future[Seq[Any]] = {
import scala.concurrent.{Future, blocking, Await}
import java.util.concurrent.Executors
import scala.concurrent.ExecutionContext
import com.databricks.WorkflowException
val numNotebooksInParallel = 5
// If you create too many notebooks in parallel the driver may crash when you submit all of the jobs at once.
// This code limits the number of parallel notebooks.
implicit val ec = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(numNotebooksInParallel))
val ctx = dbutils.notebook.getContext()
Future.sequence(
notebooks.map { notebook =>
Future {
dbutils.notebook.setContext(ctx)
runWithRetry(notebook.path, notebook.timeout, notebook.parameters)
}
.recover {
case NonFatal(e) => s"ERROR: ${e.getMessage}"
}
}
)
}
In akka, I want to put the elements in stream and return an object. I know the elements could be a source to run a graph. But how can I put the element and return an object on runtime?
import akka.actor.ActorSystem
import akka.stream.QueueOfferResult.{Dropped, Enqueued, Failure, QueueClosed}
import akka.stream.{ActorMaterializer, OverflowStrategy}
import akka.stream.scaladsl.{Keep, Sink, Source}
import scala.Array.range
import scala.util.Success
object StreamElement {
implicit val system = ActorSystem("StreamElement")
implicit val materializer = ActorMaterializer()
implicit val executionContext = system.dispatcher
def main(args: Array[String]): Unit = {
val (queue, value) = Source
.queue[Int](10, OverflowStrategy.backpressure)
.map(x => {
x * x
})
.toMat(Sink.asPublisher(false))(Keep.both)
.run()
range(0, 10)
.map(x => {
queue.offer(x).onComplete {
case Success(Enqueued) => {
}
case Success(Dropped) => {}
case _ => {
println("others")
}
}
})
}
}
How can I get the value returned?
Actually, you want to return the int value for each element.
So you could create the flow, then connect to source and Sink for each time.
package tech.parasol.scala.akka
import akka.actor.ActorSystem
import akka.stream.QueueOfferResult.{Dropped, Enqueued, Failure, QueueClosed}
import akka.stream.{ActorMaterializer, OverflowStrategy}
import akka.stream.scaladsl.{Flow, Keep, Sink, Source}
import scala.Array.range
import scala.util.Success
object StreamElement {
implicit val system = ActorSystem("StreamElement")
implicit val materializer = ActorMaterializer()
implicit val executionContext = system.dispatcher
val flow = Flow[Int]
.buffer(16, OverflowStrategy.backpressure)
.map(x => x * x)
def main(args: Array[String]): Unit = {
range(0, 10)
.map(x => {
Source.single(x).via(flow).runWith(Sink.head)
}.map( v => println("v ===> " + v)
))
}
}
It's unclear to me why the Scala collection isn't fed to the Stream as a Source in your sample code. Given that you've already composed a Stream with materialized values to be captured in a Source Queue and a publisher Sink, you could create a subscriber Source using Source.fromPublisher to collect the wanted values, as shown below:
import akka.actor.ActorSystem
import akka.stream.scaladsl._
import akka.stream._
implicit val system = ActorSystem("system")
implicit val materializer = ActorMaterializer() // Not needed for Akka 2.6+
val (queue, pub) = Source
.queue[Int](10, OverflowStrategy.backpressure)
.map(x => x * x)
.toMat(Sink.asPublisher(false))(Keep.both)
.run()
val fromQueue = Source(0 until 10).runForeach(queue.offer(_))
val source = Source.fromPublisher(pub)
source.runForeach(x => print(x + " "))
// Output:
// 0 1 4 9 16 25 36 49 64 81
Why does last line of following piece of code throw error in scala REPL?
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
// This is okay
val futureInt = Future[Int] { 42 }
val v1 = Await.result(futureInt, 1.second)
// This throw error: java.lang.NoClassDefFoundError: Could not initialize class $line8.$read$$iw$$iw$$iw$$iw$$iw$$iw$
val v2 = Await.result(Future[Int]{ 42 }, 1.second)
But when I create a main class and execute the same code it works fine:
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
object Main {
def main(args: Array[String]): Unit = {
val futureInt = Future[Int] { 42 }
val v1 = Await.result(futureInt, 1.second)
val v2 = Await.result(Future[Int] { 44 }, 1.second)
println(s"v1=$v1, v2=$v2 ")
}
}
Output:
v1=42, v2=44
This bug (including your exact source) is being tracked in this bug against scala:
https://github.com/scala/bug/issues/9076#issuecomment-292435527
I am trying to add support for serializing and deserializing LocalDateTime in akka-http. I'm a little confused by the following error:
Error:(42, 20) could not find implicit value for parameter um:
akka.http.scaladsl.unmarshalling.FromRequestUnmarshaller[SomeData]
entity(as[SomeData]) { evt => complete(StatusCodes.Created) }
Error:(42, 20) not enough arguments for method as: (implicit um:
akka.http.scaladsl.unmarshalling.FromRequestUnmarshaller[SomeData])akka.http.scaladsl.unmarshalling.FromRequestUnmarshaller[SomeData].
Unspecified value parameter um.
entity(as[SomeData]) { evt => complete(StatusCodes.Created) }
What an I missing in the following code? I pasted my code into the minimalist sample from the Akka-http documentation. I am running Akka-http 10.0.6.
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.http.scaladsl.server.Directives._
import akka.stream.ActorMaterializer
import scala.io.StdIn
import java.time.LocalDateTime
import java.time.format.DateTimeFormatter
import spray.json.{DefaultJsonProtocol, JsString, JsValue, JsonFormat}
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport
import akka.http.scaladsl.server.Route
final case class SomeData (dateTime: LocalDateTime)
trait JsonSupport extends SprayJsonSupport with DefaultJsonProtocol {
implicit val eventDataFormat: JsonFormat[SomeData] = jsonFormat1(SomeData)
implicit val localDateTimeFormat = new JsonFormat[LocalDateTime] {
private val iso_date_time = DateTimeFormatter.ISO_DATE_TIME
def write(x: LocalDateTime) = JsString(iso_date_time.format(x))
def read(value: JsValue) = value match {
case JsString(x) => LocalDateTime.parse(x, iso_date_time)
case x => throw new RuntimeException(s"Unexpected type ${x.getClass.getName} when trying to parse LocalDateTime")
}
}
// implicit val eventDataFormat: JsonFormat[SomeData] = jsonFormat1(SomeData) ** moved here **
}
object Main extends JsonSupport {
def main(args: Array[String]): Unit = {
implicit val system = ActorSystem("my-system")
implicit val materializer = ActorMaterializer()
implicit val executionContext = system.dispatcher
val route:Route =
path("hello") {
post {
entity(as[SomeData]) { evt => complete(StatusCodes.Created) }
}
}
val bindingFuture = Http().bindAndHandle(route, "localhost", 8080)
println(s"Server online at http://localhost:8080/\nPress RETURN to stop...")
StdIn.readLine() // let it run until user presses return
bindingFuture
.flatMap(_.unbind()) // trigger unbinding from the port
.onComplete(_ => system.terminate()) // and shutdown when done
}
}
implicit val eventDataFormat: JsonFormat[SomeData] = jsonFormat1(SomeData)
↓
implicit val eventDataFormat = jsonFormat1(SomeData)
I have an external future operation which I can override it's onOperation complete method . I want to wrap it and complete by closure of a Promise. however I couldn't complete that future within other future. for example :
import scala.concurrent.{Future, Promise}
import scala.util.{Failure, Success}
import scala.concurrent.ExecutionContext.Implicits.global
def foo():Future[String] = {
val p = Promise[String]()
channel.addListener(new ChannelFutureListener[IoReadFuture] {
override def operationComplete(future: IoReadFuture): Unit = {
p.success("hello")}
}
p.future
}
val x = foo()
x: scala.concurrent.Future[String] = List()
x.onComplete{
case Success(msg) => println(s"$msg world")
case Failure(e) => println(e.getMessage)
}
res1: Unit = ()
is there idiomatic way to do that without blocking?
I think you are trying to create a function which takes a delay as input and returns a delayed future.
If you want to do that, you can do it in a non-sleeping way using Await and Promise.
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.{Future, Promise}
import scala.concurrent.duration._
// returns a future delayed by `delay` seconds
def getDelayedFutureOfStringValue(delay: Int, value: String): Future[String] = {
// we will use this promise for wait only
val waitProxyPromise = Promise[Int]()
val delayedFuture = Await.ready(waitProxyPromise.future, delay.second).map({
case _ => value
})
delayedFuture
}
val helloFuture = getDelayedFutureOfStringValue(2, "hello")
The above amy seem like a decent implementation but it is actually not. Await actually blocks the thread. Sadly... there is not easy way to get a delayedFutue in a totally non-blocking way in idiomatic Scala.
You can get a nice non-blocking and non-sleeping delayed future using Timer utility from Java,
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.{Future, Promise}
import java.util.{Timer, TimerTask}
// delay is number of seconds
def getDelayedFutureOfStringValue(delay: Int, value: String): Future[String] = {
val promise = Promise[String]()
val timer = new Timer()
val timerTask = new TimerTask {
override def run(): Unit = promise.success(value)
}
timer.schedule(timerTask, delay * 1000)
promise.future
}
val niceHelloFuture = getDelayedFutureOfStringValue(2, "hello")
And in case you already have a future with you and you want to use that to compose a dependant future, then its pretty easy.
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.{Future, Promise}
import scala.util.{Failure, Success}
// I assume you IoReadFuture is either similar to Future or wraps the actual Future
def foo(value: String):Future[String] = {
val p = Promise[String]()
channel.addListener(new ChannelFutureListener[IoReadFuture] {
override def operationComplete(ioFuture: IoReadFuture): Unit = {
ioFuture.future.onComplete(_ => p.success(value))
}
}
p.future
}
scheduleOne of the actor scheduler is non blocking way to wait for some code to execute
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext
import ExecutionContext.Implicits.global
val p = Promise(doSomething())
val system = akka.actor.ActorSystem("system")
system.scheduler.scheduleOne(deplay seconds)(p.future)
val f = p.future
f.flatMap { println(s"${_}") }