ZIO Fiber orElse generate exception messages - scala

I want to use the combinator orElse on ZIO Fibers.
From docs:
If the first fiber succeeds, the composed fiber will succeed with its result; otherwise, the composed fiber will complete with the exit value of the second fiber (whether success or failure).
import zio._
import zio.console._
object MyApp extends App {
def f1 :Task[Int] = IO.fail(new Exception("f1 fail"))
def f2 :Task[Int] = IO.succeed(2)
val myAppLogic =
for {
f1f <- f1.fork
f2f <- f2.fork
ff = f1f.orElse(f2f)
r <- ff.join
_ <- putStrLn(s"Result is [$r]")
} yield ()
def run(args: List[String]) =
myAppLogic.fold(_ => 1, _ => 0)
}
I run it with sbt in console. And output:
[info] Running MyApp
Fiber failed.
A checked error was not handled.
java.lang.Exception: f1 fail
at MyApp$.f1(MyApp.scala:6)
at MyApp$.<init>(MyApp.scala:11)
at MyApp$.<clinit>(MyApp.scala)
at MyApp.main(MyApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Result is [2]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sbt.Run.invokeMain(Run.scala:93)
at sbt.Run.run0(Run.scala:87)
at sbt.Run.execute$1(Run.scala:65)
at sbt.Run.$anonfun$run$4(Run.scala:77)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at sbt.util.InterfaceUtil$$anon$1.get(InterfaceUtil.scala:10)
at sbt.TrapExit$App.run(TrapExit.scala:252)
at java.lang.Thread.run(Thread.java:748)
Fiber:Id(1574829590403,2) was supposed to continue to: <empty trace>
Fiber:Id(1574829590403,2) ZIO Execution trace: <empty trace>
Fiber:Id(1574829590403,2) was spawned by:
Fiber:Id(1574829590397,1) was supposed to continue to:
a future continuation at MyApp$.myAppLogic(MyApp.scala:12)
a future continuation at MyApp$.run(MyApp.scala:19)
Fiber:Id(1574829590397,1) ZIO Execution trace: <empty trace>
Fiber:Id(1574829590397,1) was spawned by:
Fiber:Id(1574829590379,0) was supposed to continue to:
a future continuation at zio.App.main(App.scala:57)
a future continuation at zio.App.main(App.scala:56)
[Fiber:Id(1574829590379,0) ZIO Execution trace: <empty trace>
I see the result of seconds Fiber, is Result is [2]
But why it output these unnecessary exception/warning messages?

By default a fiber failure warning is generated when a fiber that is not joined back fails so that errors do not get lost. But as you correctly note in some cases this is not necessary as the error is handled internally by the program logic, in this case by the orElse combinator. We have been working through a couple of other cases of spurious warnings being generated and I just opened a ticket for this one here. I expect we will have this resolved in the next release.

This happens because the default instance of Platform being created by zio.App has a default which reports uninterrupted, failed fibers to the console:
def reportFailure(cause: Cause[_]): Unit =
if (!cause.interrupted)
System.err.println(cause.prettyPrint)
To avoid this, you can provide your own Platform instance which doesn't do so:
import zio._
import zio.console._
import zio.internal.{Platform, PlatformLive}
object MyApp extends App {
override val Platform: Platform = PlatformLive.Default.withReportFailure(_ => ())
def f1: Task[Int] = IO.fail(new Exception("f1 fail"))
def f2: Task[Int] = IO.succeed(2)
val myAppLogic =
for {
f1f <- f1.fork
f2f <- f2.fork
ff = f1f.orElse(f2f)
r <- ff.join
_ <- putStrLn(s"Result is [$r]")
} yield ()
def run(args: List[String]) =
myAppLogic.fold(_ => 1, _ => 0)
}
Which yields:
Result is [2]
As #Adam Fraser noted, this will probably get fixed in a nearby release.
Edit:
Should be fixed after https://github.com/zio/zio/pull/2339 was merged

Related

Akka test case assertion failure

I am trying to write test for an actor which sends:
sender() ! Status.Failure(message)
I have written the test case as:
class TestActorSpec
extends TestKit(ActorSystem("system"))
with ImplicitSender
with WordSpecLike
with Matchers {
implicit val executionContext: ExecutionContextExecutor = system.dispatcher
implicit val futureDuration: FiniteDuration = 60.seconds
val timeout: FiniteDuration = 5.seconds
val testActorRef: ActorRef = system.actorOf(Props(new TestActor()), "test-actor")
val futureResult: Future[Any] = (testActorRef ? "Exception")(timeout)
"TestSpec" should {
"send the exception in {
expectMsgType[Status.Failure]
}
}
}
But i am getting the test failure with reason:
[info] java.lang.AssertionError: assertion failed: timeout (3 seconds) during expectMsgClass waiting for class akka.actor.Status$Failure
[info] at scala.Predef$.assert(Predef.scala:223)
If you want to receive Status.Failure as a message, you should not use ask pattern ? and use tell ! instead.
The reason is that ask pattern will consume the Failure and pass back a failed Future. In case of ask pattern, you need to assert a failed Future.
From Scala docs of akka.actor.Status.Failure
This class/message type is preferably used to indicate failure of some operation performed.
As an example, it is used to signal failure with AskSupport is used (ask/?).

Null pointer exception with GenericWriteAheadSink and FileCheckpointCommitter in Flink 1.8

I'm implementing a custom NSQ sink for Flink. I have it working as a subclass of RichSinkFunction, but I'd like to get the write-ahead log implementation working for extra data integrity.
Using O'Reilly's WriteAheadSinkExample available here, I attempted to implement my own:
package com.wistia.analytics
import java.net.{InetSocketAddress, SocketAddress}
import com.github.mitallast.nsq._
import org.apache.flink.api.scala.createTypeInformation
import java.lang.Iterable
import java.nio.file.{Files, Paths}
import java.util.UUID
import org.apache.commons.lang3.StringUtils
import org.apache.flink.api.common.ExecutionConfig
import org.apache.flink.streaming.runtime.operators.{CheckpointCommitter, GenericWriteAheadSink}
import scala.collection.mutable
class WALNsqSink(val topic: String) extends GenericWriteAheadSink[String](
// CheckpointCommitter that commits checkpoints to the local filesystem
new FileCheckpointCommitter(System.getProperty("java.io.tmpdir")),
// Serializer for records
createTypeInformation[String]
.createSerializer(new ExecutionConfig),
// Random JobID used by the CheckpointCommitter
UUID.randomUUID.toString) {
var client: NSQClient = _
var producer: NSQProducer = _
override def open(): Unit = {
val lookup = new NSQLookup {
def nodes(): List[SocketAddress] = List(new InetSocketAddress("127.0.0.1",4150))
def lookup(topic: String): List[SocketAddress] = List(new InetSocketAddress("127.0.0.1",4150))
}
client = NSQClient(lookup)
producer = client.producer()
}
def sendValues(readings: Iterable[String], checkpointId: Long, timestamp: Long): Boolean = {
val arr = mutable.Seq()
readings.forEach{ reading =>
arr :+ reading
}
producer.mpubStr(topic=topic, data=arr)
true
}
}
reusing FileCheckpointCommitter at the bottom of the class, but I get a null pointer exception inside GenericWriteAheadSink:
Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:638)
at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:123)
at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
at com.wistia.analytics.NsqProcessor$.main(NsqProcessor.scala:24)
at com.wistia.analytics.NsqProcessor.main(NsqProcessor.scala)
Caused by: java.lang.NullPointerException
at org.apache.flink.streaming.runtime.operators.GenericWriteAheadSink.processElement(GenericWriteAheadSink.java:277)
at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:748)
Nonzero exit code returned from runner: 1
(Compile / run) Nonzero exit code returned from runner: 1
Total time: 45 s, completed Feb 10, 2020 6:41:06 PM
I have no idea where to go from here. Any help appreciated
The issue here is most certainly the fact that You never call the open() method of the superclass. This will cause some of the variables to be uninitialized.
This should be solved by calling the super.open() inside Your open() method.

Why does my Scala future not propagate an Error?

This code works as expected:
it("should propagate exceptions") {
intercept[RuntimeException] {
val future = Future { Thread.sleep(10); sys.error("whoops"); 22 }
Await.result(future, Duration.Inf)
}.getMessage should equal ("whoops")
}
But this doesn't:
it("should propagate errors") {
intercept[StackOverflowError] {
val future = Future { Thread.sleep(10); throw new StackOverflowError("dang"); 22 }
Await.result(future, Duration.Inf)
}.getMessage should equal ("dang")
}
The future in this second test never returns. Why doesn't an Error subclass (as opposed to an Exception subclass) terminate my future? How should I handle Errors?
EDIT: This is possibly related, but not identical, to Why does Scala Try not catching java.lang.StackOverflowError?. I'm not using Try here. The core issue is that the Future never returns at all; I can't catch any error from it because it just hangs.
The reporter facility is for catastrophes, which just hooks into the thread's UncaughtExceptionHandler, but it looks like it works out of the box with just the default thread factory:
scala 2.13.0-M5> import concurrent._,java.util.concurrent.Executors
import concurrent._
import java.util.concurrent.Executors
scala 2.13.0-M5> val ec = ExecutionContext.fromExecutor(null, e => println(s"Handle: $e"))
ec: scala.concurrent.ExecutionContextExecutor = scala.concurrent.impl.ExecutionContextImpl$$anon$3#5e7c141d[Running, parallelism = 4, size = 0, active = 0, running = 0, steals = 0, tasks = 0, submissions = 0]
scala 2.13.0-M5> val f = Future[Int](throw new NullPointerException)(ec)
f: scala.concurrent.Future[Int] = Future(<not completed>)
scala 2.13.0-M5> f
res0: scala.concurrent.Future[Int] = Future(Failure(java.lang.NullPointerException))
scala 2.13.0-M5> val f = Future[Int](throw new StackOverflowError)(ec)
Handle: java.lang.StackOverflowError
f: scala.concurrent.Future[Int] = Future(<not completed>)
whereas
scala 2.13.0-M5> val ec = ExecutionContext.fromExecutor(Executors.newSingleThreadExecutor, e => println(s"Handle: $e"))
ec: scala.concurrent.ExecutionContextExecutor = scala.concurrent.impl.ExecutionContextImpl#317a118b
scala 2.13.0-M5> val f = Future[Int](throw new StackOverflowError)(ec)
f: scala.concurrent.Future[Int] = Future(<not completed>)
Exception in thread "pool-1-thread-1" java.lang.StackOverflowError
at $line14.$read$$iw$$iw$$iw$$iw$.$anonfun$f$1(<console>:1)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
at scala.util.Success.$anonfun$map$1(Try.scala:261)
at scala.util.Success.map(Try.scala:209)
at scala.concurrent.impl.Promise$Transformation.doMap(Promise.scala:420)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:402)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
You could construct a rig that registers a future when it runs, and a safe await that knows when threads have blown up. Maybe you want to retry an algorithm with a lower max recursion depth, for example.
As pointed out in the comments, this is a duplicate of Why does Scala Try not catching java.lang.StackOverflowError?
According to Scala documentation.
Note: only non-fatal exceptions are caught by the combinators on Try (see >scala.util.control.NonFatal). Serious system errors, on the other hand, will be >thrown.
No Throwable -> Errors are catched by Try
Also to answer your question about how error handling is usually done.
In Scala you can use try / catch for code that can cause exceptions (very similar to Java):
try {
// ... Your dangerous code in here
} catch {
case ioe: IOException => ... //
case e: Exception => ...
}
And you should always have the more specific exceptions first.
The code you provided would look something like this:
https://scastie.scala-lang.org/2DJXJ6ESS9ySJZSwSodmZg
Also I tried out your code and it definitely produces the StackOverFlowerror.
But it can't catch it properly like the above mentioned link explains.

How to retry failed Unmarshalling of a stream of akka-http requests?

I know it's possible to restart an akka-stream on error with a supervision strategy on the ActorMaterialzer
val decider: Supervision.Decider = {
case _: ArithmeticException => Supervision.Resume
case _ => Supervision.Stop
}
implicit val materializer = ActorMaterializer(
ActorMaterializerSettings(system).withSupervisionStrategy(decider))
val source = Source(0 to 5).map(100 / _)
val result = source.runWith(Sink.fold(0)(_ + _))
// the element causing division by zero will be dropped
// result here will be a Future completed with Success(228)
source: http://doc.akka.io/docs/akka/2.4.2/scala/stream/stream-error.html
I have the following use case.
/***
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-http-experimental" % "2.4.2",
"com.typesafe.akka" %% "akka-http-spray-json-experimental" % "2.4.2"
)
*/
import akka.http.scaladsl.unmarshalling.Unmarshal
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
import spray.json._
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import Uri.Query
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.stream.scaladsl._
import scala.util.{Success, Failure}
import scala.concurrent.Await
import scala.concurrent.duration.Duration
import scala.concurrent.Future
object SO extends DefaultJsonProtocol {
implicit val system = ActorSystem()
import system.dispatcher
implicit val materializer = ActorMaterializer()
val httpFlow = Http().cachedHostConnectionPoolHttps[HttpRequest]("example.org")
def search(query: Char) = {
val request = HttpRequest(uri = Uri("https://example.org").withQuery(Query("q" -> query.toString)))
(request, request)
}
case class Hello(name: String)
implicit val helloFormat = jsonFormat1(Hello)
val searches =
Source('a' to 'z').map(search).via(httpFlow).mapAsync(1){
case (Success(response), _) => Unmarshal(response).to[Hello]
case (Failure(e), _) => Future.failed(e)
}
def main(): Unit = {
Await.result(searches.runForeach(_ => println), Duration.Inf)
()
}
}
Sometime a query will fail to unmarshall. I want to use a retry strategy on that single query
https://example.org/?q=v without restarting the whole alphabet.
I think it will be hard (or impossible) to implement it with a supervsior strategy, mostly because you want to retry "n" times (according to the discussion in comments), and I don't think you can track the number of times the element was tried when using supervision.
I think there are two ways to solve this issue. Either handle the risky operation as a separate stream or create a graph, which will do error handling. I will propose two solutions.
Note also that Akka Streams distinguishes between errors and failures, so if you wont' handle your failures they will eventually collapse the flow (if no strategy is intriduced), so in the example below I convert them to Either, which represent either success or error.
Separate stream
What you can do is to treat each alphabet letter as a separate stream and handle failures for each letter separately with the retry strategy, and some delay.
// this comes after your helloFormat
// note that the method is somehow simpler because it's
// using implicit dispatcher and scheduler from outside scope,
// you may also want to pass it as implicit arguments
def retry[T](f: => Future[T], delay: FiniteDuration, c: Int): Future[T] =
f.recoverWith {
// you may want to only handle certain exceptions here...
case ex: Exception if c > 0 =>
println(s"failed - will retry ${c - 1} more times")
akka.pattern.after(delay, system.scheduler)(retry(f, delay, c - 1))
}
val singleElementFlow = httpFlow.mapAsync[Hello](1) {
case (Success(response), _) =>
val f = Unmarshal(response).to[Hello]
f.recoverWith {
case ex: Exception =>
// see https://github.com/akka/akka/issues/20192
response.entity.dataBytes.runWith(Sink.ignore).flatMap(_ => f)
}
case (Failure(e), _) => Future.failed(e)
}
// so the searches can either go ok or not, for each letter, we will retry up to 3 times
val searches =
Source('a' to 'z').map(search).mapAsync[Either[Throwable, Hello]](1) { elem =>
println(s"trying $elem")
retry(
Source.single(elem).via(singleElementFlow).runWith(Sink.head[Hello]),
1.seconds, 3
).map(ok => Right(ok)).recover { case ex => Left(ex) }
}
// end
Graph
This method will integrate failures into the graph, and will allow for retries. This example makes all requests run in parallel and prefer to retry those which failed, but if you don't want this behaviour and run them one by one this is something you can also do I believe.
// this comes after your helloFormat
// you may need to have your own class if you
// want to propagate failures for example, but we will use
// right value to keep track of how many times we have
// tried the request
type ParseResult = Either[(HttpRequest, Int), Hello]
def search(query: Char): (HttpRequest, (HttpRequest, Int)) = {
val request = HttpRequest(uri = Uri("https://example.org").withQuery(Query("q" -> query.toString)))
(request, (request, 0)) // let's use this opaque value to count how many times we tried to search
}
val g = GraphDSL.create() { implicit b =>
import GraphDSL.Implicits._
val searches = b.add(Flow[Char])
val tryParse =
Flow[(Try[HttpResponse], (HttpRequest, Int))].mapAsync[ParseResult](1) {
case (Success(response), (req, tries)) =>
println(s"trying parse response to $req for $tries")
Unmarshal(response).to[Hello].
map(h => Right(h)).
recoverWith {
case ex: Exception =>
// see https://github.com/akka/akka/issues/20192
response.entity.dataBytes.runWith(Sink.ignore).map { _ =>
Left((req, tries + 1))
}
}
case (Failure(e), _) => Future.failed(e)
}
val broadcast = b.add(Broadcast[ParseResult](2))
val nonErrors = b.add(Flow[ParseResult].collect {
case Right(x) => x
// you may also handle here Lefts which do exceeded retries count
})
val errors = Flow[ParseResult].collect {
case Left(x) if x._2 < 3 => (x._1, x)
}
val merge = b.add(MergePreferred[(HttpRequest, (HttpRequest, Int))](1, eagerComplete = true))
// #formatter:off
searches.map(search) ~> merge ~> httpFlow ~> tryParse ~> broadcast ~> nonErrors
merge.preferred <~ errors <~ broadcast
// #formatter:on
FlowShape(searches.in, nonErrors.out)
}
def main(args: Array[String]): Unit = {
val source = Source('a' to 'z')
val sink = Sink.seq[Hello]
source.via(g).toMat(sink)(Keep.right).run().onComplete {
case Success(seq) =>
println(seq)
case Failure(ex) =>
println(ex)
}
}
Basically what happens here is we run searches through httpFlow and then try to parse the response, we
then broadcast the result and split errors and non-errors, the non errors go to sink, and errors get sent
back to the loop. If the number of retries exceed the count, we ignore the element, but you can also do
something else with it.
Anyway I hope this gives you some idea.
For the streams solution above, any retries for the last element in the stream won't execute. That's because when the upstream completes after sending the last element the merge will also complete. After that the only output came come from the non-retry outlet but since the element goes to retry that gets completed too.
If you need all input elements to generate an output you'll need an extra mechanism to stop the upstream complete from reaching the process&retry graph. One possibility is to use a BidiFlow which monitors the input and output from the process&retry graph to ensure all the required outputs have been generated (for the observed inputs) before propagating the oncomplete. In the simple case that could just be counting input and output elements.

Slick 3.0-RC3 fails with java.util.concurrent.RejectedExecutionException

I'm trying to get familiar with Slick 3.0 and Futures (using Scala 2.11.6). I use simple code based on Slick's Multi-DB Cake Pattern example. Why does the following code terminate with an exception and how to fix it?
import scala.concurrent.Await
import scala.concurrent.duration._
import slick.jdbc.JdbcBackend.Database
import scala.concurrent.ExecutionContext.Implicits.global
class Dispatcher(db: Database, dal: DAL) {
import dal.driver.api._
def init() = {
db.run(dal.create)
try db.run(dal.stuffTable += Stuff(23,"hi"))
finally db.close
val x = {
try db.run(dal.stuffTable.filter(_.serial === 23).result)
finally db.close
}
// This crashes:
val result = Await.result(x, 2 seconds)
}
}
Execution fails with:
java.util.concurrent.RejectedExecutionException: Task slick.backend.DatabaseComponent$DatabaseDef$$anon$2#5c73f637 rejected from java.util.concurrent.ThreadPoolExecutor#4129c44c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:136)
at slick.backend.DatabaseComponent$DatabaseDef$class.runSynchronousDatabaseAction(DatabaseComponent.scala:224)
at slick.jdbc.JdbcBackend$DatabaseDef.runSynchronousDatabaseAction(JdbcBackend.scala:38)
at slick.backend.DatabaseComponent$DatabaseDef$class.runInContext(DatabaseComponent.scala:201)
at slick.jdbc.JdbcBackend$DatabaseDef.runInContext(JdbcBackend.scala:38)
at slick.backend.DatabaseComponent$DatabaseDef$class.runInternal(DatabaseComponent.scala:75)
at slick.jdbc.JdbcBackend$DatabaseDef.runInternal(JdbcBackend.scala:38)
at slick.backend.DatabaseComponent$DatabaseDef$class.run(DatabaseComponent.scala:72)
at slick.jdbc.JdbcBackend$DatabaseDef.run(JdbcBackend.scala:38)
at Dispatcher.init(Dispatcher.scala:15)
at SlickDemo$.main(SlickDemo.scala:16)
at SlickDemo.main(SlickDemo.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
I think that something is not correct in what you are trying to do: Slick's run method doesn't return Unit and doesn't fail with an exception - as it used to in previous versions. run now returns a Future, so if you want to run actions in sequence you need to flatMap the steps, or use a for-comprehension:
def init() = {
val = results for {
_ <- db.run(dal.create)
_ <- db.run(dal.stuffTable += Stuff(23, "hi"))
r <- db.run(dal.stuffTable.filter(_.serial === 23).result)
} yield r
}
I am not sure that you really need to use db.close that way: that is actually what may be causing the error (i.e. the db is closed in concurrence with the future that runs the actual queries so the execution can't happen).
If you want to handle errors use Future's capabilities, e.g.:
result.onFailure { case NonFatal(ex) => // do something with the exception }