It is easy to handle a particular set of exceptions using the Catch operator. How could we ignore certain exceptions but handle the rest in the Catch block?
Say for example I would like to let ArgumentNullException, ArgumentOutOfRangeException and TimeoutException exceptions bubble and report an error but for anything else I would like to retry.
The catch version below would catch all the exceptions, and there is no way to selectively ignore some types, as the signature requires it to return a IObservable<T>
source.Catch<T, Exception>(e=> Observable.Return(default(T)))
If I have to retry on certain Exception then I could write something like (I think)
source.Catch<T, WebException>(e=> source.Retry())
A very meaningful operator in these situations is RetryWhen.
RetryWhen(selector) gives you an observable of Exception, and expects an observable which, triggers a retry every time it produces a value.
RetryWhen(e => e) is equivalent to Retry() (all exceptions trigger a retry).
RetryWhen(e => e.Take(3)) is equivalent to Retry(3) (first three exceptions trigger a retry).
Now you can simply transform the inner sequence into one that emits on your desired types of exceptions. For example, to only retry on TimeoutException:
source.RetryWhen(e => e.Where(exn => exn is TimeoutException))
Related
In the Dart docs at https://dart.dev/guides/language/language-tour#exceptions, it states the following:
Dart provides Exception and Error types, as well as numerous predefined subtypes. You can, of course, define your own exceptions. However, Dart programs can throw any non-null object—not just Exception and Error objects—as an exception.
The example they give for this behavior is throw 'Out of llamas!';.
Why would I ever want to throw something that isn't an Error or Exception? What is the design decision behind allowing this?
I think it is because you may already have an object that you want to inspect if an error occurs, and can just throw that object, or just so you can throw a string like in the example.
It's worth noting that a catch block can optionally catch the stack trace, and since the stack trace is not part of the Exception, it makes sense to allow arbitrary objects to be thrown.
try {
throw 'Error!';
} catch (error, stacktrace) {
print(stacktrace);
}
Many times you will just see catch (e) in code, but you may also see catch (e, s).
I am saving DStream to Cassandra. There is a column in Cassandra with map<text, text> datatype. Cassandra does not support null value in Map, but null value can occur in the stream.
I have added try catch if case something goes wrong, but the program stopped despite that and I don't see error message in the log:
try {
cassandraStream.saveToCassandra("table", "keyspace")
} catch {
case e: Exception => log.error("Error in saving data in Cassandra" + e.getMessage, e)
}
Exception
Caused by: java.lang.NullPointerException: Map values cannot be null
at com.datastax.driver.core.TypeCodec$AbstractMapCodec.serialize(TypeCodec.java:2026)
at com.datastax.driver.core.TypeCodec$AbstractMapCodec.serialize(TypeCodec.java:1909)
at com.datastax.driver.core.AbstractData.set(AbstractData.java:530)
at com.datastax.driver.core.AbstractData.set(AbstractData.java:536)
at com.datastax.driver.core.BoundStatement.set(BoundStatement.java:870)
at com.datastax.spark.connector.writer.BoundStatementBuilder.com$datastax$spark$connector$writer$BoundStatementBuilder$$bindColumnUnset(BoundStatementBuilder.scala:73)
at com.datastax.spark.connector.writer.BoundStatementBuilder$$anonfun$6.apply(BoundStatementBuilder.scala:84)
at com.datastax.spark.connector.writer.BoundStatementBuilder$$anonfun$6.apply(BoundStatementBuilder.scala:84)
at com.datastax.spark.connector.writer.BoundStatementBuilder$$anonfun$bind$1.apply$mcVI$sp(BoundStatementBuilder.scala:106)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
at com.datastax.spark.connector.writer.BoundStatementBuilder.bind(BoundStatementBuilder.scala:101)
at com.datastax.spark.connector.writer.GroupingBatchBuilder.next(GroupingBatchBuilder.scala:106)
at com.datastax.spark.connector.writer.GroupingBatchBuilder.next(GroupingBatchBuilder.scala:31)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at com.datastax.spark.connector.writer.GroupingBatchBuilder.foreach(GroupingBatchBuilder.scala:31)
at com.datastax.spark.connector.writer.TableWriter$$anonfun$writeInternal$1.apply(TableWriter.scala:233)
at com.datastax.spark.connector.writer.TableWriter$$anonfun$writeInternal$1.apply(TableWriter.scala:210)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:112)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:111)
at com.datastax.spark.connector.cql.CassandraConnector.closeResourceAfterUse(CassandraConnector.scala:145)
at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:111)
at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:210)
at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:197)
at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:183)
at com.datastax.spark.connector.streaming.DStreamFunctions$$anonfun$saveToCassandra$1$$anonfun$apply$1.apply(DStreamFunctions.scala:54)
at com.datastax.spark.connector.streaming.DStreamFunctions$$anonfun$saveToCassandra$1$$anonfun$apply$1.apply(DStreamFunctions.scala:54)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
... 3 more
I'd like to know why the program got stops, despite the try/catch block. Why is the exception not caught?
To understand the source of the failure you have to acknowledge that DStreamFunctions.saveToCassandra, same as DStream output operations in general, is not an action in strict sense. In practice it just invokes foreachRDD:
dstream.foreachRDD(rdd => rdd.sparkContext.runJob(rdd, writer.write _))
which in turn:
Apply a function to each RDD in this DStream. This is an output operator, so 'this' DStream will be registered as an output stream and therefore materialized.
The difference is subtle, but important - the operation is registered but the actual execution happens in different context, at later point in time.
It means there are no runtime failures to caught at the point you invoke saveToCassandra.
As already pointed out, try or Try would contain the driver exception, if applied directly on an action. So you'd for example re-implement saveToCassandra as
dstream.foreachRDD(rdd => try {
rdd.sparkContext.runJob(rdd, writer.write _)
} catch {
case e: Exception => log.error("Error in saving data in Cassandra" + e. getMessage, e)
})
the stream should be able to proceed, although the current batch will be completely or partially lost.
It is important to note that this is not the same as catching the original exception, which will be thrown, uncaught and visible in the log. To catch problem at its source you'd have to apply try / catch block directly in writer, and this is obviously not an option when you execute code, over which you don't have control.
Take away message is (already stated in this thread) - make sure to sanitize your data to avoid known sources of failure.
The problem is that you don't catch the exception you think you do. The code you have will catch a driver exception, and in fact code structured like this will do it.
It doesn't however mean that
the program should never stop.
While driver failure, which would be a consequence of fatal executor failure, is contained and driver can exit gracefully, stream as such is already gone. Therefore your code exits, because there is no more stream to run.
If the code in question was under your control, exception handling should be delegated to the task, but in case of 3rd party code, there is no such option.
Instead you should validate your data, and remove problematic records, before these are passed to saveToCassandra.
i.e., by passing the error condition and not halting the entire Observable?
My Observable starts with a user-supplied list of package tracking numbers from common delivery services (FedEx, UPS, DHL, etc), looks up the expected delivery date online, then returns those dates in terms of number of days from today (i.e. "in 3 days" rather than "Jan 22"). The problem is that if any individual lookup results in an exception, the entire stream halts, and the rest of the codes won't be looked up. There's no ability to gracefully handle, say, UnknownTrackingCode Exception, and so the Observable can't guarantee that it will look up all the codes the user submitted.
public void getDaysTillDelivery(List<String> tracking_code_list) {
Observable o = Observable.from(tracking_code_list)
// LookupDeliveryDate performs network calls to UPS, FedEx, USPS web sites or APIs
// it might throw: UnknownTrackingCode Exception, NoResponse Exception, LostPackage Exception
.map(tracking_code -> LookupDeliveryDate(tracking_code))
.map(delivery_date -> CalculateDaysFromToday(delivery_date));
o.subscribe(mySubscriber); // will handle onNext, onError, onComplete
}
Halting the Observable stream as a result of one error is by design:
http://reactivex.io/documentation/operators/catch.html
Handling Exceptions in Reactive Extensions without stopping sequence
https://groups.google.com/forum/#!topic/rxjava/trm2n6S4FSc
The default behavior can be overcome, but only by eliminating many of the benefits of Rx in the first place:
I can wrap LookupDeliveryDate so it returns Dates in place of Exceptions (such as 1899-12-31 for UnknownTrackingCode Exception) but this prevents "loosely coupled code", because CalculateDaysFromToday would need to handle these special cases
I can surround each anonymous function with try/catch and blocks, but this essentially prevents me from using lambdas
I can use if/thens to direct the code path, but this will likely require maintaining some state and eliminating deterministic evaluation
Error handling of each step, obviously, prevents consolidating all error handling in the Subscriber
Writing my own error-handling operator is possible, but thinly documented
Is there a better way to handle this?
What exactly do you want to happen if there is an error? Do you just want to throw that entry away or do you want something downstream to do something with it?
If you want something downstream to take some action, then you are really turning the error into data (sort of like your example of returning a sentinel value of 1899-12-31 to represent the error). This strategy by definition means that everything downstream needs to understand that the data stream may contain errors instead of data and they must be modified to deal with it.
But rather than yielding a magic value, you can turn your Observable stream into a stream of Either values. Either the value is a date, or it is an error. Everything downstream receives this Either object and can ask it if it has a value or an error. If it has a value, they can produce a new Either object with the result of their calculation. If it has an error and they cannot do anything with it, they can yield an error Either themselves.
I don't know Java syntax, but this is what it might look like in c#:
Observable.From(tracking_code_list)
.Select(t =>
{
try { return Either.From(LookupDeliveryDate(t)); }
catch (Exception e)
{
return Either.FromError<Date>(e);
}
})
.Select(dateEither =>
{
return dateEither.HasValue ?
Either.From(CalculateDaysFromToday(dateEither.Value)) :
Either.FromError<int>(dateEither.Error);
})
.Subscribe(value =>
{
if (value.HasValue) mySubscriber.OnValue(value.Value);
else mySubscribe.OnError(value.Error);
});
Your other option is the "handle"/suppress the error when it occurs. This may be sufficient depending on your needs. In this case, just have LookupDeliveryDate return magic dates instead of exceptions and then add a .filter to filter out the magic dates before they get to CalculateDaysFromToay.
If a Scala future fails, and there is no continuation that "observes" that failure (or the only continuations use map/flatMap and don't run in case of failure), then errors go undetected. I would like such errors to be at least logged, so I can find bugs.
I use the term "observed error" because in .Net Tasks there is the chance to catch "unobserved task exceptions", when the Task object is collected by the GC. Similarly, with synchronous methods, uncaught exceptions that terminate the thread can be logged.
In Scala futures, to 'observe' a failure would mean that some continuation or other code reads the Exception stored in the future value before that future is disposed. I'm aware that finalization is not deterministic or reliable, and presumably that's why it's not used to catch unhandled errors, although .Net does succeed in doing this.
Is there a way to achieve this in Scala? If not, how should I organize my code to prevent unhandled error bugs?
Today I have andThen checkResult appended to various futures. But it's hard to know when to use this and when not to: if a library method returns a Future, it shouldn't checkResult and log errors itself, because the library user may handle the failure, so the responsibility falls onto the user. As I edit code I sometimes need to add checks and sometimes to remove them, and such manual management is surely wrong.
I have concluded there is no way to generically notice unhandled errors in Scala futures.
You can just use Future.recover in the function that returns the Future.
So for instance, you could just "log" the error and rethrow the original exception, in the simplest case:
def libraryFunction(): Future[Int] = {
val f = ...
f.recover {
case NonFatal(t) =>
println("Error : " + t)
throw t
}
}
Note the use of NonFatal to match all the exception types it is sensible to catch.
That recover block could equally return an alternative result if you wish.
I am writing a web app where exceptions are used to handle error cases. Often, I find myself writing helpers like this:
def someHelper(...) : Boolean {...}
and then using it like this:
if (!someHelper(...)){
throw new SomeException()
}
These exceptions represent things like invalid parameters, and when handled they send a useful error message to the user, eg
try {
...
} catch {
case e: SomeException => "Bad user!"
}
Is this a reasonable approach? And how could I pass the exception into the helper function and have it thrown there? I have had trouble constructing a type for such a function.
I use Either most of the time, not exceptions. I generally use exceptions, as you have done or some similar way, when the control flow has to go way, way back to some distant point, and otherwise there's nothing sensible to do. However, when the exceptions can be handled fairly locally, I will instead
def myMethod(...): Either[String,ValidatedInputForm] = {
...
if (!someHelper(...)) Left("Agree button not checked")
else Right(whateverForm)
}
and then when I call this method, I can
myMethod(blah).fold({ err =>
doSomething(err)
saneReturnValue
}, { form =>
foo(form)
form.usefulField
})
or match on Left(err) vs Right(form), or various other things.
If I don't want to handle the error right there, but instead want to process the return value, I
myMethod(blah).right.map{ form =>
foo(form)
bar(form)
}
and I'll get an Either with the error message unchanged as a Left, if it was an error message, or with the result of { foo(form); bar(form) } as a Right if it was okay. You can also chain your error processing using flatMap, e.g. if you wanted to perform an additional check on so-far-correct values and reject some of them, you could
myMethod(blah).right.flatMap{ form =>
if (!checkSomething(form)) Left("Something didn't check out.")
else Right(form)
}
It's this sort of processing that makes using Either more convenient (and usually better-performing, if exceptions are common) than exceptions, which is why I use them.
(In fact, in very many cases I don't care why something went wrong, only that it went wrong, in which case I just use an Option.)
There's nothing special about passing an exception instance to some method:
def someMethod(e: SomeException) {
throw e
}
someMethod(new SomeException)
But I have to say that I get a very distinct feeling that your whole idea just smells. If you want to validate a user input just write validators, e.g. UserValidator which will have some method like isValid to test a user input and return a boolean, you can implement some messaging there too. Exceptions are really intended for different purposes.
The two most common ways to approach what you're trying to do is to either just have the helper create and throw an exception itself, or exactly what you're doing: have the calling code check the results, and throw a meaningful exception, if needed.
I've never seen a library where you pass in the exception you expect the helper to throw. As I said on another answer, there's a surprisingly substantial cost to simply instantiating an exception, and if you followed this pattern throughout your code you could see an overall performance problem. This could be mitigated through the use of by-name parameters, but if you just forget to put => in a few key functions, you've got a performance problem that's difficult to track down.
At the end of the day, if you want the helper to throw an exception, it makes sense that the helper itself already knows what sort of exception it wants to throw. If I had to choose between A and B:
def helperA(...) { if (stuff) throw new InvalidStuff() }
def helperB(..., onError: => Exception) { if (stuff) throw onError }
I would choose A every time.
Now, if I had to choose between A and what you have now, that's a toss up. It really depends on context, what you're trying to accomplish with the helpers, how else they may be used, etc.
On a final note, naming is very important in these sorts of situations. If your go the return-code-helper route, your helpers should have question names, such as isValid. If you have exception-throwing-helpers, they should have action names, such as validate. Maybe even give it emphasis, like validate_!.
For an alternative approach you could check out scalaz Validators, which give a lot of flexibility for this kind of case (e.g. should I crash on error, accumulate the errors and report at the end or ignore them completely?). A few
examples might help you decide if this is the right approach for you.
If you find it hard to find a way in to the library, this answer gives some pointers to some introductory material; or check out .