Rxjava User-Retry observable with .cache operator? - reactive-programming

i've an observable that I create with the following code.
Observable.create(new Observable.OnSubscribe<ReturnType>() {
#Override
public void call(Subscriber<? super ReturnType> subscriber) {
try {
if (!subscriber.isUnsubscribed()) {
subscriber.onNext(performRequest());
}
subscriber.onCompleted();
} catch (Exception e) {
subscriber.onError(e);
}
}
});
performRequest() will perform a long running task as you might expect.
Now, since i might be launching the same Observable twice or more in a very short amount of time, I decided to write such transformer:
protected Observable.Transformer<ReturnType, ReturnType> attachToRunningTaskIfAvailable() {
return origObservable -> {
synchronized (mapOfRunningTasks) {
// If not in maps
if ( ! mapOfRunningTasks.containsKey(getCacheKey()) ) {
Timber.d("Cache miss for %s", getCacheKey());
mapOfRunningTasks.put(
getCacheKey(),
origObservable
.doOnTerminate(() -> {
Timber.d("Removed from tasks %s", getCacheKey());
synchronized (mapOfRunningTasks) {
mapOfRunningTasks.remove(getCacheKey());
}
})
.cache()
);
} else {
Timber.d("Cache Hit for %s", getCacheKey());
}
return mapOfRunningTasks.get(getCacheKey());
}
};
}
Which basically puts the original .cache observable in a HashMap<String, Observable>.
This basically disallows multiple requests with the same getCacheKey() (Example login) to call performRequest() in parallel. Instead, if a second login request arrives while another is in progress, the second request observable gets "discarded" and the already-running will be used instead. => All the calls to onNext are going to be cached and sent to both subscribers actually hitting my backend only once.
Now, suppouse this code:
// Observable loginTask
public void doLogin(Observable<UserInfo> loginTask) {
loginTask.subscribe(
(userInfo) -> {},
(throwable) -> {
if (userWantsToRetry()) {
doLogin(loinTask);
}
}
);
}
Where loginTask was composed with the previous transformer. Well, when an error occurs (might be connectivity) and the userWantsToRetry() then i'll basically re-call the method with the same observable. Unfortunately that has been cached and I'll receive the same error without hitting performRequest() again since the sequence gets replayed.
Is there a way I could have both the "same requests grouping" behavior that the transformer provides me AND the retry button?

Your question has a lot going on and it's hard to put it into direct terms. I can make a couple recommendations though. Firstly your Observable.create can be simplified by using an Observable.defer(Func0<Observable<T>>). This will run the func every time a new subscriber is subscribed and catch and channel any exceptions to the subscriber's onError.
Observable.defer(() -> {
return Observable.just(performRequest());
});
Next, you can use observable.repeatWhen(Func1<Observable<Void>, Observable<?>>) to decide when you want to retry. Repeat operators will re-subscribe to the observable after an onComplete event. This particular overload will send an event to a subject when an onComplete event is received. The function you provide will receive this subject. Your function should call something like takeWhile(predicate) and onComplete when you do not want to retry again.
Observable.just(1,2,3).flatMap((Integer num) -> {
final AtomicInteger tryCount = new AtomicInteger(0);
return Observable.just(num)
.repeatWhen((Observable<? extends Void> notifications) ->
notifications.takeWhile((x) -> num == 2 && tryCount.incrementAndGet() != 3));
})
.subscribe(System.out::println);
Output:
1
2
2
2
3
The above example shows that retries are aloud when the event is not 2 and up to a max of 22 retries. If you switch to a repeatWhen then the flatMap would contain your decision as to use a cached observable or the realWork observable. Hope this helps!

Related

What would cause SingleEmitter.onSuccess() to generate a NoSuchElement exception?

I have a Single flow organized like this:
getSomething() // returns Single<>
.flatMap(something -> {
// various things
return Single.defer( () -> {
// various other things
return Single.<SomeType>create(emitter -> {
// some more stuff
someCallbackApi(result -> {
if (result.isError()) {
emitter.onError( result.getCause() );
} else {
// guaranteed non-null data
emitter.onSuccess( result.getData() ); // this generates NoSuchElement
}
});
});
})
.retryWhen( ... )
.flatMap( data -> handle(data) )
.retryWhen( ... );
})
.retryWhen( ... )
.onErrorResumeNext(error -> process(error))
.subscribe(data -> handleSuccess(data), error -> handleError(error));
In test cases, the callback api Single successfully retries a number of times (determined by the test case), and every time on the last retry, the call to emitter.onSuccess() generates the exception below. What is going on? I haven't been able to restructure or change the downstream operators or subscribers to avoid the problem.
java.util.NoSuchElementException: null
at io.reactivex.internal.operators.flowable.FlowableSingleSingle$SingleElementSubscriber.onComplete(FlowableSingleSingle.java:116)
at io.reactivex.subscribers.SerializedSubscriber.onComplete(SerializedSubscriber.java:168)
at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenReceiver.onComplete(FlowableRepeatWhen.java:118)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:426)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onComplete(FlowableFlatMap.java:338)
at io.reactivex.internal.operators.flowable.FlowableZip$ZipCoordinator.drain(FlowableZip.java:210)
at io.reactivex.internal.operators.flowable.FlowableZip$ZipSubscriber.onNext(FlowableZip.java:381)
at io.reactivex.processors.UnicastProcessor.drainFused(UnicastProcessor.java:363)
at io.reactivex.processors.UnicastProcessor.drain(UnicastProcessor.java:396)
at io.reactivex.processors.UnicastProcessor.onNext(UnicastProcessor.java:458)
at io.reactivex.processors.SerializedProcessor.onNext(SerializedProcessor.java:103)
at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenSourceSubscriber.again(FlowableRepeatWhen.java:171)
at io.reactivex.internal.operators.flowable.FlowableRetryWhen$RetryWhenSubscriber.onError(FlowableRetryWhen.java:76)
at io.reactivex.internal.operators.single.SingleToFlowable$SingleToFlowableObserver.onError(SingleToFlowable.java:67)
at io.reactivex.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:116)
at io.reactivex.internal.operators.flowable.FlowableSingleSingle$SingleElementSubscriber.onError(FlowableSingleSingle.java:97)
at io.reactivex.subscribers.SerializedSubscriber.onError(SerializedSubscriber.java:142)
at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenReceiver.onError(FlowableRepeatWhen.java:112)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.checkTerminate(FlowableFlatMap.java:567)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:374)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.innerError(FlowableFlatMap.java:606)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onError(FlowableFlatMap.java:672)
at io.reactivex.internal.subscriptions.EmptySubscription.error(EmptySubscription.java:55)
at io.reactivex.internal.operators.flowable.FlowableError.subscribeActual(FlowableError.java:40)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14865)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:163)
at io.reactivex.internal.operators.flowable.FlowableZip$ZipCoordinator.drain(FlowableZip.java:249)
at io.reactivex.internal.operators.flowable.FlowableZip$ZipSubscriber.onNext(FlowableZip.java:381)
at io.reactivex.processors.UnicastProcessor.drainFused(UnicastProcessor.java:363)
at io.reactivex.processors.UnicastProcessor.drain(UnicastProcessor.java:396)
at io.reactivex.processors.UnicastProcessor.onNext(UnicastProcessor.java:458)
at io.reactivex.processors.SerializedProcessor.onNext(SerializedProcessor.java:103)
at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenSourceSubscriber.again(FlowableRepeatWhen.java:171)
at io.reactivex.internal.operators.flowable.FlowableRetryWhen$RetryWhenSubscriber.onError(FlowableRetryWhen.java:76)
at io.reactivex.internal.operators.single.SingleToFlowable$SingleToFlowableObserver.onError(SingleToFlowable.java:67)
at io.reactivex.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:116)
at io.reactivex.internal.disposables.EmptyDisposable.error(EmptyDisposable.java:78)
at io.reactivex.internal.operators.single.SingleError.subscribeActual(SingleError.java:42)
at io.reactivex.Single.subscribe(Single.java:3603)
at io.reactivex.internal.operators.single.SingleFlatMap$SingleFlatMapCallback.onSuccess(SingleFlatMap.java:84)
at io.reactivex.internal.operators.flowable.FlowableSingleSingle$SingleElementSubscriber.onComplete(FlowableSingleSingle.java:114)
at io.reactivex.subscribers.SerializedSubscriber.onComplete(SerializedSubscriber.java:168)
at io.reactivex.internal.operators.flowable.FlowableRetryWhen$RetryWhenSubscriber.onComplete(FlowableRetryWhen.java:82)
at io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:134)
at io.reactivex.internal.operators.single.SingleToFlowable$SingleToFlowableObserver.onSuccess(SingleToFlowable.java:62)
at io.reactivex.internal.operators.single.SingleCreate$Emitter.onSuccess(SingleCreate.java:67)
Solved:
Many thanks to #dano for pointing out the retryWhen behavior when used with Single. In this case, the outermost retryWhen operator had a bad terminating condition, roughly like:
.retryWhen(errors -> errors.zipWith( Flowable.range(1, maxRetries), ...)
.flatMap( zipped -> {
if (zipped.retryCount() <= maxRetries) {
return Flowable.just(0L);
}
return Flowable.error( new Exception() );
})
...Flowable.range() will complete when it has generated the last number, which will cause the Single to emit NoSuchElement. Just bumping the count argument to Flowable.range() by one is enough to fix the problem:
.retryWhen(errors -> errors.zipWith( Flowable.range(1, maxRetries + 1), ...)
.flatMap( zipped -> {
if (zipped.retryCount() <= maxRetries) {
return Flowable.just(0L);
}
return Flowable.error( new Exception() );
})
This is happening because of the way you implemented the callback you passed to retryWhen. The retryWhen docuementation states (emphasis mine):
Re-subscribes to the current Single if and when the Publisher returned
by the handler function signals a value.
If the Publisher signals an onComplete, the resulting Single will
signal a NoSuchElementException.
One of the Flowable instances you're returning inside of the calls to retryWhen is emitting onComplete, which leads to the NoSuchElementException.
Here's a very simple example that produces the same error:
Single.error(new Exception("hey"))
.retryWhen(e -> Flowable.just(1))
.subscribe(System.out::println, e -> e.printStackTrace());
The stacktrace this produces starts with this, same as yours:
java.util.NoSuchElementException
at io.reactivex.internal.operators.flowable.FlowableSingleSingle$SingleElementSubscriber.onComplete(FlowableSingleSingle.java:116)
at io.reactivex.subscribers.SerializedSubscriber.onComplete(SerializedSubscriber.java:168)
at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenReceiver.onComplete(FlowableRepeatWhen.java:118)
You don't include any of your code from inside the retryWhen calls, so I can't say exactly what you did wrong, but generally you want to chain whatever you do to the Flowable that is passed in. So my example above would look like this, if we really wanted to retry forever:
Single.error(new Exception("hey"))
.retryWhen(e -> e.flatMap(ign -> Flowable.just(1)))
.subscribe(System.out::println, e -> e.printStackTrace());

Vert.x: How to wait for a future to complete

Is there a way to wait for a future to complete without blocking the event loop?
An example of a use case with querying Mongo:
Future<Result> dbFut = Future.future();
mongo.findOne("myusers", myQuery, new JsonObject(), res -> {
if(res.succeeded()) {
...
dbFut.complete(res.result());
}
else {
...
dbFut.fail(res.cause());
}
}
});
// Here I need the result of the DB query
if(dbFut.succeeded()) {
doSomethingWith(dbFut.result());
}
else {
error();
}
I know the doSomethingWith(dbFut.result()); can be moved to the handler, yet if it's long, the code will get unreadable (Callback hell ?) It that the right solution ? Is that the omny solution without additional libraries ?
I'm aware that rxJava simplifies the code, but as I don't know it, learning Vert.x and rxJava is just too much.
I also wanted to give a try to vertx-sync. I put the dependency in the pom.xml; everything got downloaded fine but when I started my app, I got the following error
maurice#mickey> java \
-javaagent:~/.m2/repository/co/paralleluniverse/quasar-core/0.7.5/quasar-core-0.7.5-jdk8.jar \
-jar target/app-dev-0.1-fat.jar \
-conf conf/config.json
Error opening zip file or JAR manifest missing : ~/.m2/repository/co/paralleluniverse/quasar-core/0.7.5/quasar-core-0.7.5-jdk8.jar
Error occurred during initialization of VM
agent library failed to init: instrument
I know what the error means in general, but I don't know in that context... I tried to google for it but didn't find any clear explanation about which manifest to put where. And as previously, unless mandatory, I prefer to learn one thing at a time.
So, back to the question : is there a way with "basic" Vert.x to wait for a future without perturbation on the event loop ?
You can set a handler for the future to be executed upon completion or failure:
Future<Result> dbFut = Future.future();
mongo.findOne("myusers", myQuery, new JsonObject(), res -> {
if(res.succeeded()) {
...
dbFut.complete(res.result());
}
else {
...
dbFut.fail(res.cause());
}
}
});
dbFut.setHandler(asyncResult -> {
if(asyncResult.succeeded()) {
// your logic here
}
});
This is a pure Vert.x way that doesn't block the event loop
I agree that you should not block in the Vertx processing pipeline, but I make one exception to that rule: Start-up. By design, I want to block while my HTTP server is initialising.
This code might help you:
/**
* #return null when waiting on {#code Future<Void>}
*/
#Nullable
public static <T>
T awaitComplete(Future<T> f)
throws Throwable
{
final Object lock = new Object();
final AtomicReference<AsyncResult<T>> resultRef = new AtomicReference<>(null);
synchronized (lock)
{
// We *must* be locked before registering a callback.
// If result is ready, the callback is called immediately!
f.onComplete(
(AsyncResult<T> result) ->
{
resultRef.set(result);
synchronized (lock) {
lock.notify();
}
});
do {
// Nested sync on lock is fine. If we get a spurious wake-up before resultRef is set, we need to
// reacquire the lock, then wait again.
// Ref: https://stackoverflow.com/a/249907/257299
synchronized (lock)
{
// #Blocking
lock.wait();
}
}
while (null == resultRef.get());
}
final AsyncResult<T> result = resultRef.get();
#Nullable
final Throwable t = result.cause();
if (null != t) {
throw t;
}
#Nullable
final T x = result.result();
return x;
}

Repeat Single based on onSuccess() value

I want to repeat a Single based on the single value emitted in onSuccess(). Here is a working example
import org.reactivestreams.Publisher;
import io.reactivex.Flowable;
import io.reactivex.Single;
import io.reactivex.functions.Function;
public class Temp {
void main() {
Job job = new Job();
Single.just(job)
.map(this::processJob)
.repeatWhen(new Function<Flowable<Object>, Publisher<?>>() {
#Override
public Publisher<?> apply(Flowable<Object> objectFlowable) throws Exception {
// TODO repeat when Single emits false
return null;
}
})
.subscribe();
}
/**
* returns true if process succeeded, false if failed
*/
boolean processJob(Job job) {
return true;
}
class Job {
}
}
I understand how repeatWhen works for Observables by relying on the "complete" notification. However since Single doesn't receive that notification I'm not sure what the Flowable<Object> is really giving me. Also why do I need to return a Publisher from this function?
Instead of relying on a boolean value, you could make your job throw an exception when it fails:
class Job {
var isSuccess: Boolean = false
}
fun processJob(job: Job): String {
if (job.isSuccess) {
return "job succeeds"
} else {
throw Exception("job failed")
}
}
val job = Job()
Single.just(job)
.map { processJob(it) }
.retry() // will resubscribe until your job succeeds
.subscribe(
{ value -> print(value) },
{ error -> print(error) }
)
i saw a small discrepancy in the latest docs and your code, so i did a little digging...
(side note - i think the semantics of retryWhen seem like the more appropriate operator for your case, so i've substituted it in for your usage of repeatWhen. but i think the root of your problem remains the same in either case).
the signature for retryWhen is:
retryWhen(Function<? super Flowable<Throwable>,? extends Publisher<?>> handler)
that parameter is a factory function whose input is a source that emits anytime onError is called upstream, giving you the ability to insert custom retry logic that may be influenced through interrogation of the underlying Throwable. this begins to answer your first question of "I'm not sure what the Flowable<Object> is really giving me" - it shouldn't be Flowable<Object> to begin with, it should be Flowable<Throwable> (for the reason i just described).
so where did Flowable<Object> come from? i managed to reproduce IntelliJ's generation of this code through it's auto-complete feature using RxJava version 2.1.17. upgrading to 2.2.0, however, produces the correct result of Flowable<Throwable>. so, see if upgrading to the latest version generates the correct result for you as well.
as for your second question of "Also why do I need to return a Publisher from this function?" - this is used to determine if re-subscription should happen. if the factory function returns a Publisher that emits a terminal state (ie calls onError() or onComplete()) re-subscription will not happen. however, if onNext() is called, it will. (this also explains why the Publisher isn't typed - the type doesn't matter. the only thing that does matter is what kind of notification it publishes).
another way to rewrite this, incorporating the above, might be as follows:
// just some type to use as a signal to retry
private class SpecialException extends RuntimeException {}
// job processing results in a Completable that either completes or
// doesn't (by way of an exception)
private Completable rxProcessJob(Job job) {
return Completable.complete();
// return Completable.error(new SpecialException());
}
...
rxProcessJob(new Job())
.retryWhen(errors -> {
return errors.flatMap(throwable -> {
if(throwable instanceof SpecialException) {
return PublishProcessor.just(1);
}
return PublishProcessor.error(throwable);
});
})
.subscribe(
() -> {
System.out.println("## onComplete()");
},
error -> {
System.out.println("## onError(" + error.getMessage() + ")");
}
);
i hope that helps!
The accepted answer would work, but is hackish. You don't need to throw an error; simply filter the output of processJob which converts the Single to a Maybe, and then use the repeatWhen handler to decide how many times, or with what delay, you may want to resubscribe. See Kotlin code below from a working example, you should be able to easily translate this to Java.
filter { it }
.repeatWhen { handler ->
handler.zipWith(1..3) { _, i -> i }
.flatMap { retryCount -> Flowable.timer(retryDelay.toDouble().pow(retryCount).toLong(), TimeUnit.SECONDS) }
.doOnNext { log.warn("Retrying...") }
}

RxJava: Merge multiple singles and complete after some have failed

I would like to merge two Single<MyData> such that if one of them fails but the other one succeeds then the error of the one that failed and the emission from the other one are reported, and then the resulting Single<MyData> (or Observable<MyData>) completes.
If both Single<MyData> fail then the result should also fail and also be marked as failed.
What I would like to have at the end is:
If both succeed then the emitted values and a producer marked as completed.
If one succeeds and the other fails, the emitted value, the thrown error and the producer marked as complete.
If all fail, the errors and the producer marked as failed.
It's like an 'OR' operation
This is not possible. There is only a single terminal event allowed. The contract for Single is success|error. If you need to receive a next event as well, you should consider to use Observable instead. The contract for Observable is next* complete|error, but you'll still not get a complete.
Observable.mergeDelayError(single1.toObservable(), single2.toObservable())
This can be accomplished with Single.create(SingleOnSubscribe). If your return type is Single<MyData> only one of the responses can be returned, but you could also modify this to instead return a Single<List<MyData>> or some other RxJava structure like Flowable<MyData> that supports multiple returns. In this example, the Single<MyData> returns whichever call returns last because that was the simplest to implement.
public Single<MyData> getCombinedSingle(List<Single<MyData>> singles) {
return Single.create(new SingleOnSubscribe<MyData> {
private boolean encounteredError = false;
private MyData myData;
#Override
public void subscribe(#NonNull Emitter<MyData> emitter) {
List<Disposable> disposables = new ArrayList<>();
Consumer<MyData> myDataConsumer = myData -> {
this.MyData = myData;
checkForFinish(emitter, disposables);
}
Consumer<Throwable> throwableConsumer = throwable -> {
throwable.printStackTrace();
encounteredError = true;
checkForFinish(emitter, disposables);
}
for (Single single: singles) {
disposables.put(single.subscribe(myDataConsumer, throwableConsumer);
}
}
private void checkForFinish(SingleEmitter<MyData> emitter, List<Disposable> disposables) {
if (disposables1.stream().allMatch(Disposable::isDisposed)) {
if (encounteredError) {
emitter.onError(new Throwable());
} else {
emitter.onSuccess(myData);
}
}
}
}
}
This could be modified to return a Throwable from the original Singles if needed.

Reactive extensions(Rx) Switch() produces new observable which is not subscribed to provided OnCompleted()

I have a problem with my Rx subscription using Switch statement.
_performSearchSubject
.AsObservable()
.Select(_ => PerformQuery())
.Switch()
.ObserveOn(_synchronizationContextService.SynchronizationContext)
.Subscribe(DataArrivedForPositions, PositionQueryError, PositionQueryCompleted)
.DisposeWith(this);
The flow is:
Some properties change and the performSearchSubject.OnNext is called
The PerformPositionQuery() is called, which returns a observer each time it is hit
The service which responds through this observer calls OnNext twice and OnCompleted once when the data receive is done
Method DataArrivedForPositions is called twice as expected
Method PositionQueryCompleted is never called, though observer.OnCompleted() is called inside my data service.
Code for dataService is:
protected override void Request(Request request, IObserver<Response> observer)
{
query.Arrive += p => QueryReceive(request.RequestId, p, observer, query);
query.Error += (type, s, message) => QueryError(observer, message);
query.NoMoreData += id => QueryCompleted(observer);
query.Execute(request);
}
private void QueryError(IObserver<PositionSheetResponse> observer, string message)
{
observer.OnError(new Exception(message));
}
private void QueryCompleted(IObserver<PositionSheetResponse> observer)
{
observer.OnCompleted();
}
private void QueryReceive(Guid requestId, Qry0079Receive receiveData, IObserver<PositionSheetResponse> observer, IQry0079PositionSheet query)
{
observer.OnNext(ConvertToResponse(requestId, receiveData));
}
Switch result will only Complete when your outer observable (_performSearchSubject) completes. I assume in your case this one never does (it's probably bound to a user action performing the search).
What's unclear is when you expect PositionQueryCompleted to be called. If It's after each and every successful query is processed, then your stream needs to be modified, because Switch lost you the information that the query stream completed, but it also lacks information about the UI (wrong scheduler even) to say whether its data was actually processed.
There may be other ways to achieve it, but basically you want your query stream complete to survive through Switch (which currently ignore this event). For instance you can transform your query stream to have n+1 events, with one extra for the complete:
_performSearchSubject
.AsObservable()
.Select(_ =>
PerformQuery()
.Select(Data => new { Data, Complete = false})
.Concat(Observable.Return(new { Data = (string)null, Complete = true })))
You can safely apply .Switch().ObserveOn(_synchronizationContextService.SynchronizationContext) on it, but then you need to modify your subscription:
.Subscribe(data => {
if (data.Complete) DataArrivedForPositions(data.Data);
else PositionQueryCompleted()
}, PositionQueryError)