I'm trying to figure out how to use Observable.groupBy to limit the number of elements pushed by key over a time frame. I end up with the following construct:
create(emitter -> {
while (true) {
publishedMeter.mark();
emitter.onNext(new Object());
}
})
.window(1000L, TimeUnit.MILLISECONDS)
.flatMap(window -> window.groupBy(o -> o.hashCode() % 10_000).flatMapMaybe(Observable::lastElement))
.subscribe(e -> receivedMeter.mark());
While subscribe's onNext callback is called a few thousand times, which I think should mean that flatMapMaybe does properly subscribe to all GroupedObservableSource. After a short while one of the thread inside RxComputationThreadPool but I don't understand what I'm missing
Related
I have a Single flow organized like this:
getSomething() // returns Single<>
.flatMap(something -> {
// various things
return Single.defer( () -> {
// various other things
return Single.<SomeType>create(emitter -> {
// some more stuff
someCallbackApi(result -> {
if (result.isError()) {
emitter.onError( result.getCause() );
} else {
// guaranteed non-null data
emitter.onSuccess( result.getData() ); // this generates NoSuchElement
}
});
});
})
.retryWhen( ... )
.flatMap( data -> handle(data) )
.retryWhen( ... );
})
.retryWhen( ... )
.onErrorResumeNext(error -> process(error))
.subscribe(data -> handleSuccess(data), error -> handleError(error));
In test cases, the callback api Single successfully retries a number of times (determined by the test case), and every time on the last retry, the call to emitter.onSuccess() generates the exception below. What is going on? I haven't been able to restructure or change the downstream operators or subscribers to avoid the problem.
java.util.NoSuchElementException: null
at io.reactivex.internal.operators.flowable.FlowableSingleSingle$SingleElementSubscriber.onComplete(FlowableSingleSingle.java:116)
at io.reactivex.subscribers.SerializedSubscriber.onComplete(SerializedSubscriber.java:168)
at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenReceiver.onComplete(FlowableRepeatWhen.java:118)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:426)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onComplete(FlowableFlatMap.java:338)
at io.reactivex.internal.operators.flowable.FlowableZip$ZipCoordinator.drain(FlowableZip.java:210)
at io.reactivex.internal.operators.flowable.FlowableZip$ZipSubscriber.onNext(FlowableZip.java:381)
at io.reactivex.processors.UnicastProcessor.drainFused(UnicastProcessor.java:363)
at io.reactivex.processors.UnicastProcessor.drain(UnicastProcessor.java:396)
at io.reactivex.processors.UnicastProcessor.onNext(UnicastProcessor.java:458)
at io.reactivex.processors.SerializedProcessor.onNext(SerializedProcessor.java:103)
at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenSourceSubscriber.again(FlowableRepeatWhen.java:171)
at io.reactivex.internal.operators.flowable.FlowableRetryWhen$RetryWhenSubscriber.onError(FlowableRetryWhen.java:76)
at io.reactivex.internal.operators.single.SingleToFlowable$SingleToFlowableObserver.onError(SingleToFlowable.java:67)
at io.reactivex.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:116)
at io.reactivex.internal.operators.flowable.FlowableSingleSingle$SingleElementSubscriber.onError(FlowableSingleSingle.java:97)
at io.reactivex.subscribers.SerializedSubscriber.onError(SerializedSubscriber.java:142)
at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenReceiver.onError(FlowableRepeatWhen.java:112)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.checkTerminate(FlowableFlatMap.java:567)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:374)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.innerError(FlowableFlatMap.java:606)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onError(FlowableFlatMap.java:672)
at io.reactivex.internal.subscriptions.EmptySubscription.error(EmptySubscription.java:55)
at io.reactivex.internal.operators.flowable.FlowableError.subscribeActual(FlowableError.java:40)
at io.reactivex.Flowable.subscribe(Flowable.java:14918)
at io.reactivex.Flowable.subscribe(Flowable.java:14865)
at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:163)
at io.reactivex.internal.operators.flowable.FlowableZip$ZipCoordinator.drain(FlowableZip.java:249)
at io.reactivex.internal.operators.flowable.FlowableZip$ZipSubscriber.onNext(FlowableZip.java:381)
at io.reactivex.processors.UnicastProcessor.drainFused(UnicastProcessor.java:363)
at io.reactivex.processors.UnicastProcessor.drain(UnicastProcessor.java:396)
at io.reactivex.processors.UnicastProcessor.onNext(UnicastProcessor.java:458)
at io.reactivex.processors.SerializedProcessor.onNext(SerializedProcessor.java:103)
at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenSourceSubscriber.again(FlowableRepeatWhen.java:171)
at io.reactivex.internal.operators.flowable.FlowableRetryWhen$RetryWhenSubscriber.onError(FlowableRetryWhen.java:76)
at io.reactivex.internal.operators.single.SingleToFlowable$SingleToFlowableObserver.onError(SingleToFlowable.java:67)
at io.reactivex.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:116)
at io.reactivex.internal.disposables.EmptyDisposable.error(EmptyDisposable.java:78)
at io.reactivex.internal.operators.single.SingleError.subscribeActual(SingleError.java:42)
at io.reactivex.Single.subscribe(Single.java:3603)
at io.reactivex.internal.operators.single.SingleFlatMap$SingleFlatMapCallback.onSuccess(SingleFlatMap.java:84)
at io.reactivex.internal.operators.flowable.FlowableSingleSingle$SingleElementSubscriber.onComplete(FlowableSingleSingle.java:114)
at io.reactivex.subscribers.SerializedSubscriber.onComplete(SerializedSubscriber.java:168)
at io.reactivex.internal.operators.flowable.FlowableRetryWhen$RetryWhenSubscriber.onComplete(FlowableRetryWhen.java:82)
at io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:134)
at io.reactivex.internal.operators.single.SingleToFlowable$SingleToFlowableObserver.onSuccess(SingleToFlowable.java:62)
at io.reactivex.internal.operators.single.SingleCreate$Emitter.onSuccess(SingleCreate.java:67)
Solved:
Many thanks to #dano for pointing out the retryWhen behavior when used with Single. In this case, the outermost retryWhen operator had a bad terminating condition, roughly like:
.retryWhen(errors -> errors.zipWith( Flowable.range(1, maxRetries), ...)
.flatMap( zipped -> {
if (zipped.retryCount() <= maxRetries) {
return Flowable.just(0L);
}
return Flowable.error( new Exception() );
})
...Flowable.range() will complete when it has generated the last number, which will cause the Single to emit NoSuchElement. Just bumping the count argument to Flowable.range() by one is enough to fix the problem:
.retryWhen(errors -> errors.zipWith( Flowable.range(1, maxRetries + 1), ...)
.flatMap( zipped -> {
if (zipped.retryCount() <= maxRetries) {
return Flowable.just(0L);
}
return Flowable.error( new Exception() );
})
This is happening because of the way you implemented the callback you passed to retryWhen. The retryWhen docuementation states (emphasis mine):
Re-subscribes to the current Single if and when the Publisher returned
by the handler function signals a value.
If the Publisher signals an onComplete, the resulting Single will
signal a NoSuchElementException.
One of the Flowable instances you're returning inside of the calls to retryWhen is emitting onComplete, which leads to the NoSuchElementException.
Here's a very simple example that produces the same error:
Single.error(new Exception("hey"))
.retryWhen(e -> Flowable.just(1))
.subscribe(System.out::println, e -> e.printStackTrace());
The stacktrace this produces starts with this, same as yours:
java.util.NoSuchElementException
at io.reactivex.internal.operators.flowable.FlowableSingleSingle$SingleElementSubscriber.onComplete(FlowableSingleSingle.java:116)
at io.reactivex.subscribers.SerializedSubscriber.onComplete(SerializedSubscriber.java:168)
at io.reactivex.internal.operators.flowable.FlowableRepeatWhen$WhenReceiver.onComplete(FlowableRepeatWhen.java:118)
You don't include any of your code from inside the retryWhen calls, so I can't say exactly what you did wrong, but generally you want to chain whatever you do to the Flowable that is passed in. So my example above would look like this, if we really wanted to retry forever:
Single.error(new Exception("hey"))
.retryWhen(e -> e.flatMap(ign -> Flowable.just(1)))
.subscribe(System.out::println, e -> e.printStackTrace());
Using RxJava 2.2.8:
Observable.fromCallable(() -> "Some data")
.subscribe(
s -> System.out.println(s),
e -> System.err.println(e),
() -> System.out.println("Completed")
);
Output
Some data
Completed
My question is why onComplete never gets called for the following?
Observable.interval(1, TimeUnit.SECONDS)
.switchMap(t -> Observable.fromCallable(() -> "Some data"))
.subscribe(
s -> System.out.println(s),
e -> System.err.println(e),
() -> System.out.println("Completed")
);
Output
Some data
Some data
Some data
...
I understand Observable.interval will create a never ending stream, so no onComplete. My understanding of switchMap is that it returns an observable which fires events produced by the inner observable (cancelling any pending and flattening), in this case Observable.fromCallable.
Now, this 'inner' observable does have a definite end (unlike the outer observable), so why doesn't onComplete gets called on this inner Observable?
Why isn't the output like this?
Some data
Completed
Some data
Completed
Some data
Completed
...
From documentation:
The resulting ObservableSource completes if both the upstream
ObservableSource and the last inner ObservableSource
Since upstream ObservableSource is an infinite stream, the resulting Observable will not complete.
Also note that according to the observable contract, onComplete indicates the observable has terminated and it will not emit any further items in the future, so you will never see "Completed" followed by some other items regardless of your implementation.
Snippet1 , I can see the sysout from both subscribers.
Snippet2 , I dont see output from the second observable.
Why is the merge not working for me?
Snippet1
x = createQ2Flowable().subscribeOn(Schedulers.computation())
.observeOn(Schedulers.io())
.filter(predicate -> !predicate.toString().contains("<log realm=\"\""))
.subscribe(onNext -> System.out.println("Q2->" + onNext));
y = createMetricsFlowable().subscribeOn(Schedulers.computation())
.observeOn(Schedulers.io())
.subscribe(onNext -> System.out.println("metrics->" + onNext));
Snippet2
createQ2Flowable().mergeWith(createMetricsFlowable())
.subscribeOn(Schedulers.computation())
.subscribe(onNext -> System.out.println(onNext));
[edit]: Added flowable creators
private Flowable<String> createMetricsFlowable() {
return Flowable.create(source -> {
Space sp = SpaceFactory.getSpace("rxObservableFeeder");
while (running()) {
String line = (String) sp.in("RXTmFeeder");
source.onNext(line);
}
}, BackpressureStrategy.BUFFER);
}
private Flowable<String> createQ2Flowable() {
return Flowable.create(source -> {
Space sp = SpaceFactory.getSpace("LoggerSpace");
while (running()) {
LogEvent line = (LogEvent) sp.in("rxLoggingKey");
source.onNext(line.toString());
}
}, BackpressureStrategy.BUFFER);
}
From the comments:
try
createQ2Flowable()
.subscribeOn(Schedulers.computation()) // <-------------------------
.mergeWith(createMetricsFlowable()
.subscribeOn(Schedulers.computation()) // <-------------------------
)
Now I need to know why it happened
Given the detailed implementation, you have two synchronous Flowables. When you merge them, the first Flowable is subscribed to and starts emitting immediately and never giving back the control to mergeWith, therefore the second Flowable is never subscribed to.
The subscribeOn after mergeWith is not equivalent to the solution provided above. You have to explicitly have both Flowables subscribed on a background thread so mergeWith can subscribe to the second Flowable after now that the synchronous looping has been moved off from the thread the mergeWith uses for subscribing to its sources.
The question is about RxJava2.
Noticed that zipping Throwable that comes from retryWhen with range emits all items from Observable.range before zipper function has been applied. Also, range emits sequence even if zipWith wasn't called. For example this source code
Observable.create<String> {
println("subscribing")
it.onError(RuntimeException("always fails"))
}
.retryWhen {
it.zipWith(Observable.range(1, 3).doOnNext { println("range $it") },
BiFunction { t: Throwable, i: Int -> i })
.flatMap {
System.out.println("delay retry by $it + second(s)")
Observable.timer(it.toLong(), TimeUnit.SECONDS)
}
}./*subscribe*/
gives the following result
range 1
range 2
range 3
subscribing
delay retry by 1 + second(s)
subscribing
delay retry by 2 + second(s)
subscribing
delay retry by 3 + second(s)
subscribing
onComplete
Replacing onError in observable creation also don't eliminate emitting range items. So the question is why it's happening as Range is cold.
Observables in 2.x don't have backpressure thus a range operator will emit all its items as soon as it can. Your case, however, can use a normal counter incremented along the error notification of the retry handler:
source.retryWhen(e -> {
int[] counter = { 0 };
return e.takeWhile(v -> ++counter[0] < 4)
.flatMap(v -> Observable.timer(counter[0], TimeUnit.SECONDS));
})
Using reactive extension, it is easy to subscribe 2 times to the same observable.
When a new value is available in the observable, both subscribers are called with this same value.
Is there a way to have each subscriber get a different value (the next one) from this observable ?
Ex of what i'm after:
source sequence: [1,2,3,4,5,...] (infinite)
The source is constantly adding new items at an unknown rate.
I'm trying to execute a lenghty async action for each item using N subscribers.
1st subscriber: 1,2,4,...
2nd subscriber: 3,5,...
...
or
1st subscriber: 1,3,...
2nd subscriber: 2,4,5,...
...
or
1st subscriber: 1,3,5,...
2nd subscriber: 2,4,6,...
I would agree with Asti.
You could use Rx to populate a Queue (Blocking Collection) and then have competing consumers read from the queue. This way if one process was for some reason faster it could pick up the next item potentially before the other consumer if it was still busy.
However, if you want to do it, against good advice :), then you could just use the Select operator that will provide you with the index of each element. You can then pass that down to your subscribers and they can fiter on a modulus. (Yuck! Leaky abstractions, magic numbers, potentially blocking, potentiall side effects to the source sequence etc)
var source = Obserservable.Interval(1.Seconds())
.Select((i,element)=>{new Index=i, Element=element});
var subscription1 = source.Where(x=>x.Index%2==0).Subscribe(x=>DoWithThing1(x.Element));
var subscription2 = source.Where(x=>x.Index%2==1).Subscribe(x=>DoWithThing2(x.Element));
Also remember that the work done on the OnNext handler if it is blocking will still block the scheduler that it is on. This could affect the speed of your source/producer. Another reason why Asti's answer is a better option.
Ask if that is not clear :-)
How about:
IObservable<TRet> SomeLengthyOperation(T input)
{
return Observable.Defer(() => Observable.Start(() => {
return someCalculatedValueThatTookALongTime;
}, Scheduler.TaskPoolScheduler));
}
someObservableSource
.SelectMany(x => SomeLengthyOperation(input))
.Subscribe(x => Console.WriteLine("The result was {0}", x);
You can even limit the number of concurrent operations:
someObservableSource
.Select(x => SomeLengthyOperation(input))
.Merge(4 /* at a time */)
.Subscribe(x => Console.WriteLine("The result was {0}", x);
It's important for the Merge(4) to work, that the Observable returned by SomeLengthyOperation be a Cold Observable, which is what the Defer does here - it makes the Observable.Start not happen until someone Subscribes.