How to ensure reactor flux process all the messages supplied - reactive-programming

Question

Suppose we want to have Flux pipeline to process all messages supplied from several threads. Let's consider the code below:
#Test
public void testFluxCreate() throws InterruptedException {
EmitterProcessor<String> processor = EmitterProcessor.create();
CountDownLatch latch = new CountDownLatch(1);
AtomicLong counter = new AtomicLong();
AtomicLong batch = new AtomicLong();
Flux<List<String>> flux = processor
.doOnSubscribe(ss -> System.out.println(nm() + " : subscribing to + ss))
.onBackpressureError()
.buffer(7)
.publishOn(Schedulers.immediate())
.doOnNext(it -> {
counter.addAndGet(it.size());
System.out.println(batch.incrementAndGet() + " : " + nm() + "Batch: " + it.size());
})
;
CompletableFuture<Void> producer = CompletableFuture.runAsync(() -> {
IntStream.range(1, 1001).forEach(it -> {
//sleep();
processor.onNext("Message-" + it);
});
});
CompletableFuture<Void> producer2 = CompletableFuture.runAsync(() -> {
IntStream.range(1, 1001).forEach(it -> {
//sleep();
processor.onNext("Message2-" + it);
});
});
CompletableFuture<Void> future = CompletableFuture.allOf(producer, producer2).thenAccept(it -> processor.onComplete());
flux.doOnComplete(latch::countDown).subscribe();
future.join();
latch.await();
System.out.println("Total: " + counter);
}
The counter shows us that each time we execute this code the actual number of messages processed is different.
What's wrong with this implementation?
How can we ensure that all the messages were processed before the program ends?

Answer 1

What's wrong with this implementation?
When I run the code I get the following in the logs early after start:
18:39:12.590 [ForkJoinPool.commonPool-worker-1] DEBUG reactor.core.publisher.Operators - Duplicate Subscription has been detected
java.lang.IllegalStateException: Spec. Rule 2.12 - Subscriber.onSubscribe MUST NOT be called more than once (based on object equality)
at reactor.core.Exceptions.duplicateOnSubscribeException(Exceptions.java:162)
at reactor.core.publisher.Operators.reportSubscriptionSet(Operators.java:502)
at reactor.core.publisher.Operators.setOnce(Operators.java:607)
at reactor.core.publisher.EmitterProcessor.onNext(EmitterProcessor.java:245)
at de.schauder.reactivethreads.demo.StackoverflowQuicky.lambda$null$2(StackoverflowQuicky.java:54)
at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
at java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:557)
at de.schauder.reactivethreads.demo.StackoverflowQuicky.lambda$main$3(StackoverflowQuicky.java:52)
I'm not familiar with EmitterProcessor but it seems onNext is not thread safe and I'm strongly suspecting that this is the cause for the missing events.
How can we ensure that all the messages were processed before the program ends?
I'd use two separate Producers and merge those. Also I think you don't need the count down latch.
public static void main(String[] args) {
AtomicLong counter = new AtomicLong();
AtomicLong batch = new AtomicLong();
EmitterProcessor<String> processor1 = EmitterProcessor.create();
EmitterProcessor<String> processor2 = EmitterProcessor.create();
Thread thread1 = constructThread(processor1);
Thread thread2 = constructThread(processor2);
Flux<List<String>> flux = processor1.mergeWith(processor2)
.buffer(7)
.onBackpressureError()
.publishOn(Schedulers.immediate())
.doOnNext(it -> {
counter.addAndGet(it.size());
System.out.println(batch.incrementAndGet() + " : Batch: " + it.size());
}).doOnComplete(() -> {
System.out.println("Total count: " + counter.get());
});
thread1.start();
thread2.start();
flux.blockLast();
}
private static Thread constructThread(EmitterProcessor<String> processor) {
return new Thread(() -> {
IntStream.range(1, 1001).forEach(it -> {
processor.onNext("Message2-" + it);
});
processor.onComplete();
});
}
Note about my comment:
onBackpressureError() causes the Flux to emit an error when the subscriber can't handle all the events fast enough, so this could explain the mismatch, but you'd see an exception.