parallelize the filter operation - reactive-programming

Question

I am trying to parallelize the filter operation of a Flux. However, from the time taken to complete the operation, it doesn't seem to be parallelizing. Any insight into what I may be doing wrong here would be greatly appreciated. Thanks.
#Test
public void testParallelFilteringFlux() {
long start = Calendar.getInstance().getTimeInMillis();
log.info("Start time ::{}",Calendar.getInstance().getTimeInMillis());
Flux<Integer> fluxFromJust = Flux.range(1, 1000000);
ParallelFlux<Integer> pfilter = fluxFromJust.filter(i -> i == 99999).parallel(4).runOn(Schedulers.parallel());//filter the even numbers only
Flux<Integer> filter = fluxFromJust.filter(i -> i == 99999);
filter.subscribe(i->log.info(">>>>>>>>> Found Integer: {}, time: {}",i, Calendar.getInstance().getTimeInMillis() - start));
pfilter.subscribe(i->log.info(">>>>>>>>> Parallel Found Integer: {}, time: {}",i, Calendar.getInstance().getTimeInMillis() - start));
}
Output is:
20:37:29.733 [main] INFO test.ReactorTest - Start time ::1614092849730
20:37:30.040 [main] DEBUG reactor.util.Loggers - Using Slf4j logging framework
20:37:30.107 [main] INFO test.ReactorTest - >>>>>>>>> Found Integer: 99999, time: 377
20:37:30.190 [parallel-1] INFO test.ReactorTest - >>>>>>>>> Parallel Found Integer: 99999, time: 460
Process finished with exit code

Answer 1

It is done in parallel, but there are several points explaining why it is longer using parallel in your test, that I will try to explain.
First, your test is not really accurate because:
The 2 processes (1 single threaded, and 1 parallel) are executed in parallel. If you want your result to be more precise, you should run one after the other
You should execute the test at least 2 times, because the first time several things are not yet initialized, so the timing take into account class loading, schedulers initialization, etc... and we should not take them into account when comparing the 2 solutions.
But that not the most important point. Actually processing the filter in parallel is requesting much more work behind the scene, to split the data and dispatch it to the different threads. So because the predicate in your filter is very simple (only a comparison), at the end it is more efficient to do it in one shot in a single thread instead of doing it in a parallel way. The parallel way will become more efficient if the processing time of the filter is more important, because this processing time will be (more or less) divided by the number of parallel threads.
I rewrite your test to illustrate those points:
I run the test 2 times to avoid to take the initialization into account in the timing
I wait for the single threaded test to finish before to launch the parallel one, so they cannot interfer
Finally, I test with a very simple predicate (as in your test), and a predicate that is longer (I just put a sleep to simulate a longer processing). Note that I reduce the number of items in the longer predicate to have the result faster.
Here is the code:
#Test
public void testParallelFilteringFlux() throws Exception {
Predicate<Integer> predicate;
System.out.println("Test with short process in the predicate");
final int nb1 = 1000000;
predicate = i -> i == nb1 - 1;
runTest(nb1, predicate);
runTest(nb1, predicate);
System.out.println("Test with longer process in the predicate");
final int nb2 = 10000;
predicate = i -> {
try {
Thread.sleep(1);
} catch (InterruptedException e) {
// ignore
}
return i == nb2 - 1;
};
runTest(nb2, predicate);
runTest(nb2, predicate);
}
private void runTest(int nb, Predicate<Integer> predicate) {
long start = System.currentTimeMillis();
List<Integer> result = testSingleThread(nb, predicate).collectList().block();
System.out.println("Found with single thread " + result + " in " + (System.currentTimeMillis() - start) + "ms.");
start = System.currentTimeMillis();
result = testParallel(nb, predicate).collectList().block();
System.out.println("Found with parallel " + result + " in " + (System.currentTimeMillis() - start) + "ms.");
}
private Flux<Integer> testSingleThread(int nb, Predicate<Integer> predicate) {
Flux<Integer> fluxFromJust = Flux.range(1, nb);
Flux<Integer> filter = fluxFromJust.filter(predicate);
return filter;
}
private Flux<Integer> testParallel(int nb, Predicate<Integer> predicate) {
Flux<Integer> fluxFromJust = Flux.range(1, nb);
ParallelFlux<Integer> pfilter = fluxFromJust.parallel(4).runOn(Schedulers.parallel()).filter(predicate);
return pfilter.sequential();
}
And here is the output:
Test with short process in the predicate
Found with single thread [999999] in 126ms.
Found with parallel [999999] in 326ms.
Found with single thread [999999] in 6ms.
Found with parallel [999999] in 191ms.
Test with longer process in the predicate
Found with single thread [9999] in 17474ms.
Found with parallel [9999] in 4528ms.
Found with single thread [9999] in 17575ms.
Found with parallel [9999] in 4563ms.
As you can see, with a short predicate, the single threaded test is faster, but if the processing time of the predicate is longer, the time is almost divided by 4.