spring webflux stream Completion Consumer - rest

Question

I have spring webflux stream consumer which calls a REST endpoint and consumes the messages received and save to an RDBMS. i am trying to find a way to batch it. I see the subscribe() has an overloaded method which gets called on Completion. I am trying to find how to get hold of the data when this completion consumer gets called since i am calling a CompletionConsumer which is of type Runnable and all i am having is the run() method which dont take any parameters.
**CLIENT**
WebClient.create("http://localhost:8080")
.get()
.uri("/objects")
.accept(MediaType.TEXT_EVENT_STREAM)
.exchange()
.flatMapMany(clientResponse ->clientResponse.bodyToFlux(MyObject.class))
.subscribe(null,null,completionProcessorSubscriber);
**COMPLETION SUBSCRIBER**
#Service
public class CompletionProcessorSubscriber implements Runnable{
#Autowired
LegacyDAOImpl dao;
Logger logger = LoggerFactory.getLogger(CompletionProcessorSubscriber.class);
public void run() {
logger.info("\ninside RUNNNNNNNNN\n\n");
// here how to get hold of the data stream ?
}
Below is the Documentation from the Flux API
*/
public final Disposable subscribe(
#Nullable Consumer<? super T> consumer,
#Nullable Consumer<? super Throwable> errorConsumer,
#Nullable Runnable completeConsumer) {
return subscribe(consumer, errorConsumer, completeConsumer, null);
}

Answer 1

You should avoid adding to much logic to subscriber methods. Instead, you should utilize the rich set of operators provided by Flux API.
In this case the operators you need are buffer to collect batches and concatMap to execute batches sequentially.
In the following example I assume the LegacyDAOImpl is a blocking service whose work should be assigned to an appropriate thread pool.
public static void main(String[] args) throws InterruptedException
{
webClient.get()
.uri("/objects")
.accept(MediaType.TEXT_EVENT_STREAM)
.exchange()
.flatMapMany(clientResponse -> clientResponse.bodyToFlux(MyObject.class))
.buffer(100) // batch size
.concatMap(batchOfMyObjects -> Mono.fromRunnable(() -> legacyDAOImpl.saveAll(batchOfMyObjects))
.subscribeOn(Schedulers.elastic())) // blocking IO goes to elastic thread pool
.subscribe();
}
private static class LegacyDAOImpl
{
public void saveAll(List<MyObject> myObjects)
{
// save here
}
}
private static class MyObject
{
}