Spring webflux/reactor using #Scheduled to read database and perform some tasks - reactive-programming

Question

I am new to spring webflux and my current spring boot application uses a scheduler(annotated as #Scheduled) to read list of data from DB, call a rest api concurrently in batches and then writes to event stream
I want to achieve the same in Spring webflux.
Should I use #Scheduled or use schedulePeriodically from Webflux?
How can I batch items from DB into smaller sets(say 10 items) and concurrently call rest api?
At present the app fetches max 100 records in one scheduler run and then it processes them. I am planning to shift to r2dbc, if i do so, can i limit the flow of data like 100?
Thanks

Answer 1

1. Should I use #Scheduled or use schedulePeriodically from Webflux?
#Scheduled is an annotation which is part of the spring framework scheduled package, while schedulePeriodically is a function which is part of reactor, so you can't really compare the two. I dont see any problems in using the annotation since it is part of the core framework.
2. How can I batch items from DB into smaller sets (say 10 items) and concurrently call rest api?
By using the Flux#buffer functions which will emit a list of items when the buffer is full.
Flux.just("1", "2", "3", "4")
.buffer(2)
.doOnNext(list -> {
System.out.println(list.size());
}).subscribe()
Will print 2 each time.
3. At present the app fetches max 100 records in one scheduler run and then it processes them. I am planning to shift to r2dbc, if i do so, can i limit the flow of data like 100?
Well you can as written before, you fetch, and then buffer the responses into lists of 100, you can then place each list in its own flux and emit items again, or process each list of 100 items. Up to you.
There are a lot of functions under the buffer segment, check them out.
Flux#buffer

Answer 2

Flux.buffer will combine the streams and will emit a list of streams of mentioned buffer size.
For batching purpose, you can use Flux.expand or Mono.expand. You only have to provide your condition in the expand to execute it again or finally end it.
Here are the examples:
public static void main(String[] args) {
List<String> list = new ArrayList<>();
list.add("1");
Flux.just(list)
.buffer(2)
.doOnNext(ls -> {
System.out.println(ls.getClass());
// Buffering a list returns the list of list of String
System.out.println(ls);
}).subscribe();
Flux.just(list)
.expand(listObj -> {
// Condition to finally end the batch
if (listObj.size()>4) {
return Flux.empty();
}
// Can return the size of data as much as you require
list.add("a");
return Flux.just(listObj);
}).map(ls -> {
// Here it returns list of String which was the original object type not list of list as in case of buffer
System.out.println(ls.getClass());
System.out.println(ls);
return ls;
}).subscribe();
}
Output:
class java.util.ArrayList
[[1]] /// Output of buffer list of list
class java.util.ArrayList
[1]
class java.util.ArrayList
[1, a]
class java.util.ArrayList
[1, a, a]
class java.util.ArrayList
[1, a, a, a]
class java.util.ArrayList
[1, a, a, a, a]