I noticed in the reactive libraries there are Tuples, but what do I do if there are more than 8 Tuples?
https://projectreactor.io/docs/core/release/api/reactor/util/function/Tuples.html#fromArray-java.lang.Object:A-
Example code that seems to work, but is there a better way to use some sort of collector?
private Mono<List<String>> getContent(List<String> ids) {
List<String> allContent = new ArrayList<>();
Mono<List<String>> allContentMono = Mono.empty();
for(String id : ids) {
allContentMono = callApi(id)
.flatMap(result -> result.bodyToMono(String.class))
.map(str -> {
allContent.add(str);
return allContent;
});
}
return allContentMono;
}
Why did the tuple size stop at 8? (haven't looked around for the documentation on why, but not my main concern)
Thanks
zip (which uses TupleN) is for when you want to create values by compositon, out of a combination of sources. Eg. out of a Flux<FirstName> and Flux<LastName> you want a Flux<FullName>, that emits one FullName for each incoming FistName/LastName pair.
For your use case, where you want to execute multiple calls (possibly in parallel) and collect the results in a list, flatMap is enough:
private Mono<List<String>> getContent(List<String> ids) {
return Flux
.fromIterable(ids)
.flatMap(id -> callApi(id))
.flatMap(response -> response.bodyToMono(String.class))
.collectList();
}
Tuple is an immutable, fixed-size data structure, used by zip as convenience when you don't want to create a dedicated POJO. It doesn't make sense to try and support unlimited sizes so we stopped at eight. There is a zip variant that will aggregate more than 8 sources, but will make you work with an Object[] instead of a Tuple.
Related
How to compare List to Flux in non blocking way
Below the code in blocking way
public static void main(String[] args) {
List<String> all = List.of("A", "B", "C", "D");
Flux<String> valid = Flux.just("A", "D");
Map<Boolean, List<String>> collect = all.stream()
.collect(Collectors.groupingBy(t -> valid.collectList().block().contains(t)));
System.out.println(collect.get(Boolean.TRUE));
System.out.println(collect.get(Boolean.FALSE));
}
how to get it working in non-blocking way?
Above is an example of what i am trying to do in web application. I receive list of object which is List all. Then i query database which return Flux . Flux returned by database will be subset of List all. I need to prepare two lists. List of items which are present in Flux of valid and List of items which are not present in Flux of valid
EDIT:
I converted Flux to Mono and List to Mono,
public static void main(String[] args) {
Mono<List<String>> all = Mono.just(List.of("A", "B", "C", "D"));
Mono<List<String>> valid = Mono.just(List.of("A", "D"));
var exist = all.flatMap(a -> valid.map(v -> a.stream().collect(Collectors.groupingBy(v::contains))));
System.out.println(exist.block().get(Boolean.TRUE));
System.out.println(exist.block().get(Boolean.FALSE));
}
There is no straightforward way of achieve this in reactive programming without breaking some of its semantics.
If you reflect back on what reactive programming tris to achieve and your problem statement, you should notice that those won't play well that much together.
Reactive programming, as the name suggests, is about reacting to events which in your case would be valid items emitted from your datastore. In a typical situation, you should have been programming your statement to compute some assertions around the emitted valid items then emit these (or some other transformations downstream). Unfortunately, you won't be able to compute the all and valid items intersection and diversion without stopping at some point (otherwise how would you know that an item you assumed non-valid is not emitted at some point by the valid publisher).
Though, to achieve the desired behavior, you will lean on memory to buffer items then trigger your validations.
Retrieving valid items should be achievable using the filterWhen operator paired with the hasElement one:
Flux<String> validItems = Flux.fromIterable(all)
.filterWhen(valid::hasElement);
To retrieve the invalid items, you can collect all and validItems merged together then filter out elements that do appear more than once:
Flux<String> inValidItems = Flux.fromIterable(all)
.mergeWith(validItems)
.collectList()
.flatMapIterable(list -> list.stream().filter(item -> Collections.frequency(list, item) == 1).collect(Collectors.toList()));
I am new to spring webflux and my current spring boot application uses a scheduler(annotated as #Scheduled) to read list of data from DB, call a rest api concurrently in batches and then writes to event stream
I want to achieve the same in Spring webflux.
Should I use #Scheduled or use schedulePeriodically from Webflux?
How can I batch items from DB into smaller sets(say 10 items) and concurrently call rest api?
At present the app fetches max 100 records in one scheduler run and then it processes them. I am planning to shift to r2dbc, if i do so, can i limit the flow of data like 100?
Thanks
1. Should I use #Scheduled or use schedulePeriodically from Webflux?
#Scheduled is an annotation which is part of the spring framework scheduled package, while schedulePeriodically is a function which is part of reactor, so you can't really compare the two. I dont see any problems in using the annotation since it is part of the core framework.
2. How can I batch items from DB into smaller sets (say 10 items) and concurrently call rest api?
By using the Flux#buffer functions which will emit a list of items when the buffer is full.
Flux.just("1", "2", "3", "4")
.buffer(2)
.doOnNext(list -> {
System.out.println(list.size());
}).subscribe()
Will print 2 each time.
3. At present the app fetches max 100 records in one scheduler run and then it processes them. I am planning to shift to r2dbc, if i do so, can i limit the flow of data like 100?
Well you can as written before, you fetch, and then buffer the responses into lists of 100, you can then place each list in its own flux and emit items again, or process each list of 100 items. Up to you.
There are a lot of functions under the buffer segment, check them out.
Flux#buffer
Flux.buffer will combine the streams and will emit a list of streams of mentioned buffer size.
For batching purpose, you can use Flux.expand or Mono.expand. You only have to provide your condition in the expand to execute it again or finally end it.
Here are the examples:
public static void main(String[] args) {
List<String> list = new ArrayList<>();
list.add("1");
Flux.just(list)
.buffer(2)
.doOnNext(ls -> {
System.out.println(ls.getClass());
// Buffering a list returns the list of list of String
System.out.println(ls);
}).subscribe();
Flux.just(list)
.expand(listObj -> {
// Condition to finally end the batch
if (listObj.size()>4) {
return Flux.empty();
}
// Can return the size of data as much as you require
list.add("a");
return Flux.just(listObj);
}).map(ls -> {
// Here it returns list of String which was the original object type not list of list as in case of buffer
System.out.println(ls.getClass());
System.out.println(ls);
return ls;
}).subscribe();
}
Output:
class java.util.ArrayList
[[1]] /// Output of buffer list of list
class java.util.ArrayList
[1]
class java.util.ArrayList
[1, a]
class java.util.ArrayList
[1, a, a]
class java.util.ArrayList
[1, a, a, a]
class java.util.ArrayList
[1, a, a, a, a]
i'm a RxJava newcomer, and i'm having some trouble wrapping my head around how to do the following.
i'm using Retrofit to invoke a network request that returns me a Single<Foo>, which is the type i ultimately want to consume via my Subscriber instance (call it SingleFooSubscriber)
Foo has an internal property items typed as List<String>.
if Foo.items is not empty, i would like to invoke separate, concurrent network requests for each of its values. (the actual results of these requests are inconsequential for SingleFooSubscriber as the results will be cached externally).
SingleFooSubscriber.onComplete() should be invoked only when Foo and all Foo.items have been fetched.
fetchFooCall
.subscribeOn(Schedulers.io())
// Approach #1...
// the idea here would be to "merge" the results of both streams into a single
// reactive type, but i'm not sure how this would work given that the item emissions
// could be far greater than one. using zip here i don't think it would every
// complete.
.flatMap { foo ->
if(foo.items.isNotEmpty()) {
Observable.zip(
Observable.fromIterable(foo.items),
Observable.just(foo),
{ source1, source2 ->
// hmmmm...
}
).toSingle()
} else {
Single.just(foo)
}
}
// ...or Approach #2...
// i think this would result in the streams for Foo and items being handled sequentially,
// which is not really ideal because
// 1) i think it would entail nested streams (i get the feeling i should be using flatMap
// instead)
// 2) and i'm not sure SingleFooSubscriber.onComplete() would depend on the completion of
// the stream for items
.doOnSuccess { data ->
if(data.items.isNotEmpty()) {
// hmmmm...
}
}
.observeOn(AndroidSchedulers.mainThread())
.subscribe(
{ data -> /* onSuccess() */ },
{ error -> /* onError() */ }
)
any thoughts on how to approach this would be greatly appreciated!
bonus points: in trying to come up with a solution to this, i've begun to question the decision to use the Single reactive type vs the Observable reactive type. most (all, except this one Foo.items case?) of my streams actually revolve around consuming a single instance of something, so i leaned toward Single to represent my streams as i thought it would add some semantic clarity around the code. anybody have any general guidance around when to use one vs the other?
You need to nest flatMaps and then convert back to Single:
retrofit.getMainObject()
.flatMap(v ->
Flowable.fromIterable(v.items)
.flatMap(w ->
retrofit.getItem(w.id).doOnNext(x -> w.property = x)
)
.ignoreElements()
.toSingle(v)
)
I am trying to implement calling read() method in the itemReader multiple times.
For Eg:
I have a list of POJO in which I will have one string variable with values either A or B or C.
I have to sort this list based on alphabetical order and segment it into three list for each value. i.e., list for value A and list for value B
and list for value C.
I need to send each list to the read() method in the itemReader one by one.
Once List for A is processed and write, then I need to send List for B and so on..
Is this doable? Any help is appreciated.
Although I am not very clear on what you are trying to achieve, I don't see any reason it cannot be done.
I assume you mean either of this:
1. You want the "item" to be process to be a whole list of POJO with same ABC Type, or
2. You want the item to be the POJO itself, and you want them to be processed in order of ABC Type
2 is straight-forward. At the first read, prepare all the POJOs, sort it. I assume they are in some kind of
In psuedo code, it looks like this
class MyReader implements ItemReader<MyPojo> {
private List<MyPojo> values;
MyPojo read() {
if (values == null) {
values = getPojos();
sort values;
}
if (values.isEmpty()){
return null;
} else {
return values.popFront();
}
}
}
1 is nothing more complicated. You will need to group POJOs with same ABC type in same list, and return the lists one by one. It can be easily done by using a TreeMap<String, List<MyPojo>>
In psuedo code, it looks like this
class MyReader implements ItemReader<List<MyPojo>> { // note the item is List<MyPojo>
private NavigableMap<String, List<MyPojo>> values;
List<MyPojo> read() {
if (values == null) {
values = new TreeMap<>();
pojos = getPojos();
for (pojo : pojos) {
if (values do not contain pojo.abcType() ) {
values.put(pojo.abcType(), new ArrayList(pojo));
} else {
values.get(pojo.abcType()).add(pojo);
}
}
}
if (values.isEmpty()){
return null;
} else {
return values.popFirstEntry().value();
}
}
}
If your list of items is fully available (you have a List<Pojo> loaded with all items) you can:
use a ListItemReader and inject into the ordered list
use a custom ItemReader and sort items after first ItemReader.read()
About break the best way is to use a custom CompletionPolicy based on pojo 'string variable'; in this manner your writer will receive a list where POJO's 'string variable' has the same values for all list items (check How to read csv lines chunked by id-column with Spring-Batch? for sample code).
I'm trying to get my head around using CoffeeScript comprehensions as efficiently as possible. I think I have basic mapping down -- turning one list into another -- but searching still seems verbose to me.
Say I have a map of items to shops:
shopMap:
toyStore: ["games", "puzzles"]
bookStore: ["novels", "picture books"]
and, given an item, I want to find out which shop it's in. What's the best way of doing that in CoffeeScript?
Here's how I could do in in JavaScript:
var shop = findShop(item);
function findShop(item) {
for (shop in shopMap)
itemList = shopMap[shop]
for (i = 0, ii = itemList.length; i<ii; i++) {
if (itemList[i] === item) {
return shop;
}
}
}
}
I used a function to allow it to quickly break out of the loops with the return statement, instead of using breaks, but the function is kind of fugly as this is only being used once.
So is there a shorter CS equivalent preferably one that doesn't require creating a new function?
You can try this:
findShop = (item) ->
for shop, items of shopMap
return shop if item in items
If you really want to try with a list comprehension, this is equivalent:
findShop = (item) ->
(shop for shop, items of shopMap when item in items)[0]
But i think the first one reads better (and also doesn't need to generate an intermediate array for the results). This would be a better approach IMO if you wanted to find all shops for a given item:
findShops = (item) ->
shop for shop, items of shopMap when item in items
If this is a common operation, you might be better off creating an intermediate data structure up front and doing the lookup directly.
shopMap =
toyStore: ["games", "puzzles"]
bookStore: ["novels", "picture books"]
categoryMap = {}
for k, v of shopMap
for category in v
categoryMap[category] = k
alert(categoryMap['puzzles'])
Demo
With this implementation you need to loop through the structure only once up front (plus possibly update it if shopMap changes). With yours and epidemian's answer, you have to loop every time you need to do this particular type of lookup. If you do this operation a lot, it could make a difference. On the other hand, if your shopMap is really large (like thousands of entries), then my implementation will take up more memory.
Depending upon how robust you want to make this, you might want to turn it into a Class and have any operations on it occur through the Class' interface. You'd need addCategory and deleteCategory methods as well as a getStoreFromCategory method, which is essentially what we are implementing above. This object-oriented approach would hide the internal data-structure/implementation so you could later alter the implementation to optimize for memory or speed.