I think this will be done similarly with most reactive implementations so I do not specify any specific library&language here.
What is the proper way to do reactive side effects on streams (not sure what is the proper terminology here)?
So lets say we have a stream of Players. We want to transform that to a stream of only ids. But in addition to that, we want to do some additional reactive processing based on the ids.
Here is how one can do in not so elegant way, but what is the idiomatic way to achieve this:
Observable<Player> playerStream = getPlayerStream();
return playerStream
.map(p -> p.id)
.flatmap(id -> {
Observable<Result> weDontCare = process(id);
// Now the hacky part
return weDontCare.map(__ -> id);
})
Is that ok? Seems not so elegant.
I don't know RxJava, but you also tagged project reactor, and there are two ways to do this with reactor depending on how you want the side-effects to affect your stream. If you want to wait for the side effects to happen so you can handle errors etc (my preferred way) use delayUntil:
return getPlayerStream()
.map(p -> p.id)
.delayUntil(id -> process(id));
If you want the id to pass straight through without waiting, and instead do a fire-and-forget style side-effect, you could use doOnNext:
return getPlayerStream()
.map(p -> p.id)
.doOnNext(id -> process(id).subscribe());
Observable<Player> playerStream = getPlayerStream();
return playerStream
.map(p -> p.id)
.doOnEach(id -> { //side effect with id
Observable<Result> weDontCare = process(id);
weDontCare.map(__ -> id);
})
Related
I'm struggling with groupBy in RxJava.
The problem is - I cant get only one element from each group.
For example i have a list of elements:
SomeModel:
class SomeModel {
int importantField1;
int mainData;
}
My list of models for example:
List<SomeModel> dataList = new ArrayList<>();
dataList.add(new SomeModel(3, 1));
dataList.add(new SomeModel(3, 1));
dataList.add(new SomeModel(2, 1));
In my real project there is more complex data model. I added same models on purpose. It is matter for my project.
Then I'm trying to take one element from group in this manner:
List<SomeModel> resultList = Observable.fromIterable(dataList)
.sorted((s1, s2) -> Long.compare(s2.importantField1, s1.importantField1))
.groupBy(s -> s.importantField1)
.firstElement()
// Some transformation to Observable. May be it is not elegant, but it is not a problem
.flatMapObservable(item -> item)
.groupBy(s -> s.mainData)
//Till this line I have all I need. But then I need to take only one element from each branch
.flatMap(groupedItem -> groupedItem.firstElement().toObservable())
.toList()
.blockingGet();
And of course it's not working. I still have two same elements in the resultList.
I cant add .firstElement(), after last .flatMap operator, because there could be situations when after last .groupBy may be more then one branch.
I need only one element from each branch.
I've tryed this way:
.flatMap(groupedItem -> groupedItem.publish(item -> item.firstElement().concatWith(item.singleElement()).toObservable())
no effect. This sample of code I took from this post: post
There author suggests this :
.flatMap(grp -> grp.publish(o -> o.first().concatWith(o.ignoreElements())))
but even if I remove last two rows of my code:
.toList()
.blockingGet();
And change resultList to Disposable disposable suggested option not working because of error:
.concatWith(o.ignoreElements()) - concatWith not taking Completable.
Some method signatures changed with 3.x since my post so you'll need these:
first -> firstElement
firstElement returns Maybe which is no good inside publish, plus there is no Maybe.concatWith(CompletableSource), thus the need to convert to Observable.
.flatMap(grp ->
grp.publish(o ->
o.firstElement()
.toObservable()
.concatWith(o.ignoreElements())
)
)
In doing my joins, I am finding that the 2nd block tends to give the expected result, whereas the 1st block does not and never hits the (aValue, bValue) -> myFunc(aValue, bValue). I didn't think the actual key mattered as long as I set the right field to join on (aKey, aValue) -> aValue.get("someField").asText(), but there is something about using .selectKey((aKey, aValue) -> aValue.get("someField").asText()) beforehand that makes the join go through correctly. I have also seen some cases that did not require the selectKey. Can someone explain the difference?
// does not join correctly and gives unexpected result
KStream<String, JsonNode> c = a
.leftJoin(b,
(aKey, aValue) -> aValue.get("someField").asText(),
(aValue, bValue) -> myFunc(aValue, bValue)
);
// does join correctly and gives expected result
KStream<String, JsonNode> c = a
.selectKey((aKey, aValue) -> aValue.get("someField").asText())
.leftJoin(b,
(aKey, aValue) -> aKey,
(aValue, bValue) -> myFunc(aValue, bValue)
);
There are many different joins with different semantics in Kafka Streams and I am not 100% sure what join you execute?
Given your example, it seems you are using a KStream-GlobalKTable join; b seems to be a GlobalKTable and the second argument (i.e., (aKey, aValue) -> aValue.get("someField").asText() is your keySelector?
If this it correct, the first code snippet looks correct to me. What version are you using (maybe there is some bug in Kafka Streams)? Can you also share the output of Topology#describe()#toString() for both cases?
I am trying to manipulate my objects received from Flux with data received from a Mono where the methods emitting the Flux of object and Mono of items are both different API calls. The problem is, I don't have control over the threads and the items received from the Mono are never assigned to my object unless I intentionally block() that thread. Kindly suggest if any non-blocking way possible for this scenario.
I have also looked into Schedulers, subscribeOn, publishOn but unable to figure out the pipeline.
public Flux<Object> test {
method1().map(obj -> {
if (obj.getTotalItems() > 20) {
obj.setItems(method2(obj).block());
}
return obj;
});
}
Here method1 is emitting Flux of objects received from API hit.
And method2 is emitting a list of items fetched from another API hit.
How can I make this whole flow non-blocking?
Try flatMap or concatMap
using flatMap operator you can flatten substream in non-blocking public
Flux<Object> test {
method1().flatMap(obj -> {
if (obj.getTotalItems() > 20) {
return method2(obj)
.map(result -> {
obj.setItems(result);
return obj;
});
}
return Mono.just(obj);
});
}
flatMap allows you to flatten several streams at a time, so in case of long-running operations, you may in more efficient process elements.
One downside of flatMap is that it does not preserve the order of elements so if you have a sequence of upstream elements like [1, 2, 3, 4] with flatMap there is a chance that the order will be changed because of asynchronous nature of substreams.
To preserve order, you can use concatMap which flatten only once streams at a time, so there are guarantees that order of flattening elements will be preserved:
Flux<Object> test {
method1().concatMap(obj -> {
if (obj.getTotalItems() > 20) {
return method2(obj)
.map(result -> {
obj.setItems(result);
return obj;
});
}
return Mono.just(obj);
});
}
Note
Mutation of the objects such a way is not the best idea, and I would prefer to use immutable object pattern object in reactive programming
I noticed in the reactive libraries there are Tuples, but what do I do if there are more than 8 Tuples?
https://projectreactor.io/docs/core/release/api/reactor/util/function/Tuples.html#fromArray-java.lang.Object:A-
Example code that seems to work, but is there a better way to use some sort of collector?
private Mono<List<String>> getContent(List<String> ids) {
List<String> allContent = new ArrayList<>();
Mono<List<String>> allContentMono = Mono.empty();
for(String id : ids) {
allContentMono = callApi(id)
.flatMap(result -> result.bodyToMono(String.class))
.map(str -> {
allContent.add(str);
return allContent;
});
}
return allContentMono;
}
Why did the tuple size stop at 8? (haven't looked around for the documentation on why, but not my main concern)
Thanks
zip (which uses TupleN) is for when you want to create values by compositon, out of a combination of sources. Eg. out of a Flux<FirstName> and Flux<LastName> you want a Flux<FullName>, that emits one FullName for each incoming FistName/LastName pair.
For your use case, where you want to execute multiple calls (possibly in parallel) and collect the results in a list, flatMap is enough:
private Mono<List<String>> getContent(List<String> ids) {
return Flux
.fromIterable(ids)
.flatMap(id -> callApi(id))
.flatMap(response -> response.bodyToMono(String.class))
.collectList();
}
Tuple is an immutable, fixed-size data structure, used by zip as convenience when you don't want to create a dedicated POJO. It doesn't make sense to try and support unlimited sizes so we stopped at eight. There is a zip variant that will aggregate more than 8 sources, but will make you work with an Object[] instead of a Tuple.
i'm a RxJava newcomer, and i'm having some trouble wrapping my head around how to do the following.
i'm using Retrofit to invoke a network request that returns me a Single<Foo>, which is the type i ultimately want to consume via my Subscriber instance (call it SingleFooSubscriber)
Foo has an internal property items typed as List<String>.
if Foo.items is not empty, i would like to invoke separate, concurrent network requests for each of its values. (the actual results of these requests are inconsequential for SingleFooSubscriber as the results will be cached externally).
SingleFooSubscriber.onComplete() should be invoked only when Foo and all Foo.items have been fetched.
fetchFooCall
.subscribeOn(Schedulers.io())
// Approach #1...
// the idea here would be to "merge" the results of both streams into a single
// reactive type, but i'm not sure how this would work given that the item emissions
// could be far greater than one. using zip here i don't think it would every
// complete.
.flatMap { foo ->
if(foo.items.isNotEmpty()) {
Observable.zip(
Observable.fromIterable(foo.items),
Observable.just(foo),
{ source1, source2 ->
// hmmmm...
}
).toSingle()
} else {
Single.just(foo)
}
}
// ...or Approach #2...
// i think this would result in the streams for Foo and items being handled sequentially,
// which is not really ideal because
// 1) i think it would entail nested streams (i get the feeling i should be using flatMap
// instead)
// 2) and i'm not sure SingleFooSubscriber.onComplete() would depend on the completion of
// the stream for items
.doOnSuccess { data ->
if(data.items.isNotEmpty()) {
// hmmmm...
}
}
.observeOn(AndroidSchedulers.mainThread())
.subscribe(
{ data -> /* onSuccess() */ },
{ error -> /* onError() */ }
)
any thoughts on how to approach this would be greatly appreciated!
bonus points: in trying to come up with a solution to this, i've begun to question the decision to use the Single reactive type vs the Observable reactive type. most (all, except this one Foo.items case?) of my streams actually revolve around consuming a single instance of something, so i leaned toward Single to represent my streams as i thought it would add some semantic clarity around the code. anybody have any general guidance around when to use one vs the other?
You need to nest flatMaps and then convert back to Single:
retrofit.getMainObject()
.flatMap(v ->
Flowable.fromIterable(v.items)
.flatMap(w ->
retrofit.getItem(w.id).doOnNext(x -> w.property = x)
)
.ignoreElements()
.toSingle(v)
)