How to process all items in List then complete with RxJava - rx-java2

I am investigating the use of RxJava in my current Android application.
I am stuck with the following use case.
For each data row on a particular database table I wish to action an HTTP POST call, once all the POST(s) have completed OK I need to clear down the database table.
The code I have is as follows:-
login()
.andThen(Single.defer(() -> DatabaseController.fetchSingleRealmObjects(UpdateDO.class)))
.toObservable()
.flatMapIterable(update -> update)
.flatMap(this::parameteriseUpdate)
.doOnNext(NetworkController::update)
.doOnComplete(() -> DatabaseController.deleteAll(UpdateDO.class))
.ignoreElements()
.retryWhen(errors -> errors.flatMap(e -> constructRetryHandler(retryCounter, e)))
.doOnComplete(onComplete)
.doOnError(onError)
.doAfterTerminate(doAfterTerminate())
.doOnSubscribe(compositeDisposable::add)
.blockingAwait();
When the UpdateDO table is empty the above code completes as expected.
However when data rows exist the process "sticks" in the doOnNext()
I realise this is because I only call emitter.onNext()
private ObservableSource<Map<String, Object>> parameteriseUpdate(final UpdateDO updateDO) {
final Map<String, Object> fields = new HashMap<>();
fields.put(FIELD_NAME_DRUG_ID, updateDO.getDrugId());
fields.put(FIELD_NAME_STORE_CONTENT_ID, updateDO.getStoreContentId());
fields.put(FIELD_NAME_STORE_ID, updateDO.getStoreID());
fields.put(FIELD_NAME_ACTUAL_QUANTITY, updateDO.getActualQty());
fields.put(FIELD_NAME_VARIANCE, updateDO.getUnitQty());
fields.put(FIELD_NAME_REMARKS, updateDO.getRemarks());
fields.put(FIELD_NAME_CREATED_BY, updateDO.getCreatedBy());
return Observable.create(emitter -> emitter.onNext(fields));
}
I cannot see how to fix this though, how to I refactor my code to allow me to have emitter.onComplete() called?

I think that is better to change that function from returning ObservableSource<Map<String, Object>>
to simple function like this
private Map<String, Object> parameteriseUpdate(final UpdateDO updateDO) {
final Map<String, Object> fields = new HashMap<>();
fields.put(FIELD_NAME_DRUG_ID, updateDO.getDrugId());
fields.put(FIELD_NAME_STORE_CONTENT_ID, updateDO.getStoreContentId());
fields.put(FIELD_NAME_STORE_ID, updateDO.getStoreID());
fields.put(FIELD_NAME_ACTUAL_QUANTITY, updateDO.getActualQty());
fields.put(FIELD_NAME_VARIANCE, updateDO.getUnitQty());
fields.put(FIELD_NAME_REMARKS, updateDO.getRemarks());
fields.put(FIELD_NAME_CREATED_BY, updateDO.getCreatedBy());
return fields;
}
and call it as a map not flatmap like this :
.map(this::parameteriseUpdate)
because in your case you are creating many streams that never complete.

Related

Reactive programming - Return response when the result is empty

I have a reactive code below which does the following.
query items table by ID - (itemRepository.findByd(itemId))
Create a map from the result from Step1
Invoke retrieveItemsQty from DB
Here, it works fine when the Step1 returns one or more result and retrieveItemsQty method fails when the result is empty. My requirement is to return back when the first step result(itemRepository.findByd) is empty. Not sure how to do this?
private Mono<Long> updateItemsQty(final Long itemId) {
return itemRepository.findByd(itemId).collectList()
.zipWhen((items) -> Mono.just(items.stream()
.collect(Collectors.toMap(ItemQty::getId, ItemQty::getQty))))
.map((tuple) -> tuple.getT2())
.zipWhen((items) -> qtyRepository
.retrieveItemsQty(items.keySet()).collectList())
.zipWhen((tuple) -> reduceItemQty(tuple.getT2(), tuple.getT1(), itemId))
.flatMap((response) -> {
return Mono.just(itemId);
});
}
I tried switchIfEmpty and defaultIfEmpty like the below.
return itemRepository.findByd(itemId).collectList()
.switchIfEmpty(). /// Looks like the return statement is not allowed here.
.zipWhen((items) -> Mono.just(items.stream()
.collect(Collectors.toMap(ItemQty::getId, ItemQty::getQty))))
In case you want to keep the current flow, the easiest way would be to use filter
return itemRepository.findByd(itemId)
.collectList()
.filter(items -> !items.isEmpty())
...
but I would suggest to simplify the flow to make it more readable and don't overuse reactive operators where you don't really need them. For example, something like
return itemRepository.findByd(itemId)
.collectList()
.flatMap(items -> {
if (items.isEmpty()) {
return Mono.empty();
}
Map<Long, Integer> itemMap = items.stream()
.collect(Collectors.toMap(ItemQty::getId, ItemQty::getQty));
return retrieveItemsQty(itemMap.keySet())
.collectList()
.flatMap(availableQty -> reduceItemQty(availableQty, itemMap, itemId));
})
.then(Mono.just(itemId));
you could simplify even futher and move inner body into a separate method

how to merge the response of webClient call after calling 5 times and save the complete response in DB

i have scenario like:
i have to check the table if entry is available in DB then if available i need to call the same external api n times using webclient, collect all the response and save them in DB. if entry is not available in DB call the old flow.
here is my implementation. need suggestions to improve it. without for-each
public Mono<List<ResponseObject>> getdata(String id, Req obj) {
return isEntryInDB(id) //checking the entry in DB
.flatMap(
x -> {
final List<Mono<ResponseObject>> responseList = new ArrayList<>();
IntStream.range(0, obj.getQuantity()) // quantity decides how many times api call t happen
.forEach(
i -> {
Mono<ResponseObject> responseMono =
webClientCall(
id,
req.getType())
.map(
res ->
MapperForMappingDataToDesriedFormat(res));
responseList.add(responseMono);
});
return saveToDb(responseList);
})
.switchIfEmpty(oldFlow(id, req)); //if DB entry is not there take this existing flow.
need some suggestions to improve it without using foreach.
I would avoid using IntStream and rather use native operator to reactor called Flux in this case.
You can replace, InsStream.range with Flux.range. Something like this:
return isEntryPresent("123")
.flatMapMany(s -> Flux.range(0, obj.getQuantity())
.flatMap(this::callApi))
.collectList()
.flatMap(this::saveToDb)
.switchIfEmpty(Mono.defer(() ->oldFlow(id, req)));
private Mono<Object> saveToDb(List<String> stringList){
return Mono.just("done");
}
private Mono<String> callApi(int id) {
return Mono.just("iterating" + id);
}
private Mono<String> isEntryPresent(String id) {
return Mono.just("string");
}

How to access a KStreams Materialized State Store from another Stream Processor

I need to be able to remove a record from a Ktable from a separate Stream Processor. Today I'm using aggregate() and passing a materialized state store. In a separate processor that reads from a "termination" topic, I'd like to query that materialized state store either in a .transform() or a different .aggregate() and 'remove' that key/value. Every time I try to access the materialized state from a separate stream processor, it keeps telling me either the store isn't added to the topology, so then I add it and run it again, then it tells me it's already be registered and errors out.
builder.stream("topic1").map().groupByKey().aggregate(() -> null,
(aggKey, newValue, aggValue) -> {
//add to the Ktable
return newValue;
},
stateStoreMaterialized);
and in a separate stream I want to delete a key from that stateStoreMaterialized
builder.stream("topic2")
.transform(stateStoreDeleteTransformer, stateStoreSupplier.name())
stateStoreDeleteTransformer will query the key and delete it.
//in ctor
KeyValueBytesStoreSupplier stateStoreSupplier = Stores.persistentKeyValueStore("store1");
stateStoreMaterialized = Materialized.<String, MyObj>as(stateStoreSupplier)
.withKeySerde(Serdes.String())
.withValueSerde(mySerDe);
I don't have a terminal flag on my topic1 stream object value that can trigger a deletion. It has to come from another stream/topic.
When I try to use the same Materialized Store on two separate stream processors I get..
Invalid topology: Topic STATE_STORE-repartition has already been registered by another source.
at org.springframework.kafka.config.StreamsBuilderFactoryBean.start(StreamsBuilderFactoryBean.java:268)
Edit:
This is the 1st error I receive.
Caused by: org.apache.kafka.streams.errors.StreamsException: Processor KSTREAM-TRANSFORMVALUES-0000000012 has no access to StateStore store1 as the store is not connected to the processor. If you add stores manually via '.addStateStore()' make sure to connect the added store to the processor by providing the processor name to '.addStateStore()' or connect them via '.connectProcessorAndStateStores()'. DSL users need to provide the store name to '.process()', '.transform()', or '.transformValues()' to connect the store to the corresponding operator. If you do not add stores manually, please file a bug report at https://issues.apache.org/jira/projects/KAFKA.
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.getStateStore(ProcessorContextImpl.java:104)
at org.apache.kafka.streams.processor.internals.ForwardingDisabledProcessorContext.getStateStore(ForwardingDisabledProcessorContext.java:85)
So then I do this:
stateStoreSupplier = Stores.persistentKeyValueStore(STATE_STORE_NAME);
storeStoreBuilder = Stores.keyValueStoreBuilder(stateStoreSupplier, Serdes.String(), jsonSerDe);
stateStoreMaterialized = Materialized.as(stateStoreSupplier);
Then I get this error:
Caused by: org.apache.kafka.streams.errors.TopologyException: Invalid topology: StateStore 'state-store' is already added.
at org.apache.kafka.streams.processor.internals.InternalTopologyBuilder.addStateStore(InternalTopologyBuilder.java:520)
at org.apache.kafka.streams.processor.internals.InternalTopologyBuilder.addStateStore(InternalTopologyBuilder.java:512)
Here's the code that fixed my issue. As it turns out, order matters when building the streams. Had to set the materialized store first and then in subsequent lines of code, setup the transformer.
/**
* Create the streams using the KStreams DSL - a method to configure the stream and add any state stores.
*/
#Bean
public KafkaStreamsConfig setup() {
final JsonSerDe<Bus> ltaSerde = new JsonSerDe<>(Bus.class);
final StudentSerde<Student> StudentSerde = new StudentSerde<>();
//start lta stream
KStream<String, Bus> ltaStream = builder
.stream(ltaInputTopic, Consumed.with(Serdes.String(), ltaSerde));
final KStream<String, Student> statusStream = this.builder
.stream(this.locoStatusInputTopic,
Consumed.with(Serdes.String(),
StudentSerde));
//create lta store
KeyValueBytesStoreSupplier ltaStateStoreSupplier = Stores.persistentKeyValueStore(LTA_STATE_STORE_NAME);
final Materialized<String, Bus, KeyValueStore<Bytes, byte[]>> ltaStateStoreMaterialized =
Materialized.
<String, Bus>as(ltaStateStoreSupplier)
.withKeySerde(Serdes.String())
.withValueSerde(ltaSerde);
KTable<String, Bus> ltaStateProcessor = ltaStream
//map and convert lta stream into Loco / LTA key value pairs
.groupByKey(Grouped.with(Serdes.String(), ltaSerde))
.aggregate(
//The 'aggregate' and 'reduce' functions ignore messages with null values FYI.
// so if the value after the groupbykey produces a null value, it won't be removed from the state store.
//which is why it's very important to send a message with some terminal flag indicating this value should be removed from the store.
() -> null, /* initializer */
(aggKey, newValue, aggValue) -> {
if (null != newValue.getAssociationEndTime()) { //if there is an endTime associated to this train/loco then remove it from the ktable
logger.trace("removing LTA: {} loco from {} train", newValue.getLocoId(), newValue.getTrainAuthorization());
return null; //Returning null removes the record from the state store as well as its changelog topic. re: https://objectpartners.com/2019/07/31/slimming-down-your-kafka-streams-data/
}
logger.trace("adding LTA: {} loco from {} train", newValue.getLocoId(), newValue.getTrainAuthorization());
return newValue;
}, /* adder */
ltaStateStoreMaterialized
);
// don't need builder.addStateStore(keyValueStoreStoreBuilder); and CANT use it
// because the ltaStateStoreMaterialized will already be added to the topology in the KTable aggregate method above.
// The below transformer can use the state store because it's already added (apparently) by the aggregate method.
// Add the KTable processors first, then if there are any transformers that need to use the store, add them after the KTable aggregate method.
statusStream.map((k, v) -> new KeyValue<>(v.getLocoId(), v))
.transform(locoStatusTransformerSupplier, ltaStateStoreSupplier.name())
.to("testing.outputtopic", Produced.with(Serdes.String(), StudentSerde));
return this; //can return anything except for void.
}
is stateStoreMaterialized and stateStoreSupplier.name() has the same name?
Use have a error in your topology
KStream.transform(stateStoreDeleteTransformer, stateStoreSupplier.name())
You have to supply new instant of StateStoreDeleteTransformer per ProcessContext in TransformerSupplier, like this:
KStream.transform(StateStoreDeleteTransformer::new, stateStoreSupplier.name())
or
KStream.transform(() -> StateStoreDeleteTransformerSupplier.get(), stateStoreSupplier.name())//StateStoreDeleteTransformerSupplier return new instant of StateStoreDeleteTransformer
in stateStoreDeleteTransformer how do you intent on using stateStoreMaterialized inside transformer directly?
I have the similar use case and I using a KeyValueStore<String, MyObj>
public void init(ProcessorContext context) {
kvStore = (KeyValueStore<String, MyObj>) context.getStateStore("store1");
}

Non-blocking functional methods with Reactive Mongo and Web client

I have a micro service which reads objects from a database using a ReactiveMongoRepository interface.
The goal is to take each one of those objects and push it to a AWS Lambda function (after converting it to a DTO). If the result of that lambda function is in the 200 range, mark the object as being a success otherwise ignore.
In the old days of a simple Mongo Repository and a RestTemplate this is would be a trivial task. However I'm trying to understand this Reactive deal, and avoid blocking.
Here is the code I've come up with, I know I'm blocking on the webClient, but how do I avoid that?
#Override
public Flux<Video> index() {
return videoRepository.findAllByIndexedIsFalse().flatMap(video -> {
final SearchDTO searchDTO = SearchDTO.builder()
.name(video.getName())
.canonicalPath(video.getCanonicalPath())
.objectID(video.getObjectID())
.userId(video.getUserId())
.build();
// Blocking call
final HttpStatus httpStatus = webClient.post()
.uri(URI.create(LAMBDA_ENDPOINT))
.body(BodyInserters.fromObject(searchDTO)).exchange()
.block()
.statusCode();
if (httpStatus.is2xxSuccessful()) {
video.setIndexed(true);
}
return videoRepository.save(video);
});
}
I'm calling the above from a scheduled task, and I don't really care about that actual result of the index() method, just what happens during.
#Scheduled(fixedDelay = 60000)
public void indexTask() {
indexService
.index()
.log()
.subscribe();
}
I've read a bunch of blog posts etc on the subject but they're all just simple CRUD operations without anything happening in the middle so don't really give me a full picture of how to implement these things.
Any help?
Your solution is actually quite close.
In those cases, you should try and decompose the reactive chain in steps and not hesitate to turn bits into independent methods for clarity.
#Override
public Flux<Video> index() {
Flux<Video> unindexedVideos = videoRepository.findAllByIndexedIsFalse();
return unindexedVideos.flatMap(video -> {
final SearchDTO searchDTO = SearchDTO.builder()
.name(video.getName())
.canonicalPath(video.getCanonicalPath())
.objectID(video.getObjectID())
.userId(video.getUserId())
.build();
Mono<ClientResponse> indexedResponse = webClient.post()
.uri(URI.create(LAMBDA_ENDPOINT))
.body(BodyInserters.fromObject(searchDTO)).exchange()
.filter(res -> res.statusCode().is2xxSuccessful());
return indexedResponse.flatMap(response -> {
video.setIndexed(true);
return videoRepository.save(video);
});
});
my approach, maybe a little bit more readable. But I admit I didn't run it so not 100% guarantee that it will work.
public Flux<Video> index() {
return videoRepository.findAll()
.flatMap(this::callLambda)
.flatMap(videoRepository::save);
}
private Mono<Video> callLambda(final Video video) {
SearchDTO searchDTO = new SearchDTO(video);
return webClient.post()
.uri(URI.create(LAMBDA_ENDPOINT))
.body(BodyInserters.fromObject(searchDTO))
.exchange()
.map(ClientResponse::statusCode)
.filter(HttpStatus::is2xxSuccessful)
.map(t -> {
video.setIndexed(true);
return video;
});
}

Flux Concatenation does not return

I'm experimenting with Spring Boot 2.0, webflux and reactiv Mongo repositories. I have two variants of first deleting and then adding data to a collection. In the first variant the thread blocks until the deletion is finished, in the second variant the adding of data is concatenated to the deletion.
Variant A
#GetMapping("init")
public String init() {
Random rand = new Random();
Flux<Power> powers = Flux.range(0, 10000)
.map(i -> new Power(i,
LocalDateTime.now().toEpochSecond(ZoneOffset.of("+1")),
rand.nextDouble()));
powerRepository.deleteAll().block();
powerRepository.save(powers).blockLast();
return "ok";
}
Variant B
#GetMapping("init")
public String init() {
Random rand = new Random();
Flux<Power> powers = Flux.range(0, 10000)
.map(i -> new Power(i,
LocalDateTime.now().toEpochSecond(ZoneOffset.of("+1")),
rand.nextDouble()));
powerRepository.deleteAll()
.concatWith((v) -> powerRepository.save(powers)).blockLast();
return "ok";
}
Variant A returns, variant B not. What is the difference? What is the right way to combine two repository operations?
Chain using .then calls if nothing better. Avoid block calls and instead return Mono.just("ok").
public Mono<String> init() {
return repo.deleteAll()
.then(() -> repo.save(...))
.then(() -> Mono.just("ok"));
}
Make the endpoint return Mono.