how to merge the response of webClient call after calling 5 times and save the complete response in DB - reactive-programming

i have scenario like:
i have to check the table if entry is available in DB then if available i need to call the same external api n times using webclient, collect all the response and save them in DB. if entry is not available in DB call the old flow.
here is my implementation. need suggestions to improve it. without for-each
public Mono<List<ResponseObject>> getdata(String id, Req obj) {
return isEntryInDB(id) //checking the entry in DB
.flatMap(
x -> {
final List<Mono<ResponseObject>> responseList = new ArrayList<>();
IntStream.range(0, obj.getQuantity()) // quantity decides how many times api call t happen
.forEach(
i -> {
Mono<ResponseObject> responseMono =
webClientCall(
id,
req.getType())
.map(
res ->
MapperForMappingDataToDesriedFormat(res));
responseList.add(responseMono);
});
return saveToDb(responseList);
})
.switchIfEmpty(oldFlow(id, req)); //if DB entry is not there take this existing flow.
need some suggestions to improve it without using foreach.

I would avoid using IntStream and rather use native operator to reactor called Flux in this case.
You can replace, InsStream.range with Flux.range. Something like this:
return isEntryPresent("123")
.flatMapMany(s -> Flux.range(0, obj.getQuantity())
.flatMap(this::callApi))
.collectList()
.flatMap(this::saveToDb)
.switchIfEmpty(Mono.defer(() ->oldFlow(id, req)));
private Mono<Object> saveToDb(List<String> stringList){
return Mono.just("done");
}
private Mono<String> callApi(int id) {
return Mono.just("iterating" + id);
}
private Mono<String> isEntryPresent(String id) {
return Mono.just("string");
}

Related

Reactive programming - Return response when the result is empty

I have a reactive code below which does the following.
query items table by ID - (itemRepository.findByd(itemId))
Create a map from the result from Step1
Invoke retrieveItemsQty from DB
Here, it works fine when the Step1 returns one or more result and retrieveItemsQty method fails when the result is empty. My requirement is to return back when the first step result(itemRepository.findByd) is empty. Not sure how to do this?
private Mono<Long> updateItemsQty(final Long itemId) {
return itemRepository.findByd(itemId).collectList()
.zipWhen((items) -> Mono.just(items.stream()
.collect(Collectors.toMap(ItemQty::getId, ItemQty::getQty))))
.map((tuple) -> tuple.getT2())
.zipWhen((items) -> qtyRepository
.retrieveItemsQty(items.keySet()).collectList())
.zipWhen((tuple) -> reduceItemQty(tuple.getT2(), tuple.getT1(), itemId))
.flatMap((response) -> {
return Mono.just(itemId);
});
}
I tried switchIfEmpty and defaultIfEmpty like the below.
return itemRepository.findByd(itemId).collectList()
.switchIfEmpty(). /// Looks like the return statement is not allowed here.
.zipWhen((items) -> Mono.just(items.stream()
.collect(Collectors.toMap(ItemQty::getId, ItemQty::getQty))))
In case you want to keep the current flow, the easiest way would be to use filter
return itemRepository.findByd(itemId)
.collectList()
.filter(items -> !items.isEmpty())
...
but I would suggest to simplify the flow to make it more readable and don't overuse reactive operators where you don't really need them. For example, something like
return itemRepository.findByd(itemId)
.collectList()
.flatMap(items -> {
if (items.isEmpty()) {
return Mono.empty();
}
Map<Long, Integer> itemMap = items.stream()
.collect(Collectors.toMap(ItemQty::getId, ItemQty::getQty));
return retrieveItemsQty(itemMap.keySet())
.collectList()
.flatMap(availableQty -> reduceItemQty(availableQty, itemMap, itemId));
})
.then(Mono.just(itemId));
you could simplify even futher and move inner body into a separate method

Non-blocking functional methods with Reactive Mongo and Web client

I have a micro service which reads objects from a database using a ReactiveMongoRepository interface.
The goal is to take each one of those objects and push it to a AWS Lambda function (after converting it to a DTO). If the result of that lambda function is in the 200 range, mark the object as being a success otherwise ignore.
In the old days of a simple Mongo Repository and a RestTemplate this is would be a trivial task. However I'm trying to understand this Reactive deal, and avoid blocking.
Here is the code I've come up with, I know I'm blocking on the webClient, but how do I avoid that?
#Override
public Flux<Video> index() {
return videoRepository.findAllByIndexedIsFalse().flatMap(video -> {
final SearchDTO searchDTO = SearchDTO.builder()
.name(video.getName())
.canonicalPath(video.getCanonicalPath())
.objectID(video.getObjectID())
.userId(video.getUserId())
.build();
// Blocking call
final HttpStatus httpStatus = webClient.post()
.uri(URI.create(LAMBDA_ENDPOINT))
.body(BodyInserters.fromObject(searchDTO)).exchange()
.block()
.statusCode();
if (httpStatus.is2xxSuccessful()) {
video.setIndexed(true);
}
return videoRepository.save(video);
});
}
I'm calling the above from a scheduled task, and I don't really care about that actual result of the index() method, just what happens during.
#Scheduled(fixedDelay = 60000)
public void indexTask() {
indexService
.index()
.log()
.subscribe();
}
I've read a bunch of blog posts etc on the subject but they're all just simple CRUD operations without anything happening in the middle so don't really give me a full picture of how to implement these things.
Any help?
Your solution is actually quite close.
In those cases, you should try and decompose the reactive chain in steps and not hesitate to turn bits into independent methods for clarity.
#Override
public Flux<Video> index() {
Flux<Video> unindexedVideos = videoRepository.findAllByIndexedIsFalse();
return unindexedVideos.flatMap(video -> {
final SearchDTO searchDTO = SearchDTO.builder()
.name(video.getName())
.canonicalPath(video.getCanonicalPath())
.objectID(video.getObjectID())
.userId(video.getUserId())
.build();
Mono<ClientResponse> indexedResponse = webClient.post()
.uri(URI.create(LAMBDA_ENDPOINT))
.body(BodyInserters.fromObject(searchDTO)).exchange()
.filter(res -> res.statusCode().is2xxSuccessful());
return indexedResponse.flatMap(response -> {
video.setIndexed(true);
return videoRepository.save(video);
});
});
my approach, maybe a little bit more readable. But I admit I didn't run it so not 100% guarantee that it will work.
public Flux<Video> index() {
return videoRepository.findAll()
.flatMap(this::callLambda)
.flatMap(videoRepository::save);
}
private Mono<Video> callLambda(final Video video) {
SearchDTO searchDTO = new SearchDTO(video);
return webClient.post()
.uri(URI.create(LAMBDA_ENDPOINT))
.body(BodyInserters.fromObject(searchDTO))
.exchange()
.map(ClientResponse::statusCode)
.filter(HttpStatus::is2xxSuccessful)
.map(t -> {
video.setIndexed(true);
return video;
});
}

How to process all items in List then complete with RxJava

I am investigating the use of RxJava in my current Android application.
I am stuck with the following use case.
For each data row on a particular database table I wish to action an HTTP POST call, once all the POST(s) have completed OK I need to clear down the database table.
The code I have is as follows:-
login()
.andThen(Single.defer(() -> DatabaseController.fetchSingleRealmObjects(UpdateDO.class)))
.toObservable()
.flatMapIterable(update -> update)
.flatMap(this::parameteriseUpdate)
.doOnNext(NetworkController::update)
.doOnComplete(() -> DatabaseController.deleteAll(UpdateDO.class))
.ignoreElements()
.retryWhen(errors -> errors.flatMap(e -> constructRetryHandler(retryCounter, e)))
.doOnComplete(onComplete)
.doOnError(onError)
.doAfterTerminate(doAfterTerminate())
.doOnSubscribe(compositeDisposable::add)
.blockingAwait();
When the UpdateDO table is empty the above code completes as expected.
However when data rows exist the process "sticks" in the doOnNext()
I realise this is because I only call emitter.onNext()
private ObservableSource<Map<String, Object>> parameteriseUpdate(final UpdateDO updateDO) {
final Map<String, Object> fields = new HashMap<>();
fields.put(FIELD_NAME_DRUG_ID, updateDO.getDrugId());
fields.put(FIELD_NAME_STORE_CONTENT_ID, updateDO.getStoreContentId());
fields.put(FIELD_NAME_STORE_ID, updateDO.getStoreID());
fields.put(FIELD_NAME_ACTUAL_QUANTITY, updateDO.getActualQty());
fields.put(FIELD_NAME_VARIANCE, updateDO.getUnitQty());
fields.put(FIELD_NAME_REMARKS, updateDO.getRemarks());
fields.put(FIELD_NAME_CREATED_BY, updateDO.getCreatedBy());
return Observable.create(emitter -> emitter.onNext(fields));
}
I cannot see how to fix this though, how to I refactor my code to allow me to have emitter.onComplete() called?
I think that is better to change that function from returning ObservableSource<Map<String, Object>>
to simple function like this
private Map<String, Object> parameteriseUpdate(final UpdateDO updateDO) {
final Map<String, Object> fields = new HashMap<>();
fields.put(FIELD_NAME_DRUG_ID, updateDO.getDrugId());
fields.put(FIELD_NAME_STORE_CONTENT_ID, updateDO.getStoreContentId());
fields.put(FIELD_NAME_STORE_ID, updateDO.getStoreID());
fields.put(FIELD_NAME_ACTUAL_QUANTITY, updateDO.getActualQty());
fields.put(FIELD_NAME_VARIANCE, updateDO.getUnitQty());
fields.put(FIELD_NAME_REMARKS, updateDO.getRemarks());
fields.put(FIELD_NAME_CREATED_BY, updateDO.getCreatedBy());
return fields;
}
and call it as a map not flatmap like this :
.map(this::parameteriseUpdate)
because in your case you are creating many streams that never complete.

How to do Async Http Call with Apache Beam (Java)?

Input PCollection is http requests, which is a bounded dataset. I want to make async http call (Java) in a ParDo , parse response and put results into output PCollection. My code is below. Getting exception as following.
I cound't figure out the reason. need a guide....
java.util.concurrent.CompletionException: java.lang.IllegalStateException: Can't add element ValueInGlobalWindow{value=streaming.mapserver.backfill.EnrichedPoint#2c59e, pane=PaneInfo.NO_FIRING} to committed bundle in PCollection Call Map Server With Rate Throttle/ParMultiDo(ProcessRequests).output [PCollection]
Code:
public class ProcessRequestsFn extends DoFn<PreparedRequest,EnrichedPoint> {
private static AsyncHttpClient _HttpClientAsync;
private static ExecutorService _ExecutorService;
static{
AsyncHttpClientConfig cg = config()
.setKeepAlive(true)
.setDisableHttpsEndpointIdentificationAlgorithm(true)
.setUseInsecureTrustManager(true)
.addRequestFilter(new RateLimitedThrottleRequestFilter(100,1000))
.build();
_HttpClientAsync = asyncHttpClient(cg);
_ExecutorService = Executors.newCachedThreadPool();
}
#DoFn.ProcessElement
public void processElement(ProcessContext c) {
PreparedRequest request = c.element();
if(request == null)
return;
_HttpClientAsync.prepareGet((request.getRequest()))
.execute()
.toCompletableFuture()
.thenApply(response -> { if(response.getStatusCode() == HttpStatusCodes.STATUS_CODE_OK){
return response.getResponseBody();
} return null; } )
.thenApply(responseBody->
{
List<EnrichedPoint> resList = new ArrayList<>();
/*some process logic here*/
System.out.printf("%d enriched points back\n", result.length());
}
return resList;
})
.thenAccept(resList -> {
for (EnrichedPoint enrichedPoint : resList) {
c.output(enrichedPoint);
}
})
.exceptionally(ex->{
System.out.println(ex);
return null;
});
}
}
The Scio library implements a DoFn which deals with asynchronous operations. The BaseAsyncDoFn might provide you the handling you need. Since you're dealing with CompletableFuture also take a look at the JavaAsyncDoFn.
Please note that you necessarily don't need to use the Scio library, but you can take the main idea of the BaseAsyncDoFn since it's independent of the rest of the Scio library.
The issue that your hitting is that your outputting outside the context of a processElement or finishBundle call.
You'll want to gather all your outputs in memory and output them eagerly during future processElement calls and at the end within finishBundle by blocking till all your calls finish.

Flux Concatenation does not return

I'm experimenting with Spring Boot 2.0, webflux and reactiv Mongo repositories. I have two variants of first deleting and then adding data to a collection. In the first variant the thread blocks until the deletion is finished, in the second variant the adding of data is concatenated to the deletion.
Variant A
#GetMapping("init")
public String init() {
Random rand = new Random();
Flux<Power> powers = Flux.range(0, 10000)
.map(i -> new Power(i,
LocalDateTime.now().toEpochSecond(ZoneOffset.of("+1")),
rand.nextDouble()));
powerRepository.deleteAll().block();
powerRepository.save(powers).blockLast();
return "ok";
}
Variant B
#GetMapping("init")
public String init() {
Random rand = new Random();
Flux<Power> powers = Flux.range(0, 10000)
.map(i -> new Power(i,
LocalDateTime.now().toEpochSecond(ZoneOffset.of("+1")),
rand.nextDouble()));
powerRepository.deleteAll()
.concatWith((v) -> powerRepository.save(powers)).blockLast();
return "ok";
}
Variant A returns, variant B not. What is the difference? What is the right way to combine two repository operations?
Chain using .then calls if nothing better. Avoid block calls and instead return Mono.just("ok").
public Mono<String> init() {
return repo.deleteAll()
.then(() -> repo.save(...))
.then(() -> Mono.just("ok"));
}
Make the endpoint return Mono.