I have created a spring Webflux application. In this application, we are trying to retrieve a record from the DB and then use that record to complete further processing.
I have wrapped my repository.get() method within a Mono like below:
find()
#NotNull
private Mono<RuntimeCallbackDetails> find(DealsFulfilmentCallback callback, DealsFulfilmentCallbackStatus status) {
String entityId = callback.getEntityId();
String itemId = callback.getItemId();
String type = getCallbackType(callback.getEntityId());
Long fulfilmentId = Optional.ofNullable(callback.getFId()).orElse(-1L);
return fulfilmentAcknowledgementService.findFulfilmentAcknowledgement(entityId, itemId, fulfilmentId, type)
.map(acknowledgement -> {
validationService.validate(acknowledgement, CALLBACK);
return RuntimeCallbackDetails.from(callback, acknowledgement, status);
})
.doOnError(throwable -> log.error("Error while retrieving fulfilment acknowledgement for callback: ", throwable))
.switchIfEmpty(Mono.defer(() -> Mono.just(RuntimeCallbackDetails.from(callback, status))));
}
FulfilmentAcknowledgementService.findFulfilmentAcknowledgement()
#org.jetbrains.annotations.NotNull
public Mono<FulfilmentAcknowledgement> findFulfilmentAcknowledgement(String entityId, String itemId, Long fulfilmentId, String type) {
return Mono.justOrEmpty(fulfilmentAcknowledgementRepository.findFulfilmentAcknowledgement(entityId, itemId, fulfilmentId, type));
}
Now the challenge here is that the flow does not even wait for the find method to execute and return the record. It moves on and since nothing is retrieved, an empty response is propagated downstream. How do I make this find method execute the query and return the Mono based on this retrieved element?
Related
I seem to be unable to store a simple object to cosmos db?
this is the database model.
public class HbModel
{
public Guid id { get; set; }
public string FormName { get; set; }
public Dictionary<string, object> Form { get; set; }
}
and this is how I store the data into the database
private static void SeedData(HbModelContext dbContext)
{
var cosmosClient = dbContext.Database.GetCosmosClient();
cosmosClient.ClientOptions.AllowBulkExecution = true;
if (dbContext.Set<HbModel>().FirstOrDefault() == null)
{
// No items could be picked hence try seeding.
var container = cosmosClient.GetContainer("hb", "hb_forms");
HbModel first = new HbModel()
{
Id = Guid.NewGuid(),//Guid.Parse(x["guid"] as string),
FormName = "asda",//x["name"] as string,
Form = new Dictionary<string, object>() //
}
string partitionKey = await GetPartitionKey(container.Database, container.Id);
var response = await container.CreateItemAsync(first, new PartitionKey(partitionKey));
}
else
{
Console.WriteLine("Already have data");
}
}
private static async Task<string> GetPartitionKey(Database database, string containerName)
{
var query = new QueryDefinition("select * from c where c.id = #id")
.WithParameter("#id", containerName);
using var iterator = database.GetContainerQueryIterator<ContainerProperties>(query);
while (iterator.HasMoreResults)
{
foreach (var container in await iterator.ReadNextAsync())
{
return container.PartitionKeyPath;
}
}
return null;
}
but when creating the item I get this error message
A host error has occurred during startup operation '3b06df1f-000c-4223-a374-ca1dc48d59d1'.
[2022-07-11T15:02:12.071Z] Microsoft.Azure.Cosmos.Client: Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 24bac0ba-f1f7-411f-bc57-3f91110c4528; Reason: ();.
Value cannot be null. (Parameter 'provider')
no idea why it fails?
the data should not be formatted incorreclty?
It also fails in case there is data in the dictionary.
What is going wrong?
There are several things wrong with the attached code.
You are enabling Bulk but you are not following the Bulk pattern
cosmosClient.ClientOptions.AllowBulkExecution = true is being set, but you are not parallelizing work. If you are going to use Bulk, make sure you are following the documentation and creating lists of concurrent Tasks. Reference: https://learn.microsoft.com/azure/cosmos-db/sql/tutorial-sql-api-dotnet-bulk-import#step-6-populate-a-list-of-concurrent-tasks. Otherwise don't use Bulk.
You are blocking threads.
The call to container.CreateItemAsync(first, new PartitionKey("/__partitionKey")).Result; is a blocking call, this can lead you to deadlocks. When using async operations (such as CreateItemAsync) please use the async/await pattern. Reference: https://github.com/davidfowl/AspNetCoreDiagnosticScenarios/blob/master/AsyncGuidance.md#avoid-using-taskresult-and-taskwait
The PartitionKey parameter should be the value not the definition.
On the call container.CreateItemAsync(first, new PartitionKey("/__partitionKey")) the Partition Key (second parameter) should be the value. Assuming your container has a Partition Key Definition of /__partitionKey then your documents should have a __partitionKey property and you should pass the Value in this parameter of such property in the current document. Reference: https://learn.microsoft.com/azure/cosmos-db/sql/troubleshoot-bad-request#wrong-partition-key-value
Optionally, if your documents do not contain such a value, just remove the parameter from the call:
container.CreateItemAsync(first)
Be advised though that this solution will not scale, you need to design your database with Partitioning in mind: https://learn.microsoft.com/azure/cosmos-db/partitioning-overview#choose-partitionkey
Missing id
The model has Id but Cosmos DB requires id, make sure the content of the document contains id when serialized.
I know pagination is somewhat against reactive principles, but due to requirements I have to make it work somehow. I'm using Spring Data 2.1.6 and I can't upgrade so ReactiveQuerydslSpecification for the dynamic query is out of the question. I figured I could use ReactiveMongoTemplate so I came up with this:
public interface IPersonRepository extends ReactiveMongoRepository<Person, String>, IPersonFilterRepository {
Flux<Person> findAllByCarId(String carId);
}
public interface IPersonFilterRepository {
Flux<Person> findAllByCarIdAndCreatedDateBetween(String carId, PersonStatus status,
OffsetDateTime from, OffsetDateTime to,
Pageable pageable);
}
#Repository
public class PersonFilterRepository implements IPersonFilterRepository {
#Autowired
private ReactiveMongoTemplate reactiveMongoTemplate;
#Override
public Flux<Person> findAllByCarIdAndCreatedDateBetween(String carId, PersonStatus status,
OffsetDateTime from, OffsetDateTime to,
Pageable pageable) {
Query query = new Query(Criteria.where("carId").is(carId));
if (status != null) {
query.addCriteria(Criteria.where("status").is(status));
}
OffsetDateTime maxLimit = OffsetDateTime.now(ZoneOffset.UTC).minusMonths(3).withDayOfMonth(1); // beginning of month
if (from == null || from.isBefore(maxLimit)) {
from = maxLimit;
}
query.addCriteria(Criteria.where("createdDateTime").gte(from));
if (to == null) {
to = OffsetDateTime.now(ZoneOffset.UTC);
}
query.addCriteria(Criteria.where("createdDateTime").lte(to));
// problem is trying to come up with a decent page-ish behavior compatible with Flux
/*return reactiveMongoTemplate.count(query, Person.class)
.flatMap(count -> reactiveMongoTemplate.find(query, Person.class)
.flatMap(p -> new PageImpl<Person>(p, pageable, count))
.collectList()
.map());*/
/* return reactiveMongoTemplate.find(query, Person.class)
.buffer(pageable.getPageSize(), pageable.getPageNumber() + 1)
//.elementAt(pageable.getPageNumber(), new ArrayList<>())
.thenMany(Flux::from);*/
}
I've tried to return a Page<Person> (assuming for once this single method could be non-reactive, for once) and it fails with the following error while running testing (Spring context does not load successfully due to: InvalidDataAccessApiUsageException: 'IDocumentFilterRepository.findAllByCustomerIdAndCreatedDateBetween' must not use sliced or paged execution. Please use Flux.buffer(size, skip). I've also tried returning Mono<Page<Person>> and then fails with "Method has to use a either multi-item reactive wrapper return type or a wrapped Page/Slice type. Offending method: 'IDocumentFilterRepository.findAllByCustomerIdAndCreatedDateBetween', so I guess my only option is returning a Flux, according to Example 133, snippet 3
Turns out you can just add the following to the query object:
query.with(pageable);
reactiveMongoTemplate.find(query, Person.class);
Return Flux<T> and it will work out of the box.
I am new to RxJava and finding it very useful for network and database processing within my Android applications.
I have two use cases that I cannot implement completely in RxJava
Use Case 1
Clear down my target database table Table A
Fetch a list of database records from Table B that contain a key field
For each row retrieved from Table B, call a Remote API and persist all the returned data into Table A
The closest I have managed is this code
final AtomicInteger id = new AtomicInteger(0);
DatabaseController.deleteAll(TableA_DO.class);
DatabaseController.fetchTable_Bs()
.subscribeOn(Schedulers.io())
.toObservable()
.flatMapIterable(b -> b)
.flatMap(b_record -> NetworkController.getTable_A_data(b_record.getKey()))
.flatMap(network -> transformNetwork(id, network, NETWORK_B_MAPPER))
.doOnNext(DatabaseController::persistRealmObjects)
.doOnComplete(onComplete)
.doOnError(onError)
.doAfterTerminate(doAfterTerminate())
.doOnSubscribe(compositeDisposable::add)
.subscribe();
Use Case 2
Clear down my target database table Table X
Clear down my target database table Table Y
Fetch a list of database records from Table Z that contain a key field
For each row retrieved from Table B, call a Remote API and persist some of the returned data into Table X the remainder of the data should be persisted into table Y
I have not managed to create any code for use case 2.
I have a number of questions regarding the use of RxJava for these use cases.
Is it possible to achieve both my use cases in RxJava?
Is it "Best Practice" to combine all these steps into an Rx "Stream"
UPDATE
I ended up with this POC test code which seems to work...
I am not sure if its the optimum solution however My API calls return Single and my database operations return Completable so I feel like this is the best solution for me.
public class UseCaseOneA {
public static void main(final String[] args) {
login()
.andThen(UseCaseOneA.deleteDatabaseTableA())
.andThen(UseCaseOneA.deleteDatabaseTableB())
.andThen(manufactureRecords())
.flatMapIterable(x -> x)
.flatMapSingle(record -> NetworkController.callApi(record.getPrimaryKey()))
.flatMapSingle(z -> transform(z))
.flatMapCompletable(p -> UseCaseOneA.insertDatabaseTableA(p))
.doOnComplete(() -> System.out.println("ON COMPLETE"))
.doFinally(() -> System.out.println("ON FINALLY"))
.subscribe();
}
private static Single<List<PayloadDO>> transform(final List<RemotePayload> payloads) {
return Single.create(new SingleOnSubscribe<List<PayloadDO>>() {
#Override
public void subscribe(final SingleEmitter<List<PayloadDO>> emitter) throws Exception {
System.out.println("transform - " + payloads.size());
final List<PayloadDO> payloadDOs = new ArrayList<>();
for (final RemotePayload remotePayload : payloads) {
payloadDOs.add(new PayloadDO(remotePayload.getPayload()));
}
emitter.onSuccess(payloadDOs);
}
});
}
private static Observable<List<Record>> manufactureRecords() {
final List<Record> records = new ArrayList<>();
records.add(new Record("111-111-111"));
records.add(new Record("222-222-222"));
records.add(new Record("3333-3333-3333"));
records.add(new Record("44-444-44444-44-4"));
records.add(new Record("5555-55-55-5-55-5555-5555"));
return Observable.just(records);
}
private static Completable deleteDatabaseTableA() {
return Completable.create(new CompletableOnSubscribe() {
#Override
public void subscribe(final CompletableEmitter emitter) throws Exception {
System.out.println("deleteDatabaseTableA");
emitter.onComplete();
}
});
}
private static Completable deleteDatabaseTableB() {
return Completable.create(new CompletableOnSubscribe() {
#Override
public void subscribe(final CompletableEmitter emitter) throws Exception {
System.out.println("deleteDatabaseTableB");
emitter.onComplete();
}
});
}
private static Completable insertDatabaseTableA(final List<PayloadDO> payloadDOs) {
return Completable.create(new CompletableOnSubscribe() {
#Override
public void subscribe(final CompletableEmitter emitter) throws Exception {
System.out.println("insertDatabaseTableA - " + payloadDOs);
emitter.onComplete();
}
});
}
private static Completable login() {
return Completable.complete();
}
}
This code doesn't address all my use case requirements. Namely being able to transform the remote payload records into multiple Database record types and insert each type into its own specific target databased table.
I could just call the Remote API twice to get the same remote data items and transform first into one database type then secondly into the second database type, however that seems wasteful.
Is there an operand in RxJava where I can reuse the output from my API calls and transform them into another database type?
You have to index the items yourself in some manner, for example, via external counting:
Observable.defer(() -> {
AtomicInteger counter = new AtomicInteger();
return DatabaseController.fetchTable_Bs()
.subscribeOn(Schedulers.io())
.toObservable()
.flatMapIterable(b -> b)
.doOnNext(item -> {
if (counter.getAndIncrement() == 0) {
// this is the very first item
} else {
// these are the subsequent items
}
});
});
The defer is necessary to isolate the counter to the inner sequence so that repetition still works if necessary.
Insert query
#Insert(onConflict = OnConflictStrategy.REPLACE)
long insertProduct(Product product); //product id is auto generated
view model
public Completable insertProduct(final String productName) {
return new CompletableFromAction(() -> {
Product newProduct = new Product();
newProduct.setProductName(productName);
mProductDataSource.insertOrUpdateProduct(newProduct);
});
}
In activity where I called this above function I used CompositeDisposable.
CompositeDisposable mDisposable = new CompositeDisposable();
mDisposable.add(mViewModel.insertProduct(productName))
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(() ->{} ,throwable -> Log.e(TAG, "Error msg", throwable)));
Am I implementing in wrong way?
According to docs. If the #Insert method receives only 1 parameter, it can return a long, which is the new rowId for the inserted item. If the parameter is an array or a collection, it should return long[] or List instead.
Since you insert only one item, the method will return only one rowID.
So, try this
Single.fromCallable(new Callable<Long>() {
#Override
public Long call() throws Exception {
return productDao.insertProduct(new Product()));
}
})
.subscribe(id -> {
} ,throwable -> Log.e(TAG, "Error msg", throwable)))
You could use Observable or Maybe as well. But I think Single fits better since in your case the id is autogenerated and the insertion should always complete.
I am working on a JPA/Jersey web app and want to know if there is a better way to update a record. Currently, I am doing:
#PUT
#Path("update/{id}")
#Produces("application/json")
#Consumes("application/x-www-form-urlencoded")
public Response createDevice(
#PathParam("id") int id,
#FormParam("name") String name,
#FormParam("type") int type
/*MultivaluedMap<String, String> formParams*/
) {
try {
Devices newDevice = entityManager.find(Devices.class, id);
if(name==null){name=newDevice.getName();}
if(type != newDevice.getType()){newDevice.setType(type);}
newDevice.setName(name);
//newDevice.setType(type);
entityManager.merge(newDevice);
return Response.status(201).entity(new ResponseObject("success")).build();
} finally {
entityManager.close();
}
}
This works, but if my Devices table had more fields, I would have to check for equality of ALL fields with the values on the original object to see if they've changed, so that my
entityManager.merge(newDevice);
will only change the values passed in.
Is there a better way to do this?