want to execute elements in the flux in asynchronously on different threads.
but its not executing them on different threads. am i missing something?
below is the code.
public Mono<Map<Object, Object>> execute(List<Empolyee> empolyeeList) {
return Flux.fromIterable(empolyeeList).subscribeOn(elastic(), true).flatMap(empolyee -> {
return empolyeeService.getDepts(empolyee).flatMap(result -> {
// ---
// ---
// ---
return Mono.just(result);
});
}).collectMap(result -> result.getName().trim(), result -> fieldResult.getValue());
}
taken from the documentation
subscribeOn applies to the subscription process, when that backward
chain is constructed. As a consequence, no matter where you place the
subscribeOn in the chain, it always affects the context of the source
emission.
It does not work as you think. It applies to when someone subscribes. Their entire request will be placed on it's own tread. So there is an absolute guarantee that no two requests will end up on the same thread.
The subscribeOn method
Made the flux as parallel flux and used runOn(elastic()). its working as expected
//Making flux as parallel flux, we can also use ParallelFlux instead of below
Flux.fromIterable(empolyeeList).parallel()
//running on elastic scheduler
.runOn(elastic()).flatMap(empolyee -> {
}
Related
I'm new to Blazor and trying to make a page with several separate components to handle a massive form. Each individual component covers a part of the form.
The problem I'm facing is that each of my components needs access to data from the back-end, and not every component uses the same data. When the page loads, each components makes an attempt to fetch data from the server, which causes a problem with Entity Framework.
A second operation started on this context before a previous operation
completed. This is usually caused by different threads using the same
instance of DbContext.
This is obviously caused by the fact that my components are initialized at the same time, and all make their attempt to load the data simultaneously. I was under the impression that the way DI is set up in Blazor, this wouldn't be a problem, but it is.
Here are the components in my template:
<CascadingValue Value="this">
<!-- BASE DATA -->
<CharacterBaseDataView />
<!-- SPECIAL RULES -->
<CharacterSpecialRulesView />
</CascadingValue>
Here is how my components are initialized:
protected async override Task OnInitializedAsync()
{
CharacterDetailsContext = new EditContext(PlayerCharacter);
await LoadCharacterAsync();
}
private async Task LoadCharacterAsync()
{
PlayerCharacter = await PlayerCharacterService.GetPlayerCharacterAsync(ViewBase.CharacterId.Value);
CharacterDetailsContext = new EditContext(PlayerCharacter);
}
When two components with the above code are in the same view, the mentioned error occurs. I thread using the synchronous version "OnInitialized()" and simply discarding the task, but that didn't fix the error.
Is there some other way to call the data so that this issue doesn't occur? Or am I going about this the wrong way?
You've hit a common problem in using async operations in EF - two or more operations trying to use the same context at once.
Take a look at the MS Docs article about EF DBContexts - there's a section further down specific to Blazor. It explains the use of a DbContextFactory and CreateDbContext to create contexts for units-of-work i.e. one context per operation so two async operations each have a separate context.
Initially to solve the threading issues, I used DbContextFactory to create contexts for each operation - however this resulted in database in-consistency issues across components, and I realised I need change tracking across components.
Therefore instead, I keep my DbContext as scoped, and I don't create a new context before each operation.
I then adapted my OnInitializedAsync() methods to check if the calls to the database have completed, before making these calls through my injected services. This works really well for my app:
#code {
static Semaphore semaphore;
//code ommitted for brevity
protected override async Task OnInitializedAsync()
{
try
{
//First open global semaphore
semaphore = Semaphore.OpenExisting("GlobalSemaphore");
while (!semaphore.WaitOne(TimeSpan.FromTicks(1)))
{
await Task.Delay(TimeSpan.FromSeconds(1));
}
//If while loop is exited or skipped, previous service calls are completed.
ApplicationUsers = await ApplicationUserService.Get();
}
finally
{
try
{
semaphore.Release();
}
catch (Exception ex)
{
Console.WriteLine("ex.Message");
}
}
}
I want to trigger longer running operation via rest request and WebFlux. The result of a call should just return an info that operation has started. The long running operation I want to run on different scheduler (e.g. Schedulers.single()). To achieve that I used subscribeOn:
Mono<RecalculationRequested> recalculateAll() {
return provider.size()
.doOnNext(size -> log.info("Size: {}", size))
.doOnNext(size -> recalculate(size))
.map(RecalculationRequested::new);
}
private void recalculate(int toRecalculateSize) {
Mono.just(toRecalculateSize)
.flatMapMany(this::toPages)
.flatMap(page -> recalculate(page))
.reduce(new RecalculationResult(), RecalculationResult::increment)
.subscribeOn(Schedulers.single())
.subscribe(result -> log.info("Result of recalculation - success:{}, failed: {}",
result.getSuccess(), result.getFailed()));
}
private Mono<RecalculationResult> recalculate(RecalculationPage pageToRecalculate) {
return provider.findElementsToRecalculate(pageToRecalculate.getPageNumber(), pageToRecalculate.getPageSize())
.flatMap(this::recalculateSingle)
.reduce(new RecalculationResult(), RecalculationResult::increment);
}
private Mono<RecalculationResult> recalculateSingle(ElementToRecalculate elementToRecalculate) {
return recalculationTrigger.recalculate(elementToRecalculate)
.doOnNext(result -> {
log.info("Finished recalculation for element: {}", elementToRecalculate);
})
.doOnError(error -> {
log.error("Error during recalculation for element: {}", elementToRecalculate, error);
});
}
From the above I want to call:
private void recalculate(int toRecalculateSize)
in a different thread. However, it does not run on a single thread pool - it uses a different thread pool. I would expect subscribeOn change it for the whole chain. What should I change and why to execute it in a single thread pool?
Just to mention - method:
provider.findElementsToRecalculate(...)
uses WebClient to get elements.
One caveat of subscribeOn is it does what it says: it runs the act of "subscribing" on the provided Scheduler. Subscribing flows from bottom to top (the Subscriber subscribes to its parent Publisher), at runtime.
Usually you see in documentation and presentations that subscribeOn affects the whole chain. That is because most operators / sources will not themselves change threads, and by default will start sending onNext/onComplete/onError signals from the thread from which they were subscribed to.
But as soon as one operator switches threads in that top-to-bottom data path, the reach of subscribeOn stops there. Typical example is when there is a publishOn in the chain.
The source of data in this case is reactor-netty and netty, which operate on their own threads and thus act as if there was a publishOn at the source.
For WebFlux, I'd say favor using publishOn in the main chain of operators, or alternatively use subscribeOn inside of inner chains, like inside flatMap.
As per the documentation , all operators prefixed with doOn , are sometimes referred to as having a “side-effect”. They let you peek inside the sequence’s events without modifying them.
If you want to chain the 'recalculate' step after 'provider.size()' do it with flatMap.
I am having a Mono object, On which I have subscribed for doOnsuccess, In this method again I am saving the data in DB(CouchBase Using ReactiveCouchbaseRepository). after that, I am not getting any logs for Line1 and line2.
But this is working fine if I do not save this object, means I am getting logs for line 2.
Mono<User> result = context.getPayload(User.class);
result.doOnSuccess( user -> {
System.out.println("############I got the user"+user);
userRepository.save(user).doOnSuccess(user2->{
System.out.println("user saved"); // LINE 1
}).subscribe();
System.out.println("############"+user); // LINE2
}).subscribe();
Your code snippet is breaking a few rules you should follow closely:
You should not call subscribe from within a method/lambda that returns a reactive type such as Mono or Flux; this will decouple the execution from the main task while they'll both still operate on that shared state. This often ends up on issues because things are trying to read twice the same stream. It's a bit like you're trying to create two separate threads that try reading on the same outputstream.
you should not do I/O operations in doOnXYZ operators. Those are "side-effects" operators, meaning they are useful for logging, increment counters.
What you should try instead to chain Reactor operators to create a single reactive pipeline and return the reactive type for the final client to subscribe on it. In a Spring WebFlux application, the HTTP clients (through the WebFlux engine) are subscribing.
Your code snippet could look like this:
Mono<User> result = context.getPayload(User.class)
.doOnSuccess(user -> System.out.println("############Received user "+user))
.flatMap(user -> {return userRepository.save(user)})
.doOnSuccess(user -> System.out.println("############ Saved "+user));
return result;
I'm trying to batch the records constantly emitted from a streaming source (Kafka) and call my service in a batch of 100.
What I get as the input is a single record. I'm trying what's the best way to achieve it in the Reactive way using Spring Reactor without having to have a mutation and locking outside the pipeline.
Here is my naive attempt which simply reflects my sequential way of thinking:
Mono.just(input)
.subscribe(i -> {
batches.add(input);
if(batches.size() >= 100) {
// Invoke another reactive pipeline.
// Clear the batch (requires locking in order to be thread safe).
}
});
What's the best way to achieve batching on a streaming source using reactor.
.buffer(100) or bufferTimeout(100, Duration.ofSeconds(xxx) comes to the rescue
Using Flux.buffer or Flux.bufferTimeout you will be capable of gathering the fixed amount of elements into the List
StepVerifier.create(
Flux.range(0, 1000)
.buffer(100)
)
.expectNextCount(10)
.expectComplete()
.verify()
Update for the use case
In case, when the input is a single value, suppose like an invocation of the method with parameter:
public void invokeMe(String element);
You may adopt UnicastProcessor technique and transfer all data to that processor so then it will take care of batching
class Batcher {
final UnicastProcessor processor = UnicastProcessor.create();
public void invokeMe(String element) {
processor.sink().next(element);
// or Mono.just(element).subscribe(processor);
}
public Flux<List<String>> listen() {
return processor.bufferTimeout(100, Duration.ofSeconds(5));
}
}
Batcher batcher = new Batcher();
StepVerifier.create(
batcher.listen()
)
.then(() -> Flux.range(0, 1000)
.subscribe(i -> batcher.invokeMe("" + i)))
.expectNextCount(10)
.thenCancel()
.verify()
From that example, we might learn how to provide a single point of receiving events and then listen to results of the batching process.
Please note that UnicastPorcessor allows only one subscriber, so it will be useful for the model when there is one interested party in batching results and many data producers. In a case when you have subscribers as many as producers you may want to use one of the next processors -> DirectProcessor, TopicProcessor, WorkerQueueProcessor. To learn more about Reactor Processors follow the link
Problem
I am trying to load all of the data my application needs before using. Once I have all of the data I want to emit one event back to my subscription in my controller with either a success or failure status.
I am struggling to come up with a way to send an onComplete status on my observable chain so I can use toArray(). Maybe there's a way or just a better approach that I have not thought of.
Loading Workflow
Basically I have three different services: version, champion, and skin.
I retrieve the latest version and get all of the champions for that version. Then I download all of the images for each skin the champion has.
Observable Chain
My observable chain in my LoadingViewModel is something like this:
return versionService.getLatest().flatMap({ (version: VersionData) in
return championService.getChampions(forVersion: version)
}).flatMap({ (champions: [Champion]) -> Observable<Champion> in
// Go through champions and update total skin count
// emit each champion
}).flatMap({ (champion: Champion) -> Observable<Skin> in
return skinService.getSkin(forChampion: champion)
}).toArray() // Doesn't work since I don't send complete
.flatMap({ (result: [Skin]) -> Observable<LoadingViewModelResult> in
return Observable.just(LoadingViewModelResult.success)
})
If I don't have the toArray() there then I emit a LoadingViewModelResult.success for each skin which floods my view controller with an abundance of unwanted successful results.
What I've Tried
I attempted to use .take(count) where I passed in the skin count but since it is initialized to 0 the chain will instantly take 0 and return.
For my [Champion] -> Observable<Champion> and Champion -> Observable<Skin> observables I needed to include an observer.onCompleted within the creation of the Observable.