Am I right to assume that "map" could essentially be a "subscribe" with a return type . They both seem to get called asynchronously when the promise gets resolved ?
For example , if I dispatch a list of 3 Async calls concurrently, would applying the map operation in manner below be blocking ?
Flux.merge(albums.stream().map(album -> {
Mono<CoverResponse> responseMono = clientRequestHandler.makeAsyncCall()
//2.call and handler for async call
return responseMono
.map(response -> processResponse());
}).collect(Collectors.toList())).then(Mono.just(monoResponse));
In the above snippet, is each map operation going to be blocking? If say, the first call takes 5 ms to to return and every other call takes 2 ms to return , are we going to wait 3ms+2ms+2m = 7ms for the enitre operation ? or just 3ms since once the first call gets resolved , the 2ms calls are already resolved by then.
First of all, nothing will happen until someone subscribes. Subscribe is the last thing in the chain, that will trigger the start of all the events.
Second of all you need to understand the difference of running something in parallell vs running something non-blocking.
to resolve your first map, it needs to make the rest call, then with the response it needs to do your second map. These two wont be run in parallell.
Your responseMono.map can't be run until Mono<Response> responseMono actually has something in it. Think of it as a Promise that will signal the application when it has been resolved.
Or you can think of it as a chain of callbacks.
So in your example you are doing a clientRequestHandler.makeAsyncCall() but you are returning a Mono<CoverResponse> the next part the responseMono.map wont trigger, until there is a CoverResponse in the mono. So your "async" call, is probably async, but still adheres to list order since its all in a sequential stream.
But map is a mapping function. It takes something out of the box, performs a computation on something in the box and then returns the new value or type.
What makes reactive better than other options is that when you do your side effect the remote call somewhere that takes time, the thread that is processing this wont hang around and wait for the external request to finish, it will start doing other things, like processing other requests.
Then when the Mono<Response> signals the system that there is "something in the box" a response, then the same thread or any other available thread will keep on processing the request.
Related
I am working on OpenCL implementation wherein the host side particular function has to call every time the clEnqueueReadBuffer is done executing.
I am calling the kernels in a loop. It will look like below in an ordered queue.
clEnqueueNDRangeKernel() -> clEnqueueReadBuffer(&Event) ->
clEnqueueNDRangeKernel() -> clEnqueueReadBuffer(&Event) .......
I have used clSetEventCall() to register Events in each read command to execute a callback function. I have observed that, though the command queue is an in-order queue, the order of the callback function does not execute in-order.
Also, in OpenCL 1.2, it has a mention as below.
The order in which the registered user callback functions are called
is undefined. There is no guarantee that the callback functions
registered for various execution status values for an event will be
called in the exact order that the execution status of a command
changes.
Can anyone give me a solution? I want to execute the callback function in order.
A simple solution could be to subscribe the same callback function to both events. In the callback code, you can check the status of each relevant event and perform the operation you want accordingly.
Note that on some implementations, the driver will batch multiple commands for execution.
The immediate effect is that that multiple events will be signaled "at once" even though the associated commands complete at a different time.
// event1 & event2 are likely to be signaled at once:
clEnqueueNDRangeKernel();
clEnqueueReadBuffer(&event1);
clEnqueueNDRangeKernel();
clEnqueueReadBuffer(&event2);
Wheres:
// event1 is likely to be signaled before event2:
clEnqueueNDRangeKernel();
clEnqueueReadBuffer(&event1);
clflush(queue);
clEnqueueNDRangeKernel();
clEnqueueReadBuffer(&event2);
clflush(queue);
I would also check on which exact thread the callbacks are invoked.
Is it the same thread each time? or a different one? If the implementation opens a new thread for this task, it might be wiser to open a single thread yourself and wait for events in the order that you wish.
I would like to clear up what it means to be a asynchronous/synchronous and blocking/non-blocking operations in Scala Play using Scala's Future.
So we can have:
Asynchronous and non-blocking
Asynchronous and blocking
Synchronous and non-blocking
Synchronous and blocking
In the Play documentation it says that all operations are asynchronous. But I would like to know how the following would be classed:
If I call an external web service and I have to wait for the response then is this considered to be a asynchronous and blocking operation? For example:
def externalWebCall(url : String): Future[SomeData]
def storeData(data : String) : Future[Unit]
for {
data <- externalWebCall(url)
_ <- storeData(data)
} yield(data)
In this example, I don't want storeData to execute until the externalWebCall service has completed. Is the above code the right way to achieve this? If so I think this is asynchronous and blocking.
The initial request that comes to Play service is treated in an asynchronous and non-blocking way according to the documentation. My understanding of this is that when the Play server receives the request it will not block the main thread. It will then service the request and then make a call to the callback of this request and then return the result. Is that correct?
When people use the term "asynchronous", they usually mean "non-blocking", in the sense that the current execution thread will not stop and wait for the result. In general, synchronous means blocking, asynchronous means non-blocking.
It seems you are confusing "blocking" with "ordering the execution". Your example with the Futures is completely asynchronous, and it equivalent to
val result: Future[Unit] = externalWebCall(url).flatMap(storeData)
storeData can only execute when externalWebCall is done, but any code that would come after val result = ... could also execute before either externalWebCall or storeData. The execution of these methods is wrapped in a Future and run in a different thread, so you have no way of knowing when it will happen. That is what asynchronous means.
The terms blocking and asynchronous have different meanings, but are often confused or used interchangeably.
blocking means that the current thread waits for another thread to do something before executing further instructions. The current thread is blocked until the other thread completes and then it can continue.
asynchronous means that an operation happens on another thread while the current thread continues to execute. The current thread initiates an operation but then continues to execute. When the asynchronous operation completes it notifies the current thread by raising an event or calling some event handler function.
Blocking functions look like normal functions and there is not necessarily any indication that the code being called is blocking or non blocking. So a println statement may be blocking for stderr but non-blocking for stdout.
Asynchronous functions will always have a mechanism for being notified of completion of the operation and this will be visible in the interface. In Scala this usually means returning Future[T] rather than T.
In your example, both externalWebCall and storeData are asynchronous because each method returns Future.
storeData takes a data argument that is only available when externalWebCall completes, so these two operations must by definition operate sequentially.
Quick question, because I feel like I must be missing something.
I'm using rxjs here because it's what I've got in-front of me, this is a general reactiveX question, I believe.
Let's say I have a set of Observables like so:
network_request = some_thing // An observable that produces the result of a network call
event_stream = network_request.flatMapLatest(function(v) {
return connectToThing(v) // This is another observable that needs v
}) // This uses the result of the network call to form a long-term event-based connection
So, this works ok.
Problem, though.
Sometimes the connection thing fails.
So, if I do event_stream.retry() it works great. When it fails, it redoes the network call and gets a new v to use to make a new connection.
Problem
What happens if I want two things chained off of my network_request?
Maybe I want the UI to do something every time the network call completes, like show something about v in the UI?
I can do:
shared = network_request.share() // Other implementations call this refCount
event_stream = shared.flatMapLatest(...) // same as above
ui_stream = shared.flatMapLatest(...) // Other transformation on network response
If I didn't do share then it would have made two requests, which isn't what I want, but with share, when event_stream later has an error, it doesn't retry the network request because the refcount is still at 1 (due to ui_stream), so it immediately returns completed.
What I want
This is obviously a small example I've made up to explain my confusion.
What I want is that every time the result of event_stream (that long term connection) has an error all of the following happens:
the network request is made again
the new response of that request is used to build a new connection and event_stream goes on with new events like nothing happened
that same response is also emitted in ui_stream to lead to further processing
This doesn't feel like a complicated thing, so I must just be misunderstanding something fundamental when it comes to splitting / fanning out RX things.
Workarounds I think I could do but would like to avoid
I'm looking to export these observables, so I can't just build them again and then say "Hey, here's the new thing". I want event_stream and all the downstream processing to not know there's been a disconnection.
Same for ui_stream. It just got a new value.
I could probably work something out using a Subject as a generation counter that I ping every time I want everything to restart, and put the network_request into a flatMap based on that, so that I can break the share...
But that feels like a really hacky solution, so I feel there has to be a better way than that.
What have I fundamentally misunderstood?
As I've been thinking about this more I've come to the same realization as ionoy, which is that retry just disconnects and reconnects, and upstream doesn't know it was due to an error.
When I thought about what I wanted, I realized I really wanted something like a chain, and also a spectator, so I've got this for now:
network_request = some_thing
network_shadow = new Rx.Subject()
event_stream = network_request.do(network_shadow).flatMapLatest(...)
ui_stream = network_shadow.whatever
This has the property where an retry in event_stream or downstream will cause the whole thing to restart, whereas ui_stream is its own thing.
Any errors over there don't do anything, since network_shadow isn't actually a subscriber to event_stream, but it does peel the values off so long as the main event chain is running.
I feel like this isn't ideal, but it is better than what I was concerned I would have to do, which is have a restartEverything.onNext() in an doOnError, which would have been gross.
I'm going to work with this for now, and we'll see where it bites me...
You need to make your cold observable hot by using Publish. Read http://www.introtorx.com/Content/v1.0.10621.0/14_HotAndColdObservables.html#HotAndCold for a great explanation.
I'm having a difficult time trying to understand how the control flow in protractor work in relation to how JS event loop works. Here is what I know so far:
Protractor control flow stores commands that return promises in a queue. The first command will be at the front of the queue and the last command will be at the back. No command will be executed until the command in front of it has its promise resolved.
JS event loop stores asynchronous task (callbacks to be specific). Callbacks are not executed until all functions in the stack have completed and the stack is empty. Before running each callback, there is a check on whether the stack is empty or not.
so lets take this code for example. The code is basically clicking a search button and a api request is made. Then after data is returned, it checks whether the field that stores the returned data exists.
elem('#searchButton').click(); //will execute a api call to retrieve data
browser.wait(ExpectedConditions.presenceOf(elem('#resultDataField'),3000));
expect(elem('#resultDataField').isPresent()).toBeTruthy();
So with this code, I'm able to get it to work. But I don't know how it does it. How is the event loop applied in this scenario?
The core of the ControlFlow implementation is in runEventLoop_ (in Selenium's promise.js implementation).
As I understand it, the ControlFlow registers a call to runEventLoop_ with the JS event loop (e.g., with a 0-second timeout or somesuch). The call to runEventLoop_ can be thought of as a single iteration of a normal event loop. It registers code to actually run a scheduled task (i.e., actually do the work you queued up during your it). Once that task completes or fails (e.g., by hooking its async promise callbacks) the next iteration of runEventLoop_ is scheduled (see the calls to scheduleEventLoop in runEventLoop_).
There is some complexity when a callback ends up registering new promises (those need to be "inserted" before the old next event, this is accomplished by creating a "nested" control flow. Mostly you should never have to know this.)
I am reading libpq reference. It has both of sync and async methods. Bu I discovered something strange.
When I see PQsendQuery function, it seems to send a query and return immediately. And I expected a callback function to get notified, but there was no such thing and the manual says to poll for data availability.
I don't understand why async method is written in polling way. Anyway, as libp is the official client implementation, I believe there should be a good reason for this design. What is that? Or am I missing correct callback stuffs mentioned somewhere else?
In the execution model of a mono-threaded program, the execution flow can't be interrupted by data coming back from an asynchronous query, or more generally a network socket. Only signals (SIGTERM and friends) may interrupt the flow, but signals can't be hooked to data coming in.
That's why having a callback to get notified of incoming data is not possible. The piece of code in libpq that would be necessary to emit the callback would never run if your code doesn't call it. And if you have to call it, that defeats the whole point of a callback.
There are libraries like Qt that provide callbacks, but they're architectured from the ground up with a main loop that acts as an event processor. The user code is organized in callbacks and event-based processing of incoming data is possible. But in this case the library takes ownership of the execution flow, meaning its mainloop polls the data sources. That just shifts the responsibility to another piece of code outside of libpq.
This page is describing how I can get be notified for async result fetch.
http://www.postgresql.org/docs/9.3/static/libpq-events.html#LIBPQ-EVENTS-PROC
PGEVT_RESULTCREATE
The result creation event is fired in response to any query execution
function that generates a result, including PQgetResult. This event
will only be fired after the result has been created successfully.
typedef struct {
PGconn *conn;
PGresult *result; } PGEventResultCreate; When a PGEVT_RESULTCREATE event is received, the evtInfo pointer should be cast to a
PGEventResultCreate *. The conn is the connection used to generate the
result. This is the ideal place to initialize any instanceData that
needs to be associated with the result. If the event procedure fails,
the result will be cleared and the failure will be propagated. The
event procedure must not try to PQclear the result object for itself.
When returning a failure code, all cleanup must be performed as no
PGEVT_RESULTDESTROY event will be sent.