Quick question, because I feel like I must be missing something.
I'm using rxjs here because it's what I've got in-front of me, this is a general reactiveX question, I believe.
Let's say I have a set of Observables like so:
network_request = some_thing // An observable that produces the result of a network call
event_stream = network_request.flatMapLatest(function(v) {
return connectToThing(v) // This is another observable that needs v
}) // This uses the result of the network call to form a long-term event-based connection
So, this works ok.
Problem, though.
Sometimes the connection thing fails.
So, if I do event_stream.retry() it works great. When it fails, it redoes the network call and gets a new v to use to make a new connection.
Problem
What happens if I want two things chained off of my network_request?
Maybe I want the UI to do something every time the network call completes, like show something about v in the UI?
I can do:
shared = network_request.share() // Other implementations call this refCount
event_stream = shared.flatMapLatest(...) // same as above
ui_stream = shared.flatMapLatest(...) // Other transformation on network response
If I didn't do share then it would have made two requests, which isn't what I want, but with share, when event_stream later has an error, it doesn't retry the network request because the refcount is still at 1 (due to ui_stream), so it immediately returns completed.
What I want
This is obviously a small example I've made up to explain my confusion.
What I want is that every time the result of event_stream (that long term connection) has an error all of the following happens:
the network request is made again
the new response of that request is used to build a new connection and event_stream goes on with new events like nothing happened
that same response is also emitted in ui_stream to lead to further processing
This doesn't feel like a complicated thing, so I must just be misunderstanding something fundamental when it comes to splitting / fanning out RX things.
Workarounds I think I could do but would like to avoid
I'm looking to export these observables, so I can't just build them again and then say "Hey, here's the new thing". I want event_stream and all the downstream processing to not know there's been a disconnection.
Same for ui_stream. It just got a new value.
I could probably work something out using a Subject as a generation counter that I ping every time I want everything to restart, and put the network_request into a flatMap based on that, so that I can break the share...
But that feels like a really hacky solution, so I feel there has to be a better way than that.
What have I fundamentally misunderstood?
As I've been thinking about this more I've come to the same realization as ionoy, which is that retry just disconnects and reconnects, and upstream doesn't know it was due to an error.
When I thought about what I wanted, I realized I really wanted something like a chain, and also a spectator, so I've got this for now:
network_request = some_thing
network_shadow = new Rx.Subject()
event_stream = network_request.do(network_shadow).flatMapLatest(...)
ui_stream = network_shadow.whatever
This has the property where an retry in event_stream or downstream will cause the whole thing to restart, whereas ui_stream is its own thing.
Any errors over there don't do anything, since network_shadow isn't actually a subscriber to event_stream, but it does peel the values off so long as the main event chain is running.
I feel like this isn't ideal, but it is better than what I was concerned I would have to do, which is have a restartEverything.onNext() in an doOnError, which would have been gross.
I'm going to work with this for now, and we'll see where it bites me...
You need to make your cold observable hot by using Publish. Read http://www.introtorx.com/Content/v1.0.10621.0/14_HotAndColdObservables.html#HotAndCold for a great explanation.
Related
I have a reactive pipeline to process incoming requests. For each request I need to call a business-relevant function (doSomeRelevantProcessing).
After that is done, I need to notify some external service about what happened. That part of the pipeline should not increase the overall response time.
Also, notifying this external system is not business critical: giving a quick response after the main part of the pipeline is finished is more important than making sure the notification is successful.
As far as I learned, the only way to run something in the background without slowing down the overall process is to subscribe to in directly in the pipeline, thus achieving a fire-and-forget mentality.
Is there a good alternative to subscribing inside the flatmap?
I am a little worried about what might happen if notifying the external service takes longer than the original processing and a lot of requests are coming in at once. Could this lead to a memory exhaustion or the overall process to block?
fun runPipeline(incoming: Mono<Request>) = incoming
.flatMap { doSomeRelevantProcessing(it) } // this should not be delayed
.flatMap { doBackgroundJob(it) } // this can take a moment, but is not super critical
fun doSomeRelevantProcessing(request: Request) = Mono.just(request) // do some processing
fun doBackgroundJob(request: Request) = Mono.deferContextual { ctx: ContextView ->
val notification = "notification" // build an object from context
// this uses non-blocking HTTP (i.e. webclient), so it can take a second or so
notifyExternalService(notification).subscribeOn(Schedulers.boundedElastic()).subscribe()
Mono.just(Unit)
}
fun notifyExternalService(notification: String) = Mono.just(Unit) // might take a while
I'm answering this assuming that you notify the external service using purely reactive mechanisms - i.e. you're not wrapping a blocking service. If you are then the answer would be different as you're bound by the size of your bounded elastic thread pool, which could quickly become overwhelmed if you have hundreds of requests a second incoming.
(Assuming you're using reactive mechanisms, then there's no need for .subscribeOn(Schedulers.boundedElastic()) as you give in your example, as that's not buying you anything - it's designed for wrapping legacy blocking services.)
Could this lead to a memory exhaustion
It's only a possibility in really extreme cases, the memory used by each individual request will be tiny. It's almost certainly not worth worrying about, if you start seeing memory issues here then you'll almost certainly be hit by other issues elsewhere.
That being said, I'd probably recommend adding .timeout(Duration.ofSeconds(5)) or similar before your inner subscribe method to make sure the requests are killed off after a while if they haven't worked for any reason - this will prevent them building up.
...or [can this cause] the overall process to block?
This one is easier - a short no, it can't.
I’m new to scala and I’m trying to make sense of what this code is doing in a codebase I want to make updates to.
Removing some of the specifics, the chunk I don’t understand is this:
I’ve seen some scala code that does things like:
val someA = something.createSomeA(....)
Future {
someA.doSomething1(....)
someA.doSomething2(.....)
}
// then log some things unrelated to the future
someA
// end of func
I don’t really understand what the future is doing in this case as it’s not assigned to anything. Could someone explain what the Future is doing here?
I know the details depend on what the doSomethings are actually doing, but could someone explain generally what this would be for? I’m only familiar with the use of Futures when they’re assigned to a variable and then checked for completion in some way at a later point.
Help would be appreciated!! (Sorry for poor formatting, I’m doing this from my phone)
Three words for you: "fire and forget".
If you understand the case, when the future is assigned to a variable, and then checked/transformed later, then you already know what's happening here: the insides of the Future are being executed asynchronously.
The only difference is that in this case it is never accessed again. Why? Probably, because nobody cares. Some operations return a result when they complete, that can be used later, others do not.
For example, if I wanted to print out a log message asynchronously, I'd write something like Future { logger.info(mymessage) } without assigning it to anything. Why? Well, I don't really care when (or even if) it completes. There is no return value I could use, and, if it fails ... well, I don't have any meaningful way to handle that, other than ignoring the error.
For an operation like this, I don't need to wait for it to complete, since it doesn't return anything useful back to me anyway. So, I can just start it, and forget. No need to assign it to anything.
I have an application that could potentitally throw an error on receiving a ExecutionReport (35=8) message.
This error is thrown at the application level and not at the fix engine level.
The fix engine records the message as seen and therefore will not send a ResendRequest (35=2). However the application has not processed it and I would like to manually trigger a re-processing of the missed ExecutionReport.
Forcing a ResendRequest (35=2) does not work as it requires modifying the expected next sequence number.
I was wonderin if FIX supports replaying of messages but without requiring a sequence number reset?
When processing an execution report, you should not throw any exceptions and expect FIX library to handle it. You either process the report or you have a system failure (i.e. call abort()). Therefore, if your code that handles execution report throws an exception and you know how to handle it, then catch it in that very same function, eliminate the cause of the problem and try processing again. For example (pseudo-code):
// This function is called by FIX library. No exceptions must be thrown because
// FIX library has no idea what to do with them.
void on_exec_report(const fix::msg &msg)
{
for (;;) {
try {
// Handle the execution report however you want.
handle_exec_report(msg);
} catch(const try_again_exception &) {
// Oh, some resource was temporarily unavailable? Try again!
continue;
} catch(const std::exception &) {
// This should never happen, but it did. Call 911.
abort();
}
}
}
Of course, it is possible to make FIX library do a re-transmission request and pass you that message again if exception was thrown. However, it does not make any sense at all because what is the point of asking the sender (over the network, using TCP/IP) to re-send a message that you already have (up your stack :)) and just need to process. Even if it did, what's the guarantee it won't happen again? Re-transmission in this case is not only doesn't sound right logically, the other side (i.e. exchange) may call you up and ask to stop doing this crap because you put too much load on their server with unnecessary re-transmit (because IRL TCP/IP does not lose messages and FIX sequence sync process happens only when connecting, unless of course some non-reliable transport is used, which is theoretically possible but doesn't happen in practice).
When aborting, however, it is FIX library`s responsibility not to increment RX sequence unless it knows for sure that user has processed the message. So that next time application starts, it actually performs synchronization and receives missing messages. If QuickFIX is not doing it, then you need to either fix this, take care of this manually (i.e. go screw with the file where it stores RX/TX sequence numbers), or use some other library that handles this correctly.
This is the wrong thing to do.
A ResendRequest tells the other side that there was some transmission error. In your case, there wasn't, so you shouldn't do that. You're misusing the protocol to cover your app's mistakes. This is wrong. (Also, as Vlad Lazarenko points out in his answer, if they did resend it, what's to say you won't have the error again?)
If an error occurs in your message handler, then that's your problem, and you need to find it and fix it, or alternately you need to catch your own exception and handle it accordingly.
(Based on past questions, I bet you are storing ExecutionReports to a DB store or something, and you want to use Resends to compensate for DB storage exceptions. Bad idea. You need to come up with your own redundancy solution.)
I'm a newbie to WF and rather lost. Here's what I have so far:
I've created a workflow service app (xamlx), added needed variables
I've created a custom NativeActivity where I'm calling CreateBookmark from within Execute, which is between the Receive & Send activity for the service. (Ultimately this will actually do something besides creating the bookmark).
The bookmark gets created just fine, but after stepping out of the Execute method, nothing happens for one minute until the service times out, giving me that message "The request channel timed out while waiting for a reply after 00:00:59.9699970. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout." (I tried posting an image of the xamlx, but as a newbie it won't let me; suffice it to say I'm getting from my Receive, into my custom native activity, but never getting as far as the SendReply).
I assume I'm missing something rather fundatmental, but I can't see what. I've originally tried using NativeActivity<T> to return what I want, but that behaves the same.
Found out what I was doing wrong: needed to use overload of CreateBookmark that has BookmarkOptions parameter and set it to BookmarkOptions.NonBlocking.
Strangely, I did not find one example anywhere that mentioned this.
I have a fairly good idea of what the Subject class does and when to use it, but I've just been looking through the language reference on msdn and see there are various other ISubject implementations such as:
AsyncSubject
BehaviorSubject
ReplaySubject
As the documentation is pretty thin on the ground, whats the point of each of these types and under what situations would you use them?
These subjects all share a common property - they take some (or all) of what gets posted to them via OnNext and record it and play it back to you - i.e. they take a Hot Observable and make it Cold. This means, that if you Subscribe to any of these more than once (i.e. Subscribe => Unsubscribe => Subscribe again), you'll see at least one of the same value again.
ReplaySubject: Every time you subscribe to the Subject, you get the entire history of what has been posted replayed back to you, as fast as possible (or a subset, like the last n items)
AsyncSubject: Always plays back the last item posted and completes, but only after the source has completed. This Subject is awesome for async functions, since you can write them without worrying about race conditions: even if someone Subscribes after the async method completes, they get the result.
BehaviorSubject: Kind of like ReplaySubject but with a buffer of one, so you always get the last thing that was posted. You also can provide an initial value. Always provides one item instantly on Subscribe.
In light of the latest version (v1.0.2856.0) and to keep this question up to date, there has been a new set of subject classes:
FastSubject, FastBehaviorSubject, FastAsyncSubject and FastReplaySubject
As per the release notes they
are much faster than regular subjects
but:
don’t decouple producer and consumer by an IScheduler
(effectively limiting them to
ImmediateScheduler);
don’t protect against stack overflow;
don’t synchronize input messages.
Fast subjects are used by Publish and
Prune operators if no scheduler is
specified.
In regards to AsyncSubject
This code:
var s = new AsyncSubject<int>();
s.OnNext(1);
s.Subscribe(Console.WriteLine);
s.OnNext(2);
s.OnNext(3);
s.OnCompleted();
prints a single value 3. And it prints same if subscription is moved to after completion. So it plays back not the first, but the last item, plays it after completion (until complete, it does not produce values), and it does not work like Subject before completion.
See this Prune discussion for more info (AsyncSubject is basically the same as Prune)
Paul's answer pretty much nails it. There's a few things worth adding, though:
AsyncSubject works as Paul says, but only after the source completes. Before that, it works like Subject (where "live" values are received by subscribers)
AsyncSubject has changed since I last ran tests against it. It no longer acts as a live subject before completion, but waits for completion before it emits a value. And, as Sergey mentions, it returns the last value, not the first (though I should have caught that as that's always been the case)
AsyncSubject is used by Prune, FromAsyncPattern, ToAsync and probably a few others
BehaviorSubject is used by overloads of Publish that accept an initial value
ReplaySubject is used by Replay
NOTE: All operator references above refer to the publishing set of operators as they were before they were replaced with generalised publish operators in rev 2838 (Christmas '10) as it has been mentioned that the original operators will be re-added