I want to fetch data from two API requests parallelly.
Single<TodayDataEntry> todayDataEntrySingle = openWeatherApi.getTodayData(
"London,uk", FetchDataUtil.API_KEY).subscribeOn(Schedulers.io());
Single<ListWrapper<ForecastDataEntry>> forecastDataEntrySingle = openWeatherApi.getForecastData(
"London,uk", FetchDataUtil.API_KEY).subscribeOn(Schedulers.io());
Single<DataAll> observable = Single.zip(
todayDataEntrySingle,
forecastDataEntrySingle,(t1, t2) -> {
return new DataAll(t2.getList(), t1);
}).subscribeOn(Schedulers.io());
After subscribing, if both API requests fail (like in case where there is no internet connection), one of the error signals is not handled (even though onError is getting called), causing app to crash.
However if I use Observable<> instead of Single<>, the error is handled
and an exception is not thrown. Is something wrong in my
implementation?
If I don't call subscribeOn(Schedulers.io) for individual observables and instead just call for zipped one, the exception is not thrown. However as far as I know, in this case the API requests are not in parallel.
We were facing the exact same issue with Completable.mergeArray() and it literally took us years to figure this out. In summary, the crash is caused by the streams running in parallel and throwing an error at the same time.
We wrote an extensive blog post series describing the issue and our workaround. I think it'll definitely help if you take a look. Let me know in the comments if you have any questions.
Related
I have a question.
In an interview, I was asked if one of the routes of scatter-gather fail, will we get the output. I replied with a no as scatter-gather always gives a consolidated payload taken from all routes, and then they asked me if I still want to get the payload from the successful routes, what should I do -> I answered with Try scope and on-error continue. (I hope I was right here, please explain if I wasn't).
The next scenario they gave me was like this: If I have an on-error-propagate set for the scatter-gather and I also have a payload in that. So when the route fails and the handler comes to on-error-propagate, will the payload present there be printed or not. My answer to this question was a yes and I said that because on-error-propagate executes all its steps, the payload present inside its scope will be printed and then the flow will exit.
I don't know if that was right or not, so please help me with the correct answer for this scenario.
I'm not sure if this is the right platform for these question but I'll try to answer them.
I was asked if one of the routes of scatter-gather fail, will we get the output. I replied with a no as scatter-gather always gives a consolidated payload taken from all routes
Scatter-gather executes all routes simultaneously and generates a LinkedHashMap of all the Mule Events (not just payloads). The payload from each event can certainly be extracted or functions like flatten() can be used to flatten the contents linearly. If one of the routes fail and no error handling is done, an error object will be introduced and scatter-gather will fail.
they asked me if I still want to get the payload from the successful routes, what should I do -> I answered with Try scope and on-error continue.
Your answer seems good to me. Although, please note that on-error-continue should be on another flow that scatter-gather is calling. Having it in the same flow as scatter-gather may suppress the error raised but will not continue executing the next component in same flow.
If I have an on-error-propagate set for the scatter-gather and I also have a payload in that. So when the route fails and the handler comes to on-error-propagate, will the payload present there be printed or not.
The statement is a bit confusing to me as it doesn't specify the placement of the on-error-propagate. If a printing component is inside the on-error-propagate scope and the handler comes to on-error-propagate, it will certainly print. However, if all it has is the payload, the flow will never go to the next component and an error object will be returned to the parent flow (or error is thrown if there's no parent flow).
Quick question, because I feel like I must be missing something.
I'm using rxjs here because it's what I've got in-front of me, this is a general reactiveX question, I believe.
Let's say I have a set of Observables like so:
network_request = some_thing // An observable that produces the result of a network call
event_stream = network_request.flatMapLatest(function(v) {
return connectToThing(v) // This is another observable that needs v
}) // This uses the result of the network call to form a long-term event-based connection
So, this works ok.
Problem, though.
Sometimes the connection thing fails.
So, if I do event_stream.retry() it works great. When it fails, it redoes the network call and gets a new v to use to make a new connection.
Problem
What happens if I want two things chained off of my network_request?
Maybe I want the UI to do something every time the network call completes, like show something about v in the UI?
I can do:
shared = network_request.share() // Other implementations call this refCount
event_stream = shared.flatMapLatest(...) // same as above
ui_stream = shared.flatMapLatest(...) // Other transformation on network response
If I didn't do share then it would have made two requests, which isn't what I want, but with share, when event_stream later has an error, it doesn't retry the network request because the refcount is still at 1 (due to ui_stream), so it immediately returns completed.
What I want
This is obviously a small example I've made up to explain my confusion.
What I want is that every time the result of event_stream (that long term connection) has an error all of the following happens:
the network request is made again
the new response of that request is used to build a new connection and event_stream goes on with new events like nothing happened
that same response is also emitted in ui_stream to lead to further processing
This doesn't feel like a complicated thing, so I must just be misunderstanding something fundamental when it comes to splitting / fanning out RX things.
Workarounds I think I could do but would like to avoid
I'm looking to export these observables, so I can't just build them again and then say "Hey, here's the new thing". I want event_stream and all the downstream processing to not know there's been a disconnection.
Same for ui_stream. It just got a new value.
I could probably work something out using a Subject as a generation counter that I ping every time I want everything to restart, and put the network_request into a flatMap based on that, so that I can break the share...
But that feels like a really hacky solution, so I feel there has to be a better way than that.
What have I fundamentally misunderstood?
As I've been thinking about this more I've come to the same realization as ionoy, which is that retry just disconnects and reconnects, and upstream doesn't know it was due to an error.
When I thought about what I wanted, I realized I really wanted something like a chain, and also a spectator, so I've got this for now:
network_request = some_thing
network_shadow = new Rx.Subject()
event_stream = network_request.do(network_shadow).flatMapLatest(...)
ui_stream = network_shadow.whatever
This has the property where an retry in event_stream or downstream will cause the whole thing to restart, whereas ui_stream is its own thing.
Any errors over there don't do anything, since network_shadow isn't actually a subscriber to event_stream, but it does peel the values off so long as the main event chain is running.
I feel like this isn't ideal, but it is better than what I was concerned I would have to do, which is have a restartEverything.onNext() in an doOnError, which would have been gross.
I'm going to work with this for now, and we'll see where it bites me...
You need to make your cold observable hot by using Publish. Read http://www.introtorx.com/Content/v1.0.10621.0/14_HotAndColdObservables.html#HotAndCold for a great explanation.
I've recently become interested in trying to adapt my Rock-Paper-Scissors game into a multiplayer-friendly program, so today I decided I'd look up a tutorial on servers. It seems I'm following it precisely (aside from using a different IDE). However, something is going wrong and I'm not sure exactly what it is, and it works fine for the tutorial maker. I've looked up EOFException but it didn't exactly help me out.
The tutorial on Youtube
My screenshot of the issue.
[Documentation on EOFException](I had a link here, but I need at least 10 reputation to post more than two links)
"Signals that an end of file or end of stream has been reached unexpectedly during input.
This exception is mainly used by data input streams to signal end of stream. Note that many other input operations return a special value on end of stream rather than throwing an exception."
[A similar but apparently unresolved question asked here](I had a link here too]
By the way, if you look up exactly what I've posted here, you'll also find that I've asked it at DaniWeb. I'm just posting in multiple places in case it isn't resolved at one or the other. If it is, well... the more knowledge, the better.
EOFException during readUTF() just means it's reached the end of the stream, like it says on the tin. Note that this method doesn't return null at end of stream, unlike readLine() (but like all other readXXX() methods).
It can also mean that your sending and receiving is out of sync, e.g. you are trying to read some ridiculously large number of bytes because you left out a readInt() or similar, or you wrote something extra at the peer that you shouldn't have, so you're trying to read the next bytes in the stream as the result of writeUTF() when it isn't. This is an application protocol error.
How this happened in the code you posted is another question, but your code doesn't close the sockets, which doesn't help. Add an out.close() to your server code, and in.close() to the client code. However I cannot reproduce your problem with or without these closes. Are you sure this is the real code?
I have an application that could potentitally throw an error on receiving a ExecutionReport (35=8) message.
This error is thrown at the application level and not at the fix engine level.
The fix engine records the message as seen and therefore will not send a ResendRequest (35=2). However the application has not processed it and I would like to manually trigger a re-processing of the missed ExecutionReport.
Forcing a ResendRequest (35=2) does not work as it requires modifying the expected next sequence number.
I was wonderin if FIX supports replaying of messages but without requiring a sequence number reset?
When processing an execution report, you should not throw any exceptions and expect FIX library to handle it. You either process the report or you have a system failure (i.e. call abort()). Therefore, if your code that handles execution report throws an exception and you know how to handle it, then catch it in that very same function, eliminate the cause of the problem and try processing again. For example (pseudo-code):
// This function is called by FIX library. No exceptions must be thrown because
// FIX library has no idea what to do with them.
void on_exec_report(const fix::msg &msg)
{
for (;;) {
try {
// Handle the execution report however you want.
handle_exec_report(msg);
} catch(const try_again_exception &) {
// Oh, some resource was temporarily unavailable? Try again!
continue;
} catch(const std::exception &) {
// This should never happen, but it did. Call 911.
abort();
}
}
}
Of course, it is possible to make FIX library do a re-transmission request and pass you that message again if exception was thrown. However, it does not make any sense at all because what is the point of asking the sender (over the network, using TCP/IP) to re-send a message that you already have (up your stack :)) and just need to process. Even if it did, what's the guarantee it won't happen again? Re-transmission in this case is not only doesn't sound right logically, the other side (i.e. exchange) may call you up and ask to stop doing this crap because you put too much load on their server with unnecessary re-transmit (because IRL TCP/IP does not lose messages and FIX sequence sync process happens only when connecting, unless of course some non-reliable transport is used, which is theoretically possible but doesn't happen in practice).
When aborting, however, it is FIX library`s responsibility not to increment RX sequence unless it knows for sure that user has processed the message. So that next time application starts, it actually performs synchronization and receives missing messages. If QuickFIX is not doing it, then you need to either fix this, take care of this manually (i.e. go screw with the file where it stores RX/TX sequence numbers), or use some other library that handles this correctly.
This is the wrong thing to do.
A ResendRequest tells the other side that there was some transmission error. In your case, there wasn't, so you shouldn't do that. You're misusing the protocol to cover your app's mistakes. This is wrong. (Also, as Vlad Lazarenko points out in his answer, if they did resend it, what's to say you won't have the error again?)
If an error occurs in your message handler, then that's your problem, and you need to find it and fix it, or alternately you need to catch your own exception and handle it accordingly.
(Based on past questions, I bet you are storing ExecutionReports to a DB store or something, and you want to use Resends to compensate for DB storage exceptions. Bad idea. You need to come up with your own redundancy solution.)
In regard to potential runtime failures, like database queries, it seems that one must use some form of Either[String, Option[T]] in order to accurately capture the following outcomes:
Some (record(s) found)
None (no record(s) found)
SQL Exception
Option simply does not have enough options.
I guess I need to dive into scalaz, but for now it's straight Either, unless I'm missing something in the above.
Have boxed myself into a corner with my DAO implementation, only employing Either for write operations, but am now seeing that some Either writes depend on Option reads (e.g. checking if email exists on new user signup), which is a majorly bad gamble to make.
Before I go all-in on Either, does anyone have alternate solutions for how to handle the runtime trifecta of success/fail/exception?
Try Box from the fantastic lift framework. It provides exactly what you want.
See this wiki (and the links at the top) for details. Fortunately lift project is well modulized, the only dependency to use Box is net.lift-web % lift-common
Use Option[T] for the cases records found and no records found and throw an exception in the case of SQLException.
Just wrap the exception inside your own exception type, like PersistenceException so that you don't have a leaky abstraction.
We do it like this because we can't and don't want to recover from unexpected database exceptions. The exception gets caught on the top level and our web service returns a 500 Internal server error in such case.
In cases where we want to recover we use Validation from scalaz, which is much like Lift's Box.
Here's my revised approach
Preserve Either returning query write operations (useful for transactional blocks where we want to rollback on for comprehension Left outcome).
For Option returning query reads, however, rather than swallowing the exception with None (and logging it), I have created a 500 error screen, letting the exception bubble up.
Why not just work with Either result type by default when working with runtime failures like query Exceptions? Option[T] reads are a bit more convenient to work with vs Either[Why-Fail, Option[T]], which you have to fold/map through to get at T. Leaving Either to write operations simplifies things (all the more so given that's how the application is currently setup, no refactoring required ;-))
The only other change required is for AJAX requests. Rather than displaying the entire 500 error page response in the AJAX status div container, we check for the status type and display 500 error message accordingly.
if(data.status == 500)
$('#status > div').html("an error occurred, please try again")
Could probably do an isAjax check server-side prior to sending the response; in which case I can send back only status + message rather than the error page itself.