I am writing a SwiftUI app that uses a single Task(priority: .low) to do lots of HTTP requests as a low priority whose results are displayed in a Table.
There's several hundreds of MBs of data being aggregated to display totals, averages etc.
Thing is, I am only creating one Task, but this is causing my UI to hang. Why is this? I think I must have a fundamental misunderstanding around how tasks are created and how they relate to processes and threads.
I am using Mongo's Realm to store the data having become frustrated with Core Data. Does anyone have any experience of agonisingly slow apps using Realm?
There is far too much code to post, but all the HTTP requests are being made using URLSession's async methods. These data are being transformed and saved in my Realm. In any case, I certainly wouldn't expect this to cause the UI to hang! What might be going on?
I know it's tough to triage without code, but as I say there is a lot of it. I’m just after general guidance really, rather than a fix for a very specific issue.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
I have been using RxJava in my project for about a year now.
With time, I grew to love it very much - now I'm thinking maybe too much...
Most methods I write now have some form of Rx in it, which is great! (until it's not).
I now notice that some methods require a lot of work to combine the different observable producing methods.
I get the feeling that although I understand what I write now, the next programmer will have a really hard time understanding my code.
Before I get to the bottom line let me give an example straight from my code in Kotlin (Don't dive too deep into it):
private fun <T : Entity> getCachedEntities(
getManyFunc: () -> Observable<Timestamped<List<T>>>,
getFromNetwork: () -> Observable<ListResult<T>>,
getFunc: (String) -> Observable<Timestamped<T>>,
insertFunc: (T) -> Unit,
updateFunc: (T) -> Unit,
deleteFunc: (String) -> Unit)
= concat(
getManyFunc().filter { isNew(it.timestampMillis) }
.map { ListResult(it.value, "") },
getFromNetwork().doOnNext {
syncWithStorage(it.entities, getFunc, insertFunc, updateFunc, deleteFunc)
}).first()
.onErrorResumeNext { e -> // If a network error occurred, return the cached data and the error
concat(getManyFunc().map { ListResult(it.value, "") }, error(e))
}
Briefly what this does is:
Retrieve some timestamped data from storage
If data is not new, fetch data from network
Sync network data again with the storage (to update it)
If a network error occured, again retrieve the older data and the error
And here comes my actual question:
Reactive programming offers some really powerful concepts. But as we know with great power comes great responsibility.
Where do we draw the line? Is it OK to fill our entire programs with awesome reactive oneliners or should we save it only for really mundane operations?
Obviously this is very subjective, but I hope someone with more experience can share his knowledge and pitfalls.
Let me phrase it better
How do I design my code to be reactive yet easy to read?
When you pick up Rx, it becomes this awesome shiny hammer and everything starts looking like a rusty nail just waiting for you to bang in.
Personally, I think the biggest clue is in the name, reactive framework. Given a requirement, you need to reflect upon whether a reactive solution truly makes sense.
In any Rx proposition, you are looking to introduce one or more event streams and carry out some action in response to an event.
I think there are two key questions to ask:
Are you in control of the event stream?
To what degree must you complete responses at the rate of the event stream?
If you do not have control of the event stream and you must respond at the rate of the event stream then Rx is a good candidate.
In any other circumstance, it is probably a poor choice.
I have seen many examples where people have jumped through hoops to create the illusion of a lack of control in order to justify Rx - which seems crazy to me. Why give up the control that you have?
Some examples:
You have to extract data from a fixed list of files and store it in a database. You decide to push each file name into a subject and create a reactive pipeline that opens each file and projects the data, then processes the data in some way and finally writes it to the database.
This fails the control test and the rate test. It would be far easier to iterate over the files and pull them in and process them as fast as you can. The phrase "decide to push" is the giveaway here.
You need to display stock prices from a stock exchange.
Clearly this is a good choice for Rx. If you can't keep up with the rate of prices in general, you are screwed. It might be the case that you conflate prices (perhaps to provide an update only once every second) - but this still qualifies as keeping up. The one thing you can't do is ask the stock exchange to slow down.
These (real world) examples pretty much fall at opposite ends of the spectrum and don't have much grey area. But there is a lot of grey area out there where control isn't clear.
Sometimes you are wearing the client hat in a client/server system and it can be easy to fall into the trap of sacrificing control, or putting control in the wrong place - which can easily be fixed with correct design. Consider this:
A client application displays news updates from a server.
News updates are submitted to the server at any time and are created in high volume.
The client should be refreshed at an interval set by the client.
Refresh interval can be changed at any time and the user can always request an immediate refresh.
The client only shows updates tagged with particular keywords, as specified by the user.
The news updates are sometimes lengthy and the client should not store the full content of news updates, but rather display the headline and summary.
At user request, the full content of an article can be shown.
Here, the frequency of news updates is not in control of the client. But the desired refresh rate and the tags of interest are.
For the client to receive all the news updates as they arrive and filter them client side isn't going to work. But there are plenty of options:
Should the server send a data stream of updates taking into account the client refresh rate? What if the client goes offline?
What if there are thousands of clients? What if the client wants an immediate refresh?
There are lots of valid ways to tackle this problem that include more or less reactive elements. But any good solution should take account of the client's control of tags and desired refresh rate, and the lack of control of news update frequency (by client or server). You might want the server to react to changes in client interest by updating the events that it pushes to the client - which it pushes only as long as the client is listening (detected via a heartbeat). When the user wants a full article, then the client would pull the article down.
There is much debate in the Rx community about back-pressure. This is the idea that the client should inform the server when it is overloaded and the server respond by somehow reducing the event stream. I think this is a misguided approach that can lead to confusing designs.
To my mind, as soon as a client needs to give this feedback, it has failed the response rate test. At this point, you are not in a reactive situation, you are in an async enumerable situation. i.e. The client should be saying "I am ready" when it is ready for more and then waiting in a non-blocking fashion for server to respond.
This would be appropriate if the first scenario were modified to be files arriving in a drop-folder, of varying lengths and complexity to process. The client should make a non-blocking call for the next file, process it, and repeat. (Add parallelism as required) - and not be responding to a stream of file-arrived events.
Wrap up
I've deliberately avoided other valid concerns such as maintainability of code, performance of Rx itself etc. Most because they are addressed elsewhere and more importantly because I think the ideas here are more divisive than those concerns.
So if you reflect on the elements of control and response rate in your scenario you and will probably stay on the right track.
The response rate issue can be subtle - and the degree aspect is important. Arrival rate can fluctuate, and there is going to be some acceptable degree of fluctuation in response rate - clearly, if you don't ultimately have a way to "catch up" then at some point the client will blow up.
I find that there are two things I keep in mind when writing Rx (or any mildly sophisticated/new technology)
Can I test it?
Can I easily hire someone that can maintain it. Not struggle to maintain it, but will be fine left alone to maintain it?
To this end, I also find that just because you can, doesn't always mean you should. As a guide I try to avoid creating queries that are over say 7 lines of code. Queries bigger than this, I try to separate into sub queries that I compose.
If code you have provided is at the core of the code base, and is at the extreme end of the complexity, then It may be fine. However, if you find all of your Rx code carries that much complexity, you may be creating a difficult to work with code base.
Event sourcing is pitched as a bonus for a number of things, e.g. event history / audit trail, complete and consistent view regeneration, etc. Sounds great. I am a fan. But those are read-side implementation details, and you could accomplish the same by moving the event store completely to the read side as another subscriber.. so why not?
Here's some thoughts:
The views/denormalizers themselves don't care about an event store. They just handle events from the domain.
Moving the event store to the read side still gives you event history / audit
You can still regenerate your views from the event store. Except now it need not be a write model leak. Give him read model citizenship!
Here seems to be one technical argument for keeping it on the write side. This from Greg Young at http://codebetter.com/gregyoung/2010/02/20/why-use-event-sourcing/:
There are however some issues that exist with using something that is storing a snapshot of current state. The largest issue revolves around the fact that you have introduced two models to your data. You have an event model and a model representing current state.
The thing I find interesting about this is the term "snapshot", which more recently has become a distinguished term in event sourcing as well. Introducing an event store on the write side adds some overhead to loading aggregates. You can debate just how much overhead, but it's apparently a perceived or anticipated problem, since there is now the concept of loading aggregates from a snapshot and all events since the snapshot. So now we have... two models again. And not only that, but the snapshotting suggestions I've seen are intended to be implemented as an infrastructure leak, with a background process going over your entire data store to keep things performant.
And after a snapshot is taken, events before the snapshot become 100% useless from the write perspective, except... to rebuild the read side! That seems wrong.
Another performance related topic: file storage. Sometimes we need to attach large binary files to entities. Conceptually, sometimes these are associated with entities, but sometimes they ARE the entities. Putting these in the event store means you have to physically load that data each and every time you load the entity. That's bad enough, but imagine several or hundreds of these in a large aggregate. Every answer I have seen to this is to basically bite the bullet and pass a uri to the file. That is a cop-out, and undermines the distributed system.
Then there's maintenance. Rebuilding views requires a process involving the event store. So now every view maintenance task you ever write further binds your write model into using the event store.. forever.
Isn't the whole point of CQRS that the use cases around the read model and write model are fundamentally incompatible? So why should we put read model stuff on the write side, sacrificing flexibility and performance, and coupling them back up again. Why spend the time?
So all in all, I am confused. In all respects from where I sit, the event store makes more sense as a read model detail. You still achieve the many benefits of keeping an event store, but you don't over-abstract write side persistence, possibly reducing flexibility and performance. And you don't couple your read/write side back up by leaky abstractions and maintenance tasks.
So could someone please explain to me one or more compelling reasons to keep it on the write side? Or alternatively, why it should NOT go on the read side as a maintenance/reporting concern? Again, I'm not questioning the usefulness of the store. Just where it should go :)
This is a long dead question that someone pointed me to. There are quite a few reasons why its better to store events on the write side.
From my understanding the architecture you are talking about is a very common one that I see ... fail. We will store our domain model in a relational database then put out events. You add the twist of them saving the events on the read side in an event store. This will likely lead to a mess.
The first issue you will run into is in the publishing of your events. What happens when I save to the database and publish to say MSMQ (I die in the middle). So DTC gets introduced between them. This is a huge thing to bring in, distributed transactions should be avoided like the plague. It is also quite inefficient as I am probably making the data durable twice (once to queue once to database). This will limit system throughput by a lot (DTC benchmarks of 200-300 messages/second are common, with events only 20-30k/second is common).
Some work around the need for DTC by putting a table in their database that has the events and operates as a queue. This will avoid the need for DTC however this will still run into the next issue.
What happens when you have a bug? I know you would never write buggy code but one of the Jrs/maintenance developers later working with the project. As an example what happens when the domain object change and the event raised do not match? Say you set State on your domain object to "LA" (hardcoded) but you properly set State on the event to cmd.State ("CT").
How will you detect such errors are occurring? The biggest problem with what is being discussed is that there are now two sources of "truth" there is the database on the write side and the event stream coming out. There is no way to prove that they are equivalent. This will cause all sorts of weird bugs down the line.
I think this is really an excellent question. Treating your aggregate as a sequence of events is useful in its own right on the write side, making command retries and the like easier. But I agree that it seems upsetting to work to create your events, then have to make yet another model of your object for persistence if you need this snapshotting performance improvement.
A system where your aggregates only stored snapshots, but sent events to the read-model for projection into read models would I think be called "CQRS", just not "Event Sourcing". If you kept the events around for re-projection, I guess you'd have a system that was very much both.
But then wouldn't you have three definitions? One for persisting your aggregates, one for communicating state changes, and any number more for answering queries?
In such a system it would be tempting to start answering queries by loading your aggregates and asking them questions directly. While this isn't forbidden by any means, it does tend to start causing those aggregates to accrete functionality they might not otherwise need, not to mention complicating threading and transactions.
One reason for having the event store on the write-side might be for resolving concurrency issues before events become "facts" and get distributed/dispatched, e.g. through optimistic locking on committing to event streams. That way, on the write side you can make sure that concurrent "commits" to the same event stream (aggregate) are resolved, one of them gets through, the other one has to resolve the conflicts in a smart way through comparing events or propagating the conflict to the client, thus rejecting the command.
I have noticed that for a mobile application, saving on the main thread seems to take a bit when it compares to other applications on the device. Is it recommended to only save Core Data when the application enters the background or when an application closes instead of anytime items are added and sent / received from the api?
That's kind of a broad question, but I've found that saving core data after VewDidAppear statements is better than viewWill statements. Giving the user something to engage with and persisting makes it less noticeable than on a load. However, if a user is used to waiting for something like an activity loop, adding the save to that doesn't tax it too much (IMHO).
Not sure this help, just my experience.
I am new to objective-c/cocoa programming. I am making an application which is to constantly sync with a server and keep its view updated.
Now in a nutshell, heres what I thought of: Initiate an NSTimer to trigger every second or two, contact the server, if there is a change, update the view. Is this a good way of doing it?
I have read elsewhere that you can have a thread running in the background which monitors the changes and updates the view. I never worked with threads before and I know they can be quite troublesome and you need a good amount of experience with memory management to get most out of them.
I have one month to get this application done. What do you guys recommend? Just use an NSTimer and do it the way I though of...or learn multithreading and get it done that way (but keep in mind my time frame).
Thanks!
I think using separate thread in this case would be too much. You need to use threads when there is some task that runs for considerable amount of time and can freeze your app for some time.
In your case do this:
Create timer and call some method (say update) every N seconds.
in update send asynchronous request to server and check for any changes.
download data using NSURLConnection delegate and parse. Note: if there is probability that you can receive a huge amount of data from server and its processing can take much time (for example parsing of 2Mb of XML data) then you do need to perform that is a separate thread.
update all listeners (appropriate view controllers for example) with processed data.
continue polling using timer.
Think about requirements. The most relevant questions, IMO, are :
does your application have to get new data while running in background?
does your application need to be responsive, that is, not sluggish when it's fetching new data?
I guess the answer to the first question is probably no. If you are updating a view depending on the data, it's only required to fetch the data when the view is visible. You cannot guarantee always fetching data in background anyway, because iOS can always just kill your application. Anyway, in your application's perspective, multithreading is not relevant to this question. Because either you are updating only in foreground or also in background, your application need no more than one thread.
Multithreading is relevant rather to the second question. If your application has to remain responsive while fetching data, then you will have to run your fetching code on a detached thread. What's more important here is, the update on the user interface (like views) must happen on the main thread again.
Learning multithreading in general is something indeed, but iOS SDK provides a lot of help. Learning how to use operation queue (I guess that's the easiest to learn, but not necessarily the easiest to use) wouldn't take many days. In a month period, you can definitely finish the job.
Again, however, think clearly why you would need multithreading.
What are good practices for asynchronously pulling large amounts of XML from a RESTful service into a Core Data store, and from this store, populating a UITableView on the fly?
I'm thinking of using libxml2's xmlParseChunk() function to parse chunks of incoming XML and translate a node and its children into the relevant managed objects, as nodes come in.
At the same time that these XML nodes are turned into managed objects, I want to generate UITableView rows, in turn. Say, 50 rows at a time. Is this realistic?
In your experience, what do you do to accomplish this task, to maintain performance and handle, potentially, thousands of rows? Are there different, simpler approaches that work as well or better?
Sure, this is a pretty standard thing. The easiest solution is to do the loading in a background thread on one MOC, and have the UI running on the main thread with its own MOC. Whenever you get a chunk of data you want to have appear (say 50 entries), you have the background MOCsave:.
Assuming you have the foreground MOC rigged to merge changes (via mergeChangesFromContextDidSaveNotification:) then whenever you save the background MOC the foreground MOC will get all of those changes. Assuming you are using NSFetchedResultsController it has delegate methods to cope with changes in its MOC, and if you are using Apple's sample code then you probably already have everything setup correctly.
In general CoreData is going to be faster than anything you roll yourself unless you really know what you are doing and are willing to spend a ton of time tuning for your specific case. The biggest thing you can do is make sure that slow things (like XML processing and synchronous flash I/O caused by save:) are not on the main thread blocking user interaction.
Joe Hewitt (Facebook app developer) has release much of his code as open-source. It is called Three20. There is a class there that is great for fetching internet data and populating it into a table, without the need for the data beforehand. The classes used for this are called TTTableViewController and TTTableViewDataSource.
From here, it would not be much of a stretch to store as CoreData, just subclass the classes as you see fit with the supplied hooks.
If you are worried about too much data, 50 at a time does sound reasonable. These classes have a built in "More" button to help you out.
From the Three20 readme:
Internet-aware table view controllers
TTTableViewController and
TTTableViewDataSource help you to
build tables which load their content
from the Internet. Rather than just
assuming you have all the data ready
to go, like UITableView does by
default, TTTableViewController lets
you communicate when your data is
loading, and when there is an error or
nothing to display. It also helps you
to add a "More" button to load the
next page of data, and optionally
supports reloading the data by shaking
the device.
No one has mentioned RestKit yet? My friends ... seriously, you have to check this out. If you are doing anything with REST on iOS (and now on OS X) and particularly if you're wanting to work with Core Data ... PLEASE have a look at RestKit. I've saved countless hours implementing some pretty complex data synchronization between a server and my Core Data models on iOS. RestKit made it so damned easy, it almost makes you sick.