Like the native iPhone Messages app, I want to code AcaniChat to return the last 50 messages sorted chronologically. Let's say there are 200 messages total in Core Data.
I know I can use fetchOffset=150 & fetchLimit=50 (Actually, do I even need fetchLimit in this case since I want to fetch all the way to the end?), but can I fetch the last 50 messages without first having to fetch the messages count? For example, with Redis, I could just set fetchOffset to -50.
Reverse the sort order, and grab the first 50.
EDIT
But then, how do I display the messages in chronological order? I'm
using an NSFetchedResultsController. – MattDiPasquale
That wasn't part of your question now, was it ;-)
Anyhow, the FRC is not used directly. Your view controller is asked to provide the information, and it then asks the FRC. You can do simple math to transform section/row to get the reverse order.
You could also use a second array internally that has a copy of the objects in the FRC, but with a different sort ordering. That's simple as well.
More complex, but more "academically interesting" is using a separate MOC with custom fetch parameters.
However, before I went too far down either path, I'd want to know what's so wrong with querying the count of objects. It's actually quite fast.
Until I had proof from Instruments that it's the bottleneck that's killing my app, I'd push for the simplest solution possible.
Related
I am writing a scala script to download all items from the hacker news API. There are ~12M items, each being a JSON of ~200 bytes.
I identified the following issues:
Storing the data: I tried to save each item as a single JSON file, but it became very hard just to barely list them (using Linux, ext4 file system). So I changed it to just append JSON items to multiple (100) files (by taking the item's id module 100).
Keeping track of what has been downloaded, because I want to be able to stop/continue the application. First I tried writing the downloaded ids to a textfile, but it turned out a little bit buggy. So now I just read all the items and collect the ids. (It works.)
All this is done with 1 Master actor and an arbitrary number of Worker actors (tens). The Master has a Queue[Int] and pops it and Workers ask for work.
The problem I am having is fairly simple but I haven't been able to solve it in a nice way.
I can collect the ids from items already downloaded in a list. But what I really need is the complement to that set; I need all the items I have not downloaded, up to the highest item id.
I tried using a range (1 to maxItemId) and subtracting the set of done jobs but it is really slow. reaaaaaaally slow.
Now I am using a Stream, and when a worker asks for a job, I check if the stream's (the next job) has already been done. If so, I give it to the Worker. Otherwise I check the next one.
The problem with this approach is that I can not put jobs back at the stream if they fail. That would be easy with the Queue; but then again I am having trouble just setting up the queue with millions of items.
What could be a better approach to this? I don't think the issues here are trivial, this is a very large number of tasks to perform and keep track of, but it shouldn't be so hard as well.
Thanks!
As far as I understood your question, I think you don't need a very complicated data structure here.
Assuming your ids are sequential from 1 to maxItemId, you can use an array of Boolean with maxItemId size to keep track of processed items. You initialize this array by reading the processed ids. And you find the next job by searching for the next false entry.
Assuming that your maxItemId is around 12M, iterating over all items is pretty much instantaneous.
at the heart of it, my app will ask the user for a bunch of numbers, store them via core data, and then my app is responsible for showing the user the average of all these numbers.
So what I figure I should do is that after the user inputs a new number, I could fire up a new thread, fetch all the objects in a NSFetchDescription instance and call it on my NSManagedObjectContext, do the proper calculations, and then update the UI on the main thread.
I'm aware that the rule for concurrency in Core Data is one thread per NSManagedObjectContext instance so what I want to know is, do you I think can what I just described without having my app explode 5 months down the line? I just don't think it's necessary to instantiate a whole a new context just to do some measly calculations...
Based on what you have described, why not just store the numbers as they are entered into a CoreData model and also into an NSMutableArray? It seems as though you are storing these for future retrieval in case someone needs to look at (and maybe modify) a previous calculation. Under that scenario, there is no need to do a fetch after a current set of numbers is entered. Just use a mutable array and populate it with all the numbers for the current calculation. As a number is entered, save it to the model AND to the array. When the user is ready to see the average, do the math on the numbers in the already populated array. If the user wants to modify a previous calculation, retrieve those numbers into an array and work from there.
Bottom line is that you shouldn't need to work with multiple threads and merging Contexts unless you are populating a model from a large data set (like initial seeding of a phonebook, etc). Modifying a Context and calling save on that context is a very fast thing for such a small change as you are describing.
I would say you may want to do some testing, especially in regard to the size of the data set. if it is pretty small, the sqlite calls are pretty fast so you may get away with doing in on the main queue. But if it is going to take some time, then it would be wise to get it off the main thread.
Apple introduced the concept of parent and child managed object contexts in 2011 to make using MO contexts on different threads easier. you may want to check out the WWDC videos on Core Data.
You can use NSExpression with you fetch to get really high performance functions like min, max, average, etc. here is a good link. There are examples on SO
http://useyourloaf.com/blog/2012/01/19/core-data-queries-using-expressions.html
Good luck!
I'm working on an app that uses Core Data and NSFetchedResultsController. A major component of the app is the filtering down items in an indexed table view based on a set of 15 or so pre-defined switches that correspond to a property or relationship of my Managed Objects. In most of my situations, I'm searching through a set of around 300-400 objects, so caching/performance is not an issue. Everything is very snappy with no caching required.
However, there is a part of my app that basically searches through all objects in my CD database (~15,000 items). Here, I'm trying to implement caching on the NSFetchedResultsController to improve performance. The 'cacheString' property for the NSFetchedResultsController is simply the predicate's string value. Whenever the user toggles a filter switch, I create a new predicate, create a new NSFetchedResultsController, and set the cache to the new predicate's string value. The first hit to get all the items (unfiltered) takes ~7 seconds, with subsequent hits taking less than one.
What's strange, though - and here's my problem - is that once I proceed to the 'next step' of the table view (I push a new view controller to the nav controller, passing it a reference to the NSFetchedResultsController's fetchedObjects), performance drops considerably. This next view is essentially a different representation (a horizontally paging scroll view) of the previous view's table list with one item on the screen at once. When I page from one item to the next, accessing the previous or next object in the fetchedObjects array locks up the phone for about 5 seconds. The 'lock up' duration increases the further you go into the fetchedObjects array. If 'i == 0', there is no perceivable lag. If 'i == 10,000', it takes about 15 seconds to access the next object. Nuts! If I disable caching (or it's a query that wasn't cached so it needed to pull fresh results), everything except for the initial filter query is fast and snappy with zero lag.
Does enabling caching ONLY cache indexing info for a table view and not the fetched objects themselves?
I'm not sure what the deal is here. I hope I explained this well enough - let me know if you want to see some code or need additional info.
Thanks!
Billy
Alright, I've found out what my problem was...
Basically, asking my NSFetchedResultsController for a managedObject via objectAtIndexPath: is IMMENSELY faster than going directly to the fetchedObjects array and asking for objectAtIndex: (which, of course, is what I was doing), especially as your index gets into the thousands. I'm not sure 100% why that is, though. I'm guessing NSFetchedResultsController does some fancy stuff to efficiently pull out single objects rather than going straight to the raw data. So, I don't think the caching had anything to do with my performance issue.
Thanks to those who checked out my question. I hope this helps anyone else having similar issues.
I have a core data application and I would like to get results from the db, based on certain parameters. For example if I want to grab only the events that occured in the last week, and the events that occured in the last month. Is it better to do a fetch for the whole entity and then work with that result array, to create arrays out of that for each situation, or is it better to use predicates and make multiple fetches?
The answer depends on a lot of factors. I'd recommend perusing the documentation's description of the various store types. If you use the SQLite store type, for example, it's far more efficient to make proper use of date range predicates and fetch only those in the given range.
Conversely, say you use a non-standard attribute like searching for a substring in an encrypted string - you'll have to pull everything in, decrypt the strings, do your search, and note the matches.
On the far end of the spectrum, you have the binary store type, which means the whole thing will always be pulled into memory regardless of what kind of fetches you might do.
You'll need to describe your managed object model and the types of fetches you plan to do in order to get a more specific answer.
SqlDataReader is a faster way to process the stored procedure. What are some of the advantage/disadvantages of using SQLDataReader?
I assume you mean "instead of loading the results into a DataTable"?
Advantages: you're in control of how the data is loaded. You can ask for specific data types, and you don't end up loading the whole set of data into memory all at the same time unless you want to. Basically, if you want the data but don't need a data table (e.g. you're going to populate your own kind of collection) you don't get the overhead of the intermediate step.
Disadvantages: you're in control of how the data is loaded, which means it's easier to make a mistake and there's more work to do.
What's your use case here? Do you have a good reason to believe that the overhead of using a normal (or strongly typed) data table is significantly hurting performance? I'd only use SqlDataReader directly if I had a good reason to do so.
The key advantage is obviously speed - that's the main reason you'd choose a SQLDataReader.
One potential disadvantage not already mentioned is that the SQLDataReader is forward only, so you can only go through the records once in sequence - that's one of the things that allows it to be so fast. In many cases that's fine but if you need to iterate over the records more than once or add/edit/delete data you'll need to use one of the alternatives.
It also remains connected until you've worked through all the records and close the reader (of course, you can opt to close it earlier, but then you can't access any of the remaining records). If you're going to perform any lengthy processing on the records as you iterate over them, you may find that you impact other connections to the database.
It depends on what you need to do. If you get back a page of results from the database (say 20 records), it would be better to use a data adapter to fill a DataSet, and bind that to something in the UI.
But if you need to process many records, 1 at a time, use SqlDataReader.
Advantages: Faster, less memory.
Disadvantages: Must remain connected, must remember to close the reader.
The data might not be concluesive and you are not in control of your actions that why the milk man down the road has always got to carry data with him or else they gona get cracked by the data and the policeman will not carry any data because they think that is wrong to keep other people's data and its wrong to do so. There is a girl who lives in Sheffield and she loves to go out and play most the times that she s in the house that is why I dont like to talk to her because her parents and her other fwends got taken to peace gardens thats a place that everyone likes to sing and stay. usually famous Celebs get to hang aroun dthere but there are always top security because we dont want to get skanked down them ends. KK see u now I need 2 go and chill in the west end PEACE!!!£"$$$ Made of MOney MAN$$$$