i am fiddling to make my own storage engine, it is mostly for my own education.
so far i have a very good working solution with memory mapped files, but something is bothering me and i dont want to get bit by this later. i am basically mapping a view of my whole file and only create a new view if i have to resize the file. i know that this approach has one obvious drawback: maximum size, for now but i can live with that, i really dont intend to store data exceeding gigabytes.
but are there other considerations? how does this one large mapping interact with the os memory management routines, what about the long run? are there performance and fragmentation issues that i could run into by using this approach?
using smaller views over areas of the file i need will certainly remove the limit of the size of the database i can make, but then that is an extra layer of management i have to do in my storage engine, that i would rather avoid if i could.
i have only just scratched the top of an iceberg, and if someone could be my radar i could surely avoid some fatal crash!
Related
I would like to ask when to use ECS. My logic is telling me, that I should use it only when I need to control a larger amount of objects. Am I right, or should I use it everywhere?
My logic is telling me, that I should use it only when I need to
control a larger amount of objects.
Not really but it helps when it's used to control many objects. This is not the only or main objective of ECS. See the list reasons why ECS should be used below:
Performance
Memory Management
Build Size
It's not only used when you are working with many objects, it's also used to manage the memory. It reduces the amount of memory required when ECS is used than when the Unity's component system is used. Also, it reduces the build size too. With ECS, you can reduce the build size of your project. ECS makes it possible to use Unity's API as modules, You can decide which modules to include in the build therefore reducing the project size. For example, you can decide to remove the physics system if you don't need it in your project.
Finally, ECS is really important when it comes to building lightwight projects. For example, if you want to build a lightweight game or interactive program that would run on other smaller devices as an Ad, ECS should be used here since it will reduce the size of the Ad, loading time, the memory footprint. You should watch this video and visit the Unity's ECS page to learn more about it.
So,when should you use ECS?
When you need performance, want to conserve memory, reduce the build size of your project or create a lightweight program. Of-course there are many other uses of it. It's still new and we will find out more about it very soon.
I have an iphone app with a large(6-7MB) keyvalue file that I would like to avoid loading into memory. It is used very rarely, and chewing up that amount of RAM seems unnecessary.
Right now it is a flat text file with lines in the format
KEY VALUE
but I am happy to store it however works best.
What is the best approach for this? SQLite is an option, but that seems quite a heavyweight solution. I'm told that coredata is a good interface to SQLite, but it seems to me that this would still involve loading the entire file to memory as an nsdictaionary
First, SQLite is not a "heavyweight" solution. Its memory requirements are quite low, and its performance is good. If you use SQLite, you'll probably want a wrapper like FMDB or PLDatabase to make it easier to use in Objective-C.
Next, Core Data is not a wrapper for SQLite. It uses SQLite internally, but that's an implementation detail that is not exposed in the Core Data API. Core Data would work in this situation, but if you think of it in SQL-style terms then I guarantee that you'll screw it up.
If you're only using the table rarely, the simplest approach would be to save it as JSON or as a property list and just read the whole thing on demand. As long as you're careful to unload it when you're done and you're not already pushing memory limits, I wouldn't worry about it. If you might get to where you need it more often, consider either SQLite or Core Data-- either will allow you to look up the values you need without loading the whole thing into memory.
How would you search for a key witout loading all keys into memory?
You could implement your own indexing scheme, or just use SQLite (whose code is likely to be already in memory because of some other app or service).
Six or seven MB isn't big. Thus you could just load it into a dictionary and then when you are through ensure that it is deleted/unlinked. Alternately you could just write your own query and read the file lines yourself scanning for the key. But 6-7 MB isn't big.
If you have control of the file content, just use the existing Apple frameworks to write it as a plist and then to read in the plist as a dictionary and do your queries.
I have some doubts about SQLite...
My app uses only one table (encapsulated in a data logic class) to store small data, the connection is opened in didFinishLaunchingWithOptions and finalized in applicationWillTerminate (yes, I know that in iOS4 applicationWillTerminate is only called by iSO if necessary, but I write on database at every change of data).
Now I have to add a second table (and relative class) that will store different data (no join between the two tables) and of normal size (about 1-2 rows for day).
I thought of applying the singleton pattern to share the connection and the statements, but I read that for performance reason it's recommended to use class-local variables to hold the connection.
So what's the best practice? Singleton or two open connection?
What solution would you reccommend for my situation?
Thanks
Singleton for me when stored in the same database.
Small amounts of data shouldn't be a performance bottleneck anytime soon.
Or, of course, go with CoreData. :-)
For such a simple use, a singleton is probably the right answer for the very reasons that #Eiko mentions.
However, it all begs the question: Why aren't you using Core Data?
Using SQLite correctly is actually quite hard and I've seen dozens of very talented/experienced engineers get it entirely wrong without realizing it. Worse, scaling an existing implementation is even harder. Adding concurrency is really really hard with straight SQLite (and the Core Data engineers have expended a huge amount of energy and applied a ton of expertise to support concurrency on top of SQLite correctly).
For such a simple use, Core Data won't be hard to learn and it will leave you with a much more solid and versatile code base.
I have a rather simple iPhone app consisting of numerous views containing a single, grouped table view. These views are held together in navigation controllers which are grouped in a tab bar. Simple stuff.
My table views do little more than list text (like "Dog", "Cat" and "Weasel") and this data is being served from a collection of plists. It's perhaps worth mentioning too that these tables are 'static' in the sense that their data is pre-determined and will only ever be amended—and if so, very rarely indeed—by the developer (in this case, moi).
This rudimentary approach has reached its limits though, and I think I'm going to need something a bit more relational. I have worked a tad with Core Data in the past, but only with apps whose data is determined by user input.
I have four closely related questions:
Is Core Data overkill for an app consisting mainly of a selection of simple table views?
Do you recommend using Core Data to manage data which is predetermine and extremely unlikely to ever change?
Can one lock Core Data down so that its data can't change, thereby relinquishing my responsibility as the developer to handle the editing and saving of the managed object context?
How do I go about giving Core Data my predetermined data, and in a format I know that it can work with?
Thanks a bunch guys.
The answer is simple. If you do not need to persistent to an out of date format (like MSWord, etc.) then you should be using Core Data. Raw SQLite is a headache that is not worth the effort 99.999% of the time.
Core Data is more efficient than plists and allows greater flexibility if the project ever evolves.
It is also very easy to pre-populate a Core Data sqlite file using a OS X machine; you know, the machine you are using to develop your application in the first place :)
NNW's use case is a singular exception to this rule that, if I were a betting man, I would bet has the Core Data team's attention and will be corrected in a future update. An update, by the way, that you will get for free if you use Core Data.
You might consider using the SQLite API directly, rather than Core Data, as it may be easier to pre-populate a database that way. You can create and modify a SQLite database on any platform (Mac, Windows, Linux), and just copy it to your application's bundle as a resource.
You can find tutorials/examples that will create a user's database by copying a SQLite database out of the application bundle. In your case, you can just use the one in the bundle. Just be sure to open it read-only.
I would recommend sticking with plists since your data will rarely change and when it does it will be developer-driven.
Core Data is very powerful, but there will be a moderate amount of plumbing and infrastructure you'll need to set up to make it work.
Core Data places its store outside of your app bundle (as it must to run on the iPhone), so all new installations will need to load data into the store on the first run. This data will probably have to be stored as resource plists anyway, so you aren't saving yourself the trouble of generating those plists. Turns out that wasn't true and you can store read-only parts of the persistent store in the App bundle.
Since I don't know exactly what kinds of limitations you're running in too, Core Data may be the solution, but I'm guessing it won't be. If object relationships are the biggest difficulty you're dealing with, you should read up on object archiving as a way to store your entire object tree in a form that can easily be saved as a resource in your bundle and recreated when necessary.
You should read why NetNewsWire switched
The two main takeaways from that post:
I bet Core Data is the right way to go 95% of the time. Or more. It’s easy to work with. It’s fast (in most cases).
And:
(Rule: always work at the highest level possible.)
I recommend using plain SQLite. It is simpler, easier to maintain and you can build the database on non-mac systems, using many popular GUI editors. For me, using Core data is still a pain. Code to populate your tableviews from a static SQlite database is simple, straightforward, and transparent, while Core Data needs more boilerplate code which you don't even understand completely at first and overall has a lot of hidden complexity which you don't really need.
I'm displaying a few images that are being downloaded over a network connection on a UITableView. Loading these images is extremely slow. Are there any performance tricks I can use?
If you are displaying images from the network, they should be cached for any sort of reasonable performance. Consider the built-in App Store application: it only loads images for table cells that are on the screen, but after an image is loaded the application stores the image for later.
Also, parsing XML on the iPhone is going to be slow--especially with binary data embedded. You should serve images to your app as PNG/JPEG over HTTP for best results.
What part is slow -- the network, or the drawing?
If it's the network, unless you control the source of the data (e.g. maybe the Web server hosting these images), there's not much you can do, unless you can switch servers somehow (say, from a small Web site to using Amazon's cloud technology).
If it's the drawing, you can use Core Graphics for drawing your views, if you're just using UIImage/UIImageViews, and you're certain that's the bottleneck, as it's a bit more work. You should also be (at the very least) caching the data you're downloading.
Edit: Have you profiled your code to see what's slow? That's always the first step in optimizing anything; measure, then optimize. If you're parsing XML, that could very well be the bottleneck, but there's no way to tell until you profile. As you can see, it's hard to tell someone how to speed up performance without knowing what the problem is first.