My app has a potentially very large CoreData datastore underneath it (could easily be upwards of 30MB). I've started noticing memory issues when using the automatic migration (addPersistentStoreWithType:configuration:URL:options:error:) so I started looking into the methods of migrating smaller parts of the store to avoid all the CoreData object buildup that happens when you migrate everything at once.
This is discussed in the official documentation in the "Multiple Passes" section, however it looks like their suggested approach is to divide up your migration by entity type, i.e. make multiple mapping models, each of which migrate a subset of the entity types from the complete data model.
The only problem is - what if one entity type is the majority of your datastore? Going by the Apple-recommended approach, that whole entity type is still going to be done in a single migration and the memory issues will presumably persist.
Are there any techniques available to actually migrate a sub-set of entities of a specific type to guarantee that you will not run out of memory when trying to migrate them all?
Thanks in advance for any help.
EDIT: After doing more digging, I have discovered that the Apple-recommended split of the DB into entity types actually only works for non-related entities (as discussed here), so it's even less likely to solve the problems of a real-world DB than I thought when I originally wrote this post.
I'm starting to think that CoreData migrations that are actually done through NSMigrationManager don't scale at all now and you basically can't have a DB that is bigger than about 20-30MB if you want to be able to migrate it on current generation iOS devices. The only viable approach seems to be to short-circuit all the NSMigrationManager / NSMappingModel stuff completely and write the migration completely custom in code. Seems like a huge oversight on Apple's part if this is actually the case.
I was able to get around this in the short-term by leveraging "lightweight" migration, as described in http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/CoreDataVersioning/Articles/vmLightweight.html#//apple_ref/doc/uid/TP40008426-DontLinkElementID_1.
The trick is to pull the NSMigrationManager subclass that is specifically for SQLite store types when you're manually invoking the migration, as shown in the last code sample on that page.
This isn't a general-purpose fix though since it only works if the schema change in your datastore is simple enough that lightweight migration is possible. Still waiting to hear back from Apple as to what the recommended solution is when you're dealing with a non-trivial mapping.
Related
I've spent the last couple of months on and off with CFwhees.
There's a lot it has going for simplicity, but I always come up against an issue or two that I have a hard time troubleshooting. Biggest of them is to get a many-many relationship to work properly.
I'm switching between two environments Railo/mySQL and CF/MsSQL so it would be nice if it could work on both.
I'm trying to roll out a web application in a limited amount of time, as I've spent already too much time on CF wheels.
Can anyone recommend a framework that will make creating many-many relationships and the related CRUD easy and has a big community?
Some of the one's I've seen mentioned frequently are MachII, FuseBox, Model-Glue, ColdBox
Most of those frameworks you list do not have built-in ORM like wheels does. This means you'll be either using straight SQL queries or the CF9(Hibernate) ORM. I think it is only fair to mention that both of those options are also available for CFWheels.
I've written a fairly large app in CFWheels. In my app there were several instances of many-to-many relationships, and I was able to make it work without too much pain. That being said, I have felt your frustration with the CFWheels ORM. It can be clunky once you get to complex relationships. In those cases, I've had to make a judgement call as to whether it was worth it to try to build a query using the ORM, or just build a custom SQL query and store it in the CFC for my model. In fact, for 99% of my report queries for this app, I just resorted to writing the SQL in the model. But for CRUD operations, this wasn't really a limiting factor.
I'm curious what specific problems you're experiencing with Wheels - care to post an example?
yes the orm in cfwheels can be buggy at times. if you encounter a bug or even what you THINK might be a bug, we want to know. please take the time to file a bug report so we can investigate. that all being said, i'm very surprised that the CF community hasn't taken noticed about Don Humphrey's ORM called CFRel. It's probably one of the biggest things to happen to CFML since fusebox.
Oh... and there is even a cfwheels plugin for it.
A few people have wrapped the SQLite library or provided alternatives. What are their relative merits?
Core Data
Yes, a bit of snarky answer.
There are three primary reasons to use SQLite directly or via a wrapper.
You are sharing databases across platforms and can't use Core Data's schema
You really do need raw SQLite performance and have the 17th level SQLite API Mastery required to actually achieve said performance advantage over SQLite.
You know SQLite inside and out, don't like learning new things, and want to re-invent the bits necessary to get between database and screen. (Slightly snarky again).
Core Data buys you a tremendous amount of functionality that is subtly very hard to do. That is, object graph management with full integrity and undo support.
bbum gave the most succinct answer and you should ponder carefully not using Core Data.
But I thought you also deserved an answer to your actual question.
There are basically two wrapper approaches I know of:
FMDB uses an approach that simply makes a much easier to use Cocoa API overlay for SQLLite. This can be great if you know SQL and database design well and just want a simple database.
Other approaches are generally more object-relational mapping systems, that attempt to give you an object view of an underlying datastore, and hide some queries from you.
In both cases if you have a really simple data store they can make sense to use if you have a specific reason... but using Core Data gives you an awful lot for free (though I'll admit the learning curve can be steep).
There is a third approach. This third approach allows you to build SQLite statements via a set of Objective-C classes. They will handle how your data is converted and sanitized. They will also ensure that your statements are correctly constructed. You can try https://github.com/ziminji/objective-c-sql-query-builder This particular library also offers Object Relational Mapping (ORM) that follows the Active Record design pattern.
I'm creating an iPhone app and I'm trying to choose between 2 solutions for a persistent store.
Core Data, or SQLitePersistentObjects. Basically, all my app needs is a way to store an array of model objects and then load them again to display in a UITableView. Its nothing too complicated. Core Data seems to have a much higher learning curve than the simple to use SQLitePersistentObjects. Are there any obvious benefits of using Core Data over SQLitePersistentObjects in my case?
As the author of SQLite Persistent Objects, I say: use Core Data.
I wrote SQLPO when Core Data didn't exist on the phone. Although I'm proud of what I did with SQLPO and even though I do like some things about its approach better than Core Data's (specifically not having to maintain separate class files and data model), the engine underlying Core Data is much more mature and has many more engineering hours invested in it. That's why I abandoned SQLPO development when Core Data came to the iPhone SDK.
I haven't done benchmarks, but I would guess that used correctly, Core Data is going to perform better in nearly all high-volume situations.
SQLPO is faster to develop with since all you do is create the header files, but unless your data needs are relatively light, I say you'd be better off using Core Data.
See this question. My answer to that question also applies to yours.
Core Data VS SQL Statement, which one is gd for iphone development?
Some infromation on my experience with SQLitePersistentObjects.
An app initially developed for iOS 3.x utilizing SQLPO works just fine. Easy to use etc.
Now I am in the process of bringing this app to iOS 4 and things started to get strange.
I now see DB corruptions at unpredictable rates.
Looking into the SQLPO code shows that there is only one sqlite3_close statement and this is called when the DB can not be opened.
I am planning to add a method to close the DB explicitely and call this from my app delegates terminate and for iOS4 didMovetoBackground methods.
Might help to avoid DB corruption issues with SQLPO.
I recently had to make the same decision. I was storing instances of a simple object with a couple of properties. From my research I understand that using Core Data will help you better manage more complex objects with multiple relationships. I ended up using Core Data only because I wanted to learn more about it (but for simple objects there wasn't much of a learning curve).
SQLitePersistentObjects aka SQLLite Persistent Objects is not the same as doing straight SQLite at all. It's an ORM in its own right. I haven't used it yet, but I wanted to correct the completely wrong answer that the previous poster gave.
And I'm seriously considering using because Core Data is a pain.
See: http://iphonedevelopment.blogspot.com/2008/08/sqlite-persistent-objects.html
I was surprised to find a public letter proposing a vote of no confidence in the entity framework (see http://efvote.wufoo.com/forms/ado-net-entity-framework-vote-of-no-confidence/)
Would the reasons stated in the letter keep you from using the current version of the entity framework? Would you rather wait for v4.0? Or rather use another ORM?
The current version of EF is definitely not perfect, and has lots of gotchas and drawbacks. I probably wouldn't use it right now - but the upgrade path to EF v2 (or is it EF4?) sure looks pretty rosy!
complete persistence ignorance - you can use your straight up POCO classes
deferred loading configurable as an option
much improved designer with support for pluralization/singularization (even in multiple languages!)
ability to do "domain first" design and create database from your model
ability to have self-tracking entities across multiple layers that allow you to send data to the client and get back changes and apply them to your entity context
All in all, EF v2 looks very promising and I'm very eager to give it a serious spin. If it really keeps all the promises out there right now, it's definitely a winner!
Check out the ADO.NET team blog for a flurry of recent blog posts on EF v2.
Marc
Another ORM.
Don't get me wrong you should get flamed with responses, but currently only nHibernate is functionally complete.
I'm a TDD fan, so want an easily testable POCO ORM solution. If that's your bag then EF3.5 is out. EF4.0 is introducing it (http://blogs.msdn.com/adonet/archive/2009/05/21/poco-in-the-entity-framework-part-1-the-experience.aspx) , but it still has at least 1 big drawback -> doesn't support inheritance.
NHibernate is more complete, but EF could be easier to use. As ever, best tool for the job... but if it's an Enterprise-scale TDD developed app, go nHibernate.
Also -> there's a profiler that makes nHibernate dev much easier -> http://www.nhprof.com/
I tried using it for my current project, which basically involves rewriting our current mess of a data layer.
It just doesn't work.
First, if you're trying to base an Entity off of a View, the designer tries to force every NOT NULL property to be an entity key... which is pretty much never what I wanted. To work around that you have to edit the xml in at least two places, and do it every time you add an object because it refreshes and re-adds the EntityKey properties. Must specify mapping for all key properties in Entity Framework?
Second, when you are creating associations you MUST use every entity key - How can you make an association without using all entity keys in entity framework?
Those two things held me up for 3 days, then I went back to Linq to SQL and had it done in a couple hours. (Well, at least the part of the system I was struggling with... ) I don't know if those are in the Vote of No Confidence, but it's just not ready in my opinion.
Also with the lack of answers I got here on every EF question I've asked, I have to assume current usage is so low that getting help and support is going to be difficult... which is possibly the BIGGEST reason not to use something.
Let's hope the next version is better...
EDIT: OUr current plan is to stick with Linq 2 SQL (I have to finish a project by Friday) and then evaluate all the other ORMs to see if anything else is better. The other developer hates L2S for the record, but I've never had any major problems using it...
EF has some rich design time support, but I have to agree that nHibernate is the way to go, despite the learning curve. If you need to make something fast and don't care about TDD or serialization (which is a large weakness of all of MS's ORM offerings) go EF.
Well My experience of Version 1 was interesting. I wanted to use POCOs but it didn't support it. After reading around I came across some code from a bod at microsoft that did this.
It was a bit messy to generate the code but on the whole this part of the process was not so bad.
A real nasty part that I came across was the lack of Concurrency checking built in, for N-tier development. You have to manage this yourself which after looking at the problem was not so bad, especially if you want to hand back the versioning back to the client for user intervention.
Second nasty and absolutely stupid thing missing was the IN keyword for LINQ queries. Not supported and so needs to be worked around. I found a solution but was a real mess bringing in some other code that quickly patched up the omissions.
Would I use EF 4.0 (2.0). Yes, absolutely, why not? In fact on stage 2 I will be using this. It looks like it supports POCOs, it looks like my concurrency model will move straight across with no problems (basically delta copy stuff). Its all good so far and I hope this time round that the Big guys at Microsoft have seen the errors of their ways and provided a solution that works.
If your buying into entity development and the whole Concept Model first thing, then its the only way to go for a complete Microsoft solution. Although the stuff being done on the M language might eclipse the idea and move the whole modelling thing back to the Database.
If you not buying into the Entity stuff then I would strongly go Enterprise Library. Its a proven technology that works every time built on a solid code foundation and Database centric paradigm. I would also go this route if you think that Stored Procedures are the bees knees and like what they bring to the table.
If your feeling really exotic and feel a bit frisky I would go with a NO-SQL approach such as CouchDB. This however does take some getting used to. Its damn weird and feels really really wrong. But things get developed in super quick time and the solutions seem to be robust and faster than expected. I would not got for this type of solution though if your big into Normalization and think that it can be applied to a NO-SQL approach. The whole model needs to be shifted on its head and the application will be needed to be modelled in a way that is driven by the technology applied.
I find the CouchDB way a bit dirty and very very wrong. But it has so many compelling reasons to use it, that I think it will seep into the psyche of every programmer, and it will definitely go mainstream in the next couple of years.
My biggest gripe still with the whole Entity thing though even in the new version 4 is that there really has not been much thought into N-tier environments. It still got a feel about it being a 2 tier solution with a lot of boiler plate code still needed to be done by the end user (developer), to get it working in a robust and dependable N-Tier way.
We been having some discussions on approaches to using the entity framework at work recently. We have a fairly large and complex n-tier web based application, which is due for a major overhaul.
The question is: If we where to starting using the entity framework, would it be better to create one big model, or a set of smaller functional/acivity based models.
I have my own opinions on this, but would be interested to hear what some other people think.
Update (17th November 2008):
I have been creating one model, wiping it out and re-creating, etc for small projects at home. Although I haven't tried, I suspect that this approach will be a bit more challenging when there are a large number of entity types involved.
Also, does anyone have any experience of using ef with a large team using TFS or similar?
In my experience with it, I would just make one big model of the database. Otherwise, it might be hard to track what tables changed where. When I make changes to the database, I just delete all the tables in the model and regenerate it.
Of course, I also didn't customize my model by adding "entity" functionality to it (not sure how that works exactly).
So I'm no expert in it, but I usually end up using the LINQ-To-SQL models/objects instead of the Entity Framework - it's worked better for me so far.