I have an app with a very large Core Data database. I have versioned it many times over the past year.
The last time I versioned the database I made one simple change to an entity: I added a new optional attribute. For some reason it would not migrate using Light-weight Migration. I found out much later that this was due to a bug in Apple's Light-weight Migration code resulting from the 'renaming identifiers' that I had needed back in another versioning.
Anyway, I digress...
Because of the bug that kept me from using Light-weight migration, I created a mapping file to help with the migration, not understanding that this would was a much heavier process and would force my users to wait while the app loaded the entire database into memory while doing the migration. It turns out that this is not really an option at all with very large databases and many of my users were unable to migrate the database at all due to memory problems, etc.
So now I want to re-release my app and clear up this problem. The trouble is, some of my users have a database that is somehow marked as being 'in the middle of migrating'. Even with my new code, which gets rid of the mapping file and supports Light-weight migration, users that are in this state, 'in the middle of a migration', don't seem to get reset.
What are my options for backing out a migration?
- I can detect that I am in this state because there is a '.myDB.sqlite.migrationdestination_41b5a6b5c6e848c462a8480cd24caef3' file in the Documents directory. Deleting this file does not clear up the migration. My guess is that the database is somehow flagged as being in this state, or is already partially migrated.
- I can detect this state and then delete the database altogether. But this forces my users to re-download their data.
Any Thoughts?
Thanks for you help.
The only thing that occurs to me would be crack open the SQL store of an affected file and look for flags or something else that might signal the db being in a transitory state. You might be able to write directly to the file and alter something.
That's really ugly problem.
Related
I have a CoreData sqlite files *.sqlite *.sqlite-shm *.sqlite-wal in a zip format from my previous application version. The file is almost few hundred MBs.
In recent version, I have done some changes in the database schema i-e adding few new fields etc.
In Importing, I am simply replacing the current database by old database and making the user to restart application, which is crude. It does work however it causes 2 issues
1- It takes a lot of time on splash screen hanging the application.
2- If the database is big enough, the hanging time passes the Timeout
of Application and closes the application automatically.
What is the better way of importing database into core data saved in Documents as zip file.
First detect whether you are doing a migration. If you are, then display a ViewController with a spinner that explains what you are you doing ("please wait while we optimize the app"). When the database migration is done, the load your regular viewController.
This exact issue is discussed in this lecture: Core Data Potpourri (Paul Goracke, February 13, 2014) around 58:00. While the lecture is a bit dated (it was made before NSPersistentContainer) it is still one of the best I have ever seen and it is worthwhile to watch the entire video.
When I'd modified or create new tables in Liferay 6.1 and deploy in production server. Liferay automatically makes a backup of each table.
This backup takes a long time when a table has more than 10k records. And a century when has 100k. Although this table hasn't been modified.
What can I do for optimice the deployment to the server?
Many thanks in advance,
At the moment I think only two options are available:
(easy way) Set "build.auto.upgrade=false" in /WEB-INF/src/service.properties to avoid any automatic updates, and perform the db changes (if any) manually.
(hard way) Reworite the Liferays ServiceBuilder so that it perform an update only on those tables which were changed. This will require an EXT development as it is a very core change, and for every next Liferay version you will need to carefully review it and upgrade.
Liferay automatically makes a backup? Where to? This is new to me!
Also, you're describing a "no go" operation: You don't modify any tables in the database. Period. That's what the API is there for. If you do, prepare for disaster, sooner (if you're lucky) or later (according to Murphy, when you forgot that you changed manually. Then you'll blame Liferay for the failure that you caused by manipulating the database).
Do you have your own backup routine implemented that runs on every server restart? This is the only thing that I can imagine to happen here - in that case you'll need to modify your backup strategy. Or the database that you use - maybe a transaction log backup makes more sense than duplicating the table content into the same database...
I'm setting up a basic sync service for an iPad application I'm developing. The goal is to have data consistent throughout several instances of the iPad app, as well as having a read-only version of the data on the web, hence rolling a custom solution.
The current flow is this:
Each entity has a 'created', 'modified' and 'UUID' field which are automatically updated by Core Data
On sync, each entity with a created or modified date after the last sync date is serialised into JSON and sent to the server
The server persists any changes to a MySQL database using the client-generated UUIDs as PKs (if there's a conflict, it just uses the most recently modified entity as the 'true' version, nothing fancy there) and sends back any updated entities to the client
The client then merges these changes back into its Core Data DB
This all seems to be working fine. My problem is how to track deleted objects using this method? I'm guessing I can add a 'deleted' flag to each entity and set this whenever a client deletes something, I can then push that change to the server with the rest of the sync data. Once the sync is complete then the client can actually delete these entities. My questions are:
Can I override Core Data's delete methods to automatically set this flag?
Will this require keeping all deleted entities indefinitely on the server? We'll have no way of knowing when every client has synced and actually deleted each entity (I'm not currently tracking client instances)
Is there a better way of doing this?
How about you keep a delta history table with UUID and created/updated/deleted field, maybe with a revision number for each update? So you keep a small check list of changes since your last successful sync.
That way, if you delete an object you could add an entry in the delta history table with the deleted UUID and mark it deleted. Same with created and updated objects, you only need to check the delta table to see what items you the server needs to delete, update, create, etc. You could even store every revision on the server to support rolling back to a previous version in the future if you feel like it.
I think a revision number is better than relying on client's clock that could potentially be changed manually.
You could use NSManagedObjectContext's insertedObjects, updatedObjects, deletedObjects methods to create the delta objects before every save procedure :)
My 2 cents
Whether or not you have to keep deleted objects on the server or not totally depends on your needs. You will need a deleted flag locally to mark as deleted for the sync, maybe also on the server depending on your desire to roll back.
I have taken care of this problem a few ways before. Here is one possibility:
When a client deletes something, just mark it to be deleted locally and delete from the server during the sync (at which point you can purge from core data). When other clients request to access that data, send back an HTTP 404 because you dont have the object any more. At that point the client can delete the entity locally. Now if a client requests a list of things and this object has been deleted, it will just be missing from the list of things he gets back so you can detect that and delete it. I do that in a client by creating an array of object IDs when I get a response from the server and deleting any local objects that don't have those IDs.
We have a deleted field on the server, but just to have the ability to roll back in case something is deleted by accident.
Of course you could return deleted objects to the client so they know to delete but if you don't want to keep a copy on the server, you would have to make some assumption that the clients would all update within a time frame. Then you could garbage collect after that time frame has expired.
I don't really like that solution though. If your data is too heavy to ask for all the objects for a complete sync, you could use your current merge strategy for creating and updating, and then run a separate call to check for deleted items. That call could simply ask for all IDs that the client should have on the device. It could delete the ones that don't exist. OR it could send all IDs on the client and get back a list of IDs to delete.
I think you have to provide more details about the nature of the data if you want a more opinionated suggestion.
Regarding your second question: You can design this so that the server doesn't have to keep deleted records around, if you want to. Let each app know if a given piece of data (based on its UUID) is stored on the server (e.g. add an existsOnServer property or similar). This starts out false when a new item is created in the app, but is set to true once it has been synced to the server for the first time. That way, if the app tries to sync later, but the UUID is not found, you can differentiate the two cases: If existsOnServer is false, then then this item is newly created and should be synced to the server, but if it is true then it can be taken to mean that it was already on the sever before, but has now been deleted, so you can delete it in the app too.
I'd probably argue against this approach, since it seems more error prone to me (I imagine a database or connection error incorrectly being interpreted as a deletion) and keeping records around on your server would usually not be a big deal, but it is possible. The "delta-approach" suggested by dzeikei could be used at the same time, so an update to a record that does not exist on the server signifies that it was deleted, while an insert does not.
You may take a look at Cross-Platform Data Synchronization by Dan Grover if you haven't. It's a very well written paper regarding synchronization and iOS.
About your questions:
You can avoid deleting a file in Core Data and set a 'deleted flag': just update the file instead of deleting it. You could make your own 'deleting' method that actually would call and update the flag on the record.
Keep always a last_sync and a last_updated for each record on the server and on each client. This way you'll always know when someone did change something anywhere and if that change was synced or not against the 'truth database'.
Keeping track of deleted files is a hard thing to do, I guess the best way to do it is keeping track of the history of syncs for each table, but is a difficult task. The easiest way, using this 'truth-database' kind of configuration is to flag the files, so that way yes, you should keep the data on the server as well as on the client.
during synchronization of data between tow table some records or deleted when the table rows are same. and when the rows are different the correctly synchronized, i used this Code click here on image
Short version
If my process is terminated in the middle of a transaction, or while SQLite is committing a transaction, what are the chances that the database file will be corrupted?
Long version
My application uses an SQLite database for storage (directly, not via Core Data). I'm working on a new version of the application which will require an update to the database schema. On launch, the app will check the database and, if it needs updating, execute a series of SQL statements to do so.
Depending on the amount of data in the database, the update may be long running (on the order of seconds), so I need to consider the possibility that the process may be terminated before the update is completed. (For context, this on an iPhone, where the processor is slow and the app may be terminated by an incoming phone call.) I will, of course, wrap the upgrade SQL statements in a transaction. Will that be enough to guarantee that the database will not be corrupted?
I'm assuming that transactions work as advertised, and that if the process is terminated in the middle of the transaction, the file will be OK. But I'm also assuming there is a window of time during the COMMIT where something can go wrong.
To play it safe, I could create a backup copy of the database file before starting the update, but if the transactions are safe then that would be overkill. It would also make the update process take longer, which increases the chance it would be interrupted, and then I'd have to consider that the file copy operation might be interrupted... I'd like to keep the code as simple as possible (but no simpler).
In the course of researching this question I've started reading "Atomic Commit In SQLite", which is more detail than I probably need to know, but is giving me faith that I don't need to second-guess SQLite's ability to protect the database file. But I'd still like to hear from Stack Overflow: is a transaction good enough, or should I be more cautious?
I have read the Atomic Commit in SQLite document. It may not be overkill if you really want to understand what's going on, but in a nutshell, a transaction goes like this:
Lock the database file
Create the rollback journal
Determine what portions of the database file are going to be changing
Write copies of those pages to the journal file
Write the journal file header
Write your intended changes to the database file
Delete the rollback journal (THIS IS THE COMMIT)
When the user is done talking to mom and re-starts your app, when it tries to open the database file, if there is a rollback journal present, it will write the original data back to the datafile using a similarly safe process. Even if you lose your transaction, and lose a rollback, it will eventually be taken care of once mom's nervous breakdown is properly thwarted and he can run the app for more than a couple seconds at a time.
If it were me, I would trust the transactions. With so many users of SQLite, even in embedded apps, I think transaction commit failures would be a very hot topic all over the net if they weren't working properly.
Are you using CoreData with a SQLite backend? If so, I actually find that the best way to handle this problem is to create two separate NSManagedObjectContexts (a read-only and an editing). When the process completes, just save the "editing" context and then the two contexts will be in sync. If something happens during your operation, the editing context won't get saved, so you'll be fine.
i know this is an often asked question on these boards. And usually the question has been about how to manage the changes being made to the database before you even get around to deploy them.Mostly the answer has been to script the database and save it under sourcecontrol and then any additional updates are saved as scripts under version control too.(ex. Tool to upgrade SQL Express database after deployment)
my question is when is it best to apply the database updates , in the installer or when the new version first runs and connects to the database? note this is a WinApp that is deployed to customers each have their own databases.
One thing to add to the script: Back up the database (or at least the tables you're changing!) before applying the changes.
As a user I think I'd prefer it happens during the install, and going a little further that the installer can roll itself back in the event of a failure. My thinking here is that if I am installing an update, I'd like to know when the update is done that it actually is done and has succeeded. I don't want a message coming up the next time I run it informing me that something failed and I've potentially lost all my data. I would assume that a system admin would probably also appreciate install time feedback (of course, that doesn't matter if your web app isn't something that will be installed on a network). Also, as ראובן said, backing up the database would be a nice convenience.
You haven't said much about the architecture of the application, but since an installer is involved I assume it's a client/server application.
If you have a server installer, that's where you want to put it, since the database structure is only going to change once. Since the client installers are going to need to know about the change, it would be nice to have a way to detect the database version change, and for the old client to be able to download the client update from the server automatically and apply it.
If you only have a client installer, I still think it's better to put it there (maybe as a custom action that fires off the executable for updating the database). But it really isn't going to matter, because conceptually one installer or first-time user of the new version is going to have to fire off the changes to the database anyway. The database changes are going to put structural locks on the database so, in practical terms, everyone is going to have to be kicked off the system at that time for the database update to be applied.
Of course, this is all BS if it's not client-server.