I am currently building my Core Data model, which I would like to sync between the Mac and iPhone versions of my application.
I will be using Bonjour for device discovery, etc but I have a question regarding the data sync part of the problem.
So far I have added a UID and modification timestamp to each object that will be involved in syncing, so I should be able to match up objects and detect which ones have changed.
Are there any good links, resources out there regarding writing sync code for this type of situation, i.e. syncing records between two instances of a model?
Sync is a problem with quite a few edge cases which have been solved many times by people in the past, so I was expecting to find some info on the subject but all I can find are links to Apple's SyncServices (which doesn't exist on iPhone) and some MS sync technology.
I'm really looking for general theory so I can implement it myself, not necessarily a ready-made solution.
The SyncML specification may be of help, but it's quite hard to read and obviously biased towards SyncML.
I've had to implement this for Task Coach, so here are a few ideas:
A modification flag is enough, a timestamp doesn't really provide much more information. Typically, my objects are in one of these states:
None
New
Deleted
Modified
The following transitions happen when the object is modified:
None -> Modified
New -> New
Deleted -> (should not happen)
Modified -> Modified
and the following ones when it is deleted:
None -> Deleted
New -> Actually deleted (it may be removed from storage)
Deleted -> (should not happen)
Modified -> Deleted
When synchronizing, the device first sends to the desktop all objects with a status different than None. The desktop asks the user to resolve conflicts if one of these has a status != None on its side. In any case, the object goes into state None on the device, or is deleted from storage if its state was Deleted.
Then, the desktop sends its own changes to the device. There are no conflicts possible since all objects are in state None on the device. Objects on the desktop go into state None or are deleted from storage as well, and sync is over.
There are two types of possible conflicts, depending on the device/desktop states:
modified/deleted. If the user chooses to trust the device, the desktop object is replaced with the device one; else, the desktop does nothing and keeps the deleted state, so that the object will be removed from the device in phase 2.
deleted/modified: If the device wins, the object is actually deleted from the desktop. Else, the object goes into state New on the desktop so that it is restored on the device in phase 2.
deleted/deleted: Duh. Just remove it from storage.
modified/modified: The user decides which values to keep, maybe on a field by field basis. The state stays to Modified on the desktop so that these choices are propagated back to the device in phase 2.
Some conflicts may be avoided if the Modified state is kept for each field, so that for instance an object with a modified Subject on the device and modified Summary on the desktop will not trigger a conflict.
You may take a look at the code for Task Coach for an example (SVN repository on SourceForge, it has both the desktop app in Python and the iPhone app). Actually, in this case I decided to use a simpler approach; I don't keep track of the state on the desktop. After phase 1 (device to desktop), I just make a full replacement of objects on the device with the ones on the desktop. Thus, there are no conflicts (the device always wins).
Obviously, this only works between two fixed devices; if you want to synchronize with several phones/desktop apps, you have to assign a unique ID to each and keep different states for different devices/apps. This may begin to get hairy.
HTH
Marcus Zarra created a framework called ZSync to simplify synching iPhone/iPad apps to their Mac counterparts. Take a look at it, it may help solve the problem.
Related
I have a Swift 1.2 app that uses Core Data with iCloud sync.
In the first screen, the user can insert some data to create custom MyNSManagedObjects.
Every MyNSManagedObject must have a specific "group" to which it belongs. This "category" is represented in my data model by another custom NSManagedObject, let's call it MyManagedObjectsCategory.
User can create many MyManagedObjectsCategory objects, but the app also needs a DEFAULT object of type MyManagedObjectsCategory, in case the user doesn't create any different MyManagedObjectsCategory.
Every MyNSManagedObjects can have only 1 MyManagedObjectsCategory, but a MyManagedObjectsCategory can have many MyNSManagedObjects.
When user launches the app, I immediately check if the DEFAULT MyManagedObjectsCategory already exists and if it doesn't I create a DEFAULT MyManagedObjectsCategory object and save it to persistent store. Of course, only the first time the app launches I need to create this object, later I always fetch the object I created on the first launch of the app and use that.
My issue started when I enabled iCloud sync; now, on the first launch of the app, this happens:
The app on launch uses STORAGE 1 (local) as expected and, not finding the DEFAULT object of type MyManagedObjectsCategory, it creates a new one.
If there's network coverage, a few seconds later the app switches to STORAGE 0 (cloud) and saves the DEFAULT MyManagedObjectsCategory object that has just been created; if the device is offline, this doesn't happen immediately, but of course it will occur when network connection becomes available later.
When I launch the app for the first time on a different device, the point 1 and 2 above happen again: since the app starts with storage 1, it doesn't fetch the DEFAULT MyManagedObjectsCategory object and it creates and saves a new one that, a few seconds later or when network is available, is synced to storage 0 as the app switches storage.
As you can imagine, when different devices sync I find myself with multiple DEFAULT objects and, since I'm new to Core Data, I have no idea how to manage this issue.
On one hand, I need the DEFAULT object immediately available when the app launches, so I can't wait the switch to storage 0 (also, because I don't know if the user has network connection, so the storage switch might happen much later); on the other hand, the purpose of the DEFAULT object is to be one, and always the same, on every device.
I understand that, even if every DEFAULT object has matching properties (the object has a name and a myID String property) being created in the exact same way, Core Data creates a unique ID for every managed object and, since the ID doesn't match between the DEFAULT objects created on different devices in different moments, it doesn't merge them in a single DEFAULT object.
So, my questions:
Is there a way to force this merge of the DEFAULT objects into a single one, if certain properties are exactly the same? Is so, how? I suppose I could do it when the app launches, since the duplication of the DEFAULT object would only happen when a new device is added to iCloud.
Is there a completely different way to handle this issue that I'm missing?
I've spent the last 2 months working on this app, but I can't ship something that duplicates objects when syncing, and I have no idea how to fix it, so any help would really, really be appreciated.
Thanks,
#cdf1982
This is fundamental to using iCloud, or really any sync mechanism. If your app creates the same instance on multiple devices, and can't sit around waiting to see if it already exists from a different device, then you'll get duplicates.
The only way to handle this is to let the duplicates happen and then clean them up. With iCloud, you do the cleanup when you receive NSPersistentStoreDidImportUbiquitousContentChangesNotification, indicating that new incoming data is available. The basic scheme is to do a fetch that finds all duplicates and then handles them according to your app's needs (merging/deleting/whatever). I described this in a previous answer and in some detail in a blog post.
You'll make this much, much easier on yourself if your category entity has an attribute that stores a unique ID, and you ensure that you always use the same unique ID value for your default category instance. Then you can simplify the de-duplication by fetching only objects that match the known unique ID value.
I am building an internal iOS application (so - it won't ever be in the app store), and I need to keep a directory of content synchronized between a server and each of the instances of the iOS application. This would be easy enough if I just wanted to delete and re-download this content each time, but I would rather use something similar to rsync to only download the elements that have changed.
I haven't found any good way to utilize rsync. I considered looking at Objective-Git as a possibility here, but at a quick glance it looked like there is still a lot of the support for remote repositories that isn't supported yet.
As a final note, while this won't be in the app store, I will not be jailbreaking these devices and I would prefer to not rely on any private API's (although if there was an elegant solution that utilized private API's I might consider it).
Thoughts?
ADDITIONAL NOTE: This needs to be an isolated solution. I won't be relying on outside services (like Dropbox, Box.net, etc...). This needs to work solely between the device and the server (which is on a local network with the device).
Use HTTP to list the contents of each folder on the server.
Compare last modification time of each file with those on the device, and identify added/removed files.
Get added and modified files, remove deleted files.
It sounds like you're maybe asking for a library that already does this, but if you don't find one it's obviously moderately easy to write this from the ground up using stat(2) on the server and the same or a higher-level equivalent on the iOS devices. Have the iPhone send a tree of files with their modification date to the server and get back a list of insert/delete/update operations to do with the url (or whatever) for each one so you can do them incrementally on a background thread. Have the information from the server for new/updated files include the mod date that the server has so you can set it to be the same on the iOS device and send that when asking the server for the status of each file (kind of hack using the file system to store that, but it works).
Why not just set up a RESTful interface and do it across HTTP; that way you could query the modification times easily enough to determine whether client or server files need to be updated. You might also want to keep track of what files on the client have been synced, so you can easily know which files to add or delete. This can be done with a simple .sync file or using a plist / sqlite / etc.
If you'll consider FTP, there are some pretty advanced client libraries available.
For example, the iOS Chilkat bundle includes an FTP client library that supports synchronization in both directions. It's not free, but it's pretty cheap -- and you get a ton of other stuff that will likely prove useful someday. Here's an example of iOS pulling down all additions and changes (mode 2):
http://www.example-code.com/ios/ftp_syncLocalTree.asp
One caveat -- judging solely from the example, it doesn't appear to synchronize deletions. If this is a requirement, you could do it yourself without too much effort immediately following a sync.
acrosync (see https://acrosync.com/library.html) seems like a good fit given the initial question, however I haven't used it myself yet.
I'm planning on developing a multiplayer strategy game for the iOS platform. However, being a strategy game with multiple "units", there will likely be imbalances in gameplay, that will need to be addressed with constant mechanic-tweaking (upgrading / nerfing certain units).
The easiest way to accomplish this would simply be to change the mechanics within the app itself, and constantly submit updates to Apple. However, updates take time to propagate through Apple's review process (so the changes wouldn't be instantaneous), and I would need to do checks to see if all the players in the game are running the same version of the game, force users to update to the latest version of the game, etc.
What I'm thinking of doing instead is something similar to what the game Uniwar does. Every time the game is launched, it appears to check if there are any gameplay tweaks available, and if there are, it downloads the updates (and shows an update message to the user detailing what has changed).
However, as a relatively new programmer, I don't really know what would be the easiest way of accomplishing this. Would I host a text file online containing the unit statistics, and get the game to check that file for changes? Or is there some better, more efficient way? And if I were to do it this way, how would I do it?
First, ensure that your rules are some sort of resource you can easily change (be it binary or text-based). The most convenient way of updating these would be to periodically poll a server, most conveniently using the HTTP protocol, fetching updates as needed. The way I see it, there are two ways of doing this.
The first method uses the excellent caching abilities of the HTTP protocol, and as such requires a server (and client library) that understands these. The basic idea would be to have a copy of the latest version of the game mechanics published on a server (say, to http://example.org/mechanics.gz, and then have the client issue conditional GET requests with the If-modified-since header set to the time the last update check was performed. The HTTP protcol will den effectively do the rest for you, issuing a 304 Not modified if there is no update, and sending the new mechanics if there is one. This method has the disadvantage that the whole mechanics file has to be downloaded on every update (no diffs can be used), and that old versions won't be available, but it has an appealing simplicity.
The second method would consist of having a list of updates (well, their URI, ID and release date), say http://example.org/updates.xml, which the client pulls on every poll. The client then checks if there are any updates it doesn't have, and downloads and applies these in chronological order. Using this method, old updates can be made available (and will have permanent links), and diffs may be used. This is useful if history is important or if game mechanic files are large.
The format of the file doesn't really matter -- use whatever works for you. The key to being able to tune the gameplay, I think, is to turn the rules of the game into objects that you can configure. If the rules are all objects, you can do things like changing the order in which they're applied, the weight given to each one, the conditions under which a rule becomes effective, etc. You might have an object that's responsible for managing and properly applying the rules, basically a "rule model." Once you have that, all you need to do is to implement NSCoding in your rule and rule model classes and you can easily write and read rule configurations.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I have been working on a method to sync core data stored in an iPhone application between multiple devices, such as an iPad or a Mac. There are not many (if any at all) sync frameworks for use with Core Data on iOS. However, I have been thinking about the following concept:
A change is made to the local core data store, and the change is saved. (a) If the device is online, it tries to send the changeset to the server, including the device ID of the device which sent the changeset. (b) If the changeset does not reach the server, or if the device is not online, the app will add the change set to a queue to send when it does come online.
The server, sitting in the cloud, merges the specific change sets it receives with its master database.
After a change set (or a queue of change sets) is merged on the cloud server, the server pushes all of those change sets to the other devices registered with the server using some sort of polling system. (I thought to use Apple's Push services, but apparently according to the comments this is not a workable system.)
Is there anything fancy that I need to be thinking about? I have looked at REST frameworks such as ObjectiveResource, Core Resource, and RestfulCoreData. Of course, these are all working with Ruby on Rails, which I am not tied to, but it's a place to start. The main requirements I have for my solution are:
Any changes should be sent in the background without pausing the main thread.
It should use as little bandwidth as possible.
I have thought about a number of the challenges:
Making sure that the object IDs for the different data stores on different devices are attached on the server. That is to say, I will have a table of object IDs and device IDs, which are tied via a reference to the object stored in the database. I will have a record (DatabaseId [unique to this table], ObjectId [unique to the item in the whole database], Datafield1, Datafield2), the ObjectId field will reference another table, AllObjects: (ObjectId, DeviceId, DeviceObjectId). Then, when the device pushes up a change set, it will pass along the device Id and the objectId from the core data object in the local data store. Then my cloud server will check against the objectId and device Id in the AllObjects table, and find the record to change in the initial table.
All changes should be timestamped, so that they can be merged.
The device will have to poll the server, without using up too much battery.
The local devices will also need to update anything held in memory if/when changes are received from the server.
Is there anything else I am missing here? What kinds of frameworks should I look at to make this possible?
I've done something similar to what you're trying to do. Let me tell you what I've learned and how I did it.
I assume you have a one-to-one relationship between your Core Data object and the model (or db schema) on the server. You simply want to keep the server contents in sync with the clients, but clients can also modify and add data. If I got that right, then keep reading.
I added four fields to assist with synchronization:
sync_status - Add this field to your core data model only. It's used by the app to determine if you have a pending change on the item. I use the following codes: 0 means no changes, 1 means it's queued to be synchronized to the server, and 2 means it's a temporary object and can be purged.
is_deleted - Add this to the server and core data model. Delete event shouldn't actually delete a row from the database or from your client model because it leaves you with nothing to synchronize back. By having this simple boolean flag, you can set is_deleted to 1, synchronize it, and everyone will be happy. You must also modify the code on the server and client to query non deleted items with "is_deleted=0".
last_modified - Add this to the server and core data model. This field should automatically be updated with the current date and time by the server whenever anything changes on that record. It should never be modified by the client.
guid - Add a globally unique id (see http://en.wikipedia.org/wiki/Globally_unique_identifier) field to the server and core data model. This field becomes the primary key and becomes important when creating new records on the client. Normally your primary key is an incrementing integer on the server, but we have to keep in mind that content could be created offline and synchronized later. The GUID allows us to create a key while being offline.
On the client, add code to set sync_status to 1 on your model object whenever something changes and needs to be synchronized to the server. New model objects must generate a GUID.
Synchronization is a single request. The request contains:
The MAX last_modified time stamp of your model objects. This tells the server you only want changes after this time stamp.
A JSON array containing all items with sync_status=1.
The server gets the request and does this:
It takes the contents from the JSON array and modifies or adds the records it contains. The last_modified field is automatically updated.
The server returns a JSON array containing all objects with a last_modified time stamp greater than the time stamp sent in the request. This will include the objects it just received, which serves as an acknowledgment that the record was successfully synchronized to the server.
The app receives the response and does this:
It takes the contents from the JSON array and modifies or adds the records it contains. Each record get set a sync_status of 0.
I used the word record and model interchangeably, but I think you get the idea.
I suggest carefully reading and implementing the sync strategy discussed by Dan Grover at iPhone 2009 conference, available here as a pdf document.
This is a viable solution and is not that difficult to implement (Dan implemented this in several of its applications), overlapping the solution described by Chris. For an in-depth, theoretical discussion of syncing, see the paper from Russ Cox (MIT) and William Josephson (Princeton):
File Synchronization with Vector Time Pairs
which applies equally well to core data with some obvious modifications. This provides an overall much more robust and reliable sync strategy, but requires more effort to be implemented correctly.
EDIT:
It seems that the Grover's pdf file is no longer available (broken link, March 2015). UPDATE: the link is available through the Way Back Machine here
The Objective-C framework called ZSync and developed by Marcus Zarra has been deprecated, given that iCloud finally seems to support correct core data synchronization.
If you are still looking for a way to go, look into the Couchbase mobile. This basically does all you want. (http://www.couchbase.com/nosql-databases/couchbase-mobile)
Similar like #Cris I've implemented class for synchronization between client and server and solved all known problems so far (send/receive data to/from server, merge conflicts based on timestamps, removed duplicate entries in unreliable network conditions, synchronize nested data and files etc .. )
You just tell the class which entity and which columns should it sync and where is your server.
M3Synchronization * syncEntity = [[M3Synchronization alloc] initForClass: #"Car"
andContext: context
andServerUrl: kWebsiteUrl
andServerReceiverScriptName: kServerReceiverScript
andServerFetcherScriptName: kServerFetcherScript
ansSyncedTableFields:#[#"licenceNumber", #"manufacturer", #"model"]
andUniqueTableFields:#[#"licenceNumber"]];
syncEntity.delegate = self; // delegate should implement onComplete and onError methods
syncEntity.additionalPostParamsDictionary = ... // add some POST params to authenticate current user
[syncEntity sync];
You can find source, working example and more instructions here: github.com/knagode/M3Synchronization.
Notice user to update data via push notification.
Use a background thread in the app to check the local data and the data on the cloud server,while change happens on server,change the local data,vice versa.
So I think the most difficult part is to estimate data in which side is invalidate.
Hope this can help u
I have just posted the first version of my new Core Data Cloud Syncing API, known as SynCloud.
SynCloud has a lot of differences with iCloud because it allows for Multi-user sync interface. It is also different from other syncing api's because it allows for multi-table, relational data.
Please find out more at http://www.syncloudapi.com
Build with iOS 6 SDK, it is very up to date as of 9/27/2012.
I think a good solution to the GUID issue is "distributed ID system". I'm not sure what the correct term is, but I think that's what MS SQL server docs used to call it (SQL uses/used this method for distributed/sync'ed databases). It's pretty simple:
The server assigns all IDs. Each time a sync is done, the first thing that is checked are "How many IDs do I have left on this client?" If the client is running low, it asks the server for a new block of IDs. The client then uses IDs in that range for new records. This works great for most needs, if you can assign a block large enough that it should "never" run out before the next sync, but not so large that the server runs out over time. If the client ever does run out, the handling can be pretty simple, just tell the user "sorry you cannot add more items until you sync"... if they are adding that many items, shouldn't they sync to avoid stale data issues anyway?
I think this is superior to using random GUIDs because random GUIDs are not 100% safe, and usually need to be much longer than a standard ID (128-bits vs 32-bits). You usually have indexes by ID and often keep ID numbers in memory, so it is important to keep them small.
Didn't really want to post as answer, but I don't know that anyone would see as a comment, and I think it's important to this topic and not included in other answers.
First you should rethink how many data, tables and relations you will have. In my solution I’ve implemented syncing through Dropbox files. I observe changes in main MOC and save these data to files (each row is saved as gzipped json). If there is an internet connection working, I check if there are any changes on Dropbox (Dropbox gives me delta changes), download them and merge (latest wins), and finally put changed files. Before sync I put lock file on Dropbox to prevent other clients syncing incomplete data. When downloading changes it’s safe that only partial data is downloaded (eg lost internet connection). When downloading is finished (fully or partial) it starts to load files into Core Data. When there are unresolved relations (not all files are downloaded) it stops loading files and tries to finish downloading later. Relations are stored only as GUID, so I can easly check which files to load to have full data integrity.
Syncing is starting after changes to core data are made. If there are no changes, than it checks for changes on Dropbox every few minutes and on app startup. Additionaly when changes are sent to server I send a broadcast to other devices to inform them about changes, so they can sync faster.
Each synced entity has GUID property (guid is used also as a filename for exchange files). I have also Sync database where I store Dropbox revision of each file (I can compare it when Dropbox delta resets it’s state). Files also contain entity name, state (deleted/not deleted), guid (same as filename), database revision (to detect data migrations or to avoid syncing with never app versions) and of course the data (if row is not deleted).
This solution is working for thousands of files and about 30 entities. Instead of Dropbox I could use key/value store as REST web service which I want to do later, but have no time for this :) For now, in my opinion, my solution is more reliable than iCloud and, which is very important, I have full control on how it’s working (mainly because it’s my own code).
Another solution is to save MOC changes as transactions - there will be much less files exchanged with server, but it’s harder to do initial load in proper order into empty core data. iCloud is working this way, and also other syncing solutions have similar approach, eg TICoreDataSync.
--
UPDATE
After a while, I migrated to Ensembles - I recommend this solution over reinventing the wheel.
I am writting a django app and Iphone app, I need to keep them in sync.
Users can delete, update and create new objects in the web app, and in the iphone app.
When they get online with the iphone both app must be in sync.
Is there simple way to do this?
Thanks,
Joaquin
In general: There's no simple way. But I'll outline an approach.
If you don't care about changes being overwritten: Keep a timestamp of the most recent change to each record, and a timestamp of each sync. When syncing, you get a list of all updates on the iPhone since the last sync, and all updates on the server. You write from the iPhone to the server if the iPhone timestamp for that record is newer than the server one, and vice versa.
But you probably care. Say you've edited a note called "Where to meet up on Friday." It started out empty. Now, on the phone, you've written, "My house." Ten minutes later, your friend edits the same note on the server and writes, "The diner." Who wins out? Stack Overflow can't answer that for you; it's application-specific.
OK, so modify the approach above: if both the server version of a record and the local version have been edited since the last sync, then you have to ask the user what to do. That's the basic algorithm.
If you care a lot about changes not being overwritten, to the point that you want to merge changes to different places in the same documents, then your system will begin to approach the complexity of version control systems like Subversion or Git. Not at all simple.
There's no built in way to do this. You need to keep a server data store, and a local data store on the iPhone, and when online, check the differences manually, and see what action you should take on the server and the iPhone side (delete, update, etc.).
Sync is usually hard. I suggest you start laying out the server and iPhone data stores, and think how they relate, and how can the server or the iPhone know the status of their counterpart record, so to keep them in sync.