Can anyone suggest the best framework, method, library I could use in iPhone (use in development, not a ready application) in order to achieve syncronization between a fileserver and a local storadge on iPhone ?
I am doing this for one of my apps and the way I've implemented it is using a Ruby & Sinatra webserver, talking to a MongoDB database. You could use any other database and webserver technology.
The basic concept is this:
Every time an object in the database is updated, the timestamp is recorded for that object.
A global, last updated timestamp is also updated.
The app contacts the webserver and asks for updates, passing along a locally stored "last updated" timestamp.
The webserver processes the request by first checking the global timestamp and making sure it is older than the app's timestamp. (This is to save having to scour the database if no changes were made to it. My model is: Big data that is not changed frequently. If you have data that changes frequently, then there is probably no benefit to this global timestamp.)
The webserver then finds every object in the database whose timestamp is newer than the app's timestamp.
The webserver packages this up inside a JSON object, and returns it to the app.
This is all RESTful in the sense that it is a stateless transaction, so the app's implementation is very simple (a simple NSURLRequest, followed by JSON decoding, followed by error handling). Now you have an array of updated objects and you can merge these with your local storage in the app.
Another nice point about this (stateless) approach is that you can run it on Heroku (for free).
From what I can tell, there is no easy way. I was looking for an rsync equivalent, but I haven't found one.
In my case, I'm manually walking the tree asking the server for differences after a certain date and I remember the last successful sync date.
Not pretty. Could spend lots of time coming up with something sophisticated.
Related
I have a Web Service and sqlite database. In this, web service will be used to store data inside database. Now I want to include sync functionality as - Whenever application starts at that time the database will start to load its table's data through web service.
Now after some time when I update my my web service the database will be updated accordingly. My question is that what are the best practices that I must follow for this update. Should I clear whole DB and start adding all rows again(I know this will take a lot time) but If not this then how do my database will add only particular data from the web service?
Thank you.
What I suggest you is:
store all your webservice content into db first when the app starts.
display your content on the screen from db only.
again when you need to refresh or recall your data just update the database.
Thus, you will always find all your fresh data into database.
Downloading and updating the entire server data will prove expensive. It will use more bandwidth and prove costly to your customer. Rather than pushing the entire load (even for minor update), send a delta. I will suggest you to maintain version information.
When application downloads the data from web service for a said version and store it successfully in the database, set the current updated version as well in the DB.
When app starts the next time, make a light weight header request to get just the version info from the server. The server should respond to this header request with the latest data version number.
Check the version from WS with the current application data version stored in the DB. If the server has an updated version, start the sync.
The version change information should be delta i.e.
For new version, server should send only the information that is changed since the version available with the device.
You server should have capability to calculate the delta between two versions.
Delta information will typically have sections like, new data, updated data, deleted data etc.
Based on this, the iOS app will make the necessary CRUD(Create, Read, Update and Delete) operations on the DB data.
Once the iOS app updates itself, then you can update the DB version to the latest received version from server. Until then let it remain dirty for proper error handling.
Hope that helps.
I would recommend you use RestKit's superb Core Data support.
By using RKEntityMapping you can map your remote objects from JSON or XML directly to Core Data entities in your database.
RestKit will automatically maintain the database for you, inserting and updating entries as appropriate from your web service. (In my experience, I've found deleting objects requires a tiny bit of extra work depending on how RESTful your web service is).
RestKit definitely does have a learning curve attached, but it's well worth it: having deployed it a couple of times now, is definitely a much better solution than manually writing your own SQLite/Web Service syncing code.
First you need to set all webservice content into your SQLITE.and what you want to display get that data from SQLITE.and perform opertaion into that sqlite table and when once all this done you need to changes made are saved it into webservice.
Follow this way.
Can people give me examples of why they would use coreData in an application?
I ask this because most apps are just clients to a central server where an API of some sort gives you the information you need.
In my case I'm writing a timesheet application for a web app which has an API and I'm debating if there is any value in replicating the data structure on my server in core data(Sqlite)
e.g
Project has many timesheets
employee has many timesheets
It seems to me that I can just connect to the API on every call for lists of projects or existing timesheets for example.
I realize for some kind of offline mode you could store locally in core data but this creates way more problems because you now have a big problem with syncing that data back to the web server when you get connection again.. e.g. the project selected for a timesheet no longer exists.
Can any experienced developer shed some light on there experiences on when core data is best practice approach?
EDIT
I realise of course there is value in storing local persistance but the key value of user defaults seems to cover most applications I can think of.
You shouldn't think of CoreData simply as an SQLite database. It's not JUST an SQLite database. Sure, SQLite is an option, but there are other options as well, such as in-memory and, as of iOS5, a whole slew of custom data stores. The biggest benefit with CoreData is persistence, obviously. But even if you are using an in-memory data store, you get the benefits of a very well structured object graph, and all of the heavy lifting with regards to pulling information out of or putting information into the data store is handled by CoreData for you, without you necessarily needing to concern yourself with what is backing that data store. Sure, today you don't care too much about persistence, so you could use an in-memory data store. What happens if tomorrow, or in a month, or a year, you decide to add a feature that would really benefit from persistence? With CoreData, you simply change or add a persistent data store, and all of your methods to get information out or in remain unchanged. The overhead for that sort of addition is minimal in comparison to if you were trying to access SQLite or some other data store directly. IMHO, that's the biggest benefit: abstraction. And, in essence, abstraction is one of the most powerful things behind OOP. Granted, building the Data Model just for in-memory storage could be overkill for your app, depending on how involved the app is. But, just as a side note, you may want to consider what is faster: Requesting information from your web service every time you want to perform some action, or requesting the information once, storing it in memory, and acting on that stored value for the remainder of the session. An in-memory data store wouldn't persistent beyond that particular session.
Additionally, with CoreData you get a lot of other great features like saving, fetching, and undo-redo.
There are basically two kinds of apps. Those that provide you with local functionality (games, professional applications, navigation systems...) and those that grant access to a remote service.
Your app seems to be in the second category. If you access remote services, your users will want to access new or real-time data (you don't want to read 2 week old Facebook posts) but in some cases, local caching makes sense (e.g. reading your mails when you're on the train with unstable network).
I assume that the value of accessing cached entries when not connected to a network is pretty low for your customers (internal or external) compared to the importance of accessing real-time-data. So local storage might be not necessary at all.
If you don't have hundreds of entries in your timetable, "normal" serialization (NSCoding-protocol) might be enough. If you only access some "dashboard-data", you will be able to get along with simple request/response-caching (NSURLCache can do a lot of things...).
Core Data does make more sense if you have complex data structures which should be synchronized with a server. This adds a lot of synchronization logic to your project as well as complexity from Core Data integration (concurrency, thread-safety, in-app-conflicts...).
If you want to create a "client"-app with a server driven user experience, local storage is not necessary at all so my suggestion is: Keep it as simple as possible unless there is a real need for offline storage.
It's ideal for if you want to store data locally on the phone.
Seriously though, if you can't see a need for it for your timesheet app, then don't worry about it and don't use it.
Solving the sync problems that you would have with an "offline" mode would be detailed in your design of your app. For example - don't allow projects to be deleted. Why would you? Wouldn't you want to go back in time and look at previous data for particular projects? Instead just have a marker on the project to show it as inactive and a date/time that it was made inactive. If the data that is being synced from the device is for that project and is before the date/time that it was marked as inactive, then it's fine to sync. Otherwise display a message and the user will have to sort it.
It depends purely on your application's design whether you need to store some data locally or not, if it is a real problem or a thin GUI client around your web service. Apart from "offline" mode the other reason to cache server data on client side might be to take traffic load from your server. Just think what does it mean for your server to send every time the whole timesheet data to the client, or just the changes. Yes, it means more implementation on both side, but in some cases it has serious advantages.
EDIT: example added
You have 1000 records per user in your timesheet application and one record is cca 1 kbyte. In this case every time a user starts your application, it has to fetch ~1Mbyte data from your server. If you cache the data locally, the server can tell you that let's say two records were updated since your last update, so you'll have to download only 2 kbyte. Now you should scale up this for several tens of thousands of user and you will immediately notice the difference of the server bandwidth and CPU usage.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I have been working on a method to sync core data stored in an iPhone application between multiple devices, such as an iPad or a Mac. There are not many (if any at all) sync frameworks for use with Core Data on iOS. However, I have been thinking about the following concept:
A change is made to the local core data store, and the change is saved. (a) If the device is online, it tries to send the changeset to the server, including the device ID of the device which sent the changeset. (b) If the changeset does not reach the server, or if the device is not online, the app will add the change set to a queue to send when it does come online.
The server, sitting in the cloud, merges the specific change sets it receives with its master database.
After a change set (or a queue of change sets) is merged on the cloud server, the server pushes all of those change sets to the other devices registered with the server using some sort of polling system. (I thought to use Apple's Push services, but apparently according to the comments this is not a workable system.)
Is there anything fancy that I need to be thinking about? I have looked at REST frameworks such as ObjectiveResource, Core Resource, and RestfulCoreData. Of course, these are all working with Ruby on Rails, which I am not tied to, but it's a place to start. The main requirements I have for my solution are:
Any changes should be sent in the background without pausing the main thread.
It should use as little bandwidth as possible.
I have thought about a number of the challenges:
Making sure that the object IDs for the different data stores on different devices are attached on the server. That is to say, I will have a table of object IDs and device IDs, which are tied via a reference to the object stored in the database. I will have a record (DatabaseId [unique to this table], ObjectId [unique to the item in the whole database], Datafield1, Datafield2), the ObjectId field will reference another table, AllObjects: (ObjectId, DeviceId, DeviceObjectId). Then, when the device pushes up a change set, it will pass along the device Id and the objectId from the core data object in the local data store. Then my cloud server will check against the objectId and device Id in the AllObjects table, and find the record to change in the initial table.
All changes should be timestamped, so that they can be merged.
The device will have to poll the server, without using up too much battery.
The local devices will also need to update anything held in memory if/when changes are received from the server.
Is there anything else I am missing here? What kinds of frameworks should I look at to make this possible?
I've done something similar to what you're trying to do. Let me tell you what I've learned and how I did it.
I assume you have a one-to-one relationship between your Core Data object and the model (or db schema) on the server. You simply want to keep the server contents in sync with the clients, but clients can also modify and add data. If I got that right, then keep reading.
I added four fields to assist with synchronization:
sync_status - Add this field to your core data model only. It's used by the app to determine if you have a pending change on the item. I use the following codes: 0 means no changes, 1 means it's queued to be synchronized to the server, and 2 means it's a temporary object and can be purged.
is_deleted - Add this to the server and core data model. Delete event shouldn't actually delete a row from the database or from your client model because it leaves you with nothing to synchronize back. By having this simple boolean flag, you can set is_deleted to 1, synchronize it, and everyone will be happy. You must also modify the code on the server and client to query non deleted items with "is_deleted=0".
last_modified - Add this to the server and core data model. This field should automatically be updated with the current date and time by the server whenever anything changes on that record. It should never be modified by the client.
guid - Add a globally unique id (see http://en.wikipedia.org/wiki/Globally_unique_identifier) field to the server and core data model. This field becomes the primary key and becomes important when creating new records on the client. Normally your primary key is an incrementing integer on the server, but we have to keep in mind that content could be created offline and synchronized later. The GUID allows us to create a key while being offline.
On the client, add code to set sync_status to 1 on your model object whenever something changes and needs to be synchronized to the server. New model objects must generate a GUID.
Synchronization is a single request. The request contains:
The MAX last_modified time stamp of your model objects. This tells the server you only want changes after this time stamp.
A JSON array containing all items with sync_status=1.
The server gets the request and does this:
It takes the contents from the JSON array and modifies or adds the records it contains. The last_modified field is automatically updated.
The server returns a JSON array containing all objects with a last_modified time stamp greater than the time stamp sent in the request. This will include the objects it just received, which serves as an acknowledgment that the record was successfully synchronized to the server.
The app receives the response and does this:
It takes the contents from the JSON array and modifies or adds the records it contains. Each record get set a sync_status of 0.
I used the word record and model interchangeably, but I think you get the idea.
I suggest carefully reading and implementing the sync strategy discussed by Dan Grover at iPhone 2009 conference, available here as a pdf document.
This is a viable solution and is not that difficult to implement (Dan implemented this in several of its applications), overlapping the solution described by Chris. For an in-depth, theoretical discussion of syncing, see the paper from Russ Cox (MIT) and William Josephson (Princeton):
File Synchronization with Vector Time Pairs
which applies equally well to core data with some obvious modifications. This provides an overall much more robust and reliable sync strategy, but requires more effort to be implemented correctly.
EDIT:
It seems that the Grover's pdf file is no longer available (broken link, March 2015). UPDATE: the link is available through the Way Back Machine here
The Objective-C framework called ZSync and developed by Marcus Zarra has been deprecated, given that iCloud finally seems to support correct core data synchronization.
If you are still looking for a way to go, look into the Couchbase mobile. This basically does all you want. (http://www.couchbase.com/nosql-databases/couchbase-mobile)
Similar like #Cris I've implemented class for synchronization between client and server and solved all known problems so far (send/receive data to/from server, merge conflicts based on timestamps, removed duplicate entries in unreliable network conditions, synchronize nested data and files etc .. )
You just tell the class which entity and which columns should it sync and where is your server.
M3Synchronization * syncEntity = [[M3Synchronization alloc] initForClass: #"Car"
andContext: context
andServerUrl: kWebsiteUrl
andServerReceiverScriptName: kServerReceiverScript
andServerFetcherScriptName: kServerFetcherScript
ansSyncedTableFields:#[#"licenceNumber", #"manufacturer", #"model"]
andUniqueTableFields:#[#"licenceNumber"]];
syncEntity.delegate = self; // delegate should implement onComplete and onError methods
syncEntity.additionalPostParamsDictionary = ... // add some POST params to authenticate current user
[syncEntity sync];
You can find source, working example and more instructions here: github.com/knagode/M3Synchronization.
Notice user to update data via push notification.
Use a background thread in the app to check the local data and the data on the cloud server,while change happens on server,change the local data,vice versa.
So I think the most difficult part is to estimate data in which side is invalidate.
Hope this can help u
I have just posted the first version of my new Core Data Cloud Syncing API, known as SynCloud.
SynCloud has a lot of differences with iCloud because it allows for Multi-user sync interface. It is also different from other syncing api's because it allows for multi-table, relational data.
Please find out more at http://www.syncloudapi.com
Build with iOS 6 SDK, it is very up to date as of 9/27/2012.
I think a good solution to the GUID issue is "distributed ID system". I'm not sure what the correct term is, but I think that's what MS SQL server docs used to call it (SQL uses/used this method for distributed/sync'ed databases). It's pretty simple:
The server assigns all IDs. Each time a sync is done, the first thing that is checked are "How many IDs do I have left on this client?" If the client is running low, it asks the server for a new block of IDs. The client then uses IDs in that range for new records. This works great for most needs, if you can assign a block large enough that it should "never" run out before the next sync, but not so large that the server runs out over time. If the client ever does run out, the handling can be pretty simple, just tell the user "sorry you cannot add more items until you sync"... if they are adding that many items, shouldn't they sync to avoid stale data issues anyway?
I think this is superior to using random GUIDs because random GUIDs are not 100% safe, and usually need to be much longer than a standard ID (128-bits vs 32-bits). You usually have indexes by ID and often keep ID numbers in memory, so it is important to keep them small.
Didn't really want to post as answer, but I don't know that anyone would see as a comment, and I think it's important to this topic and not included in other answers.
First you should rethink how many data, tables and relations you will have. In my solution I’ve implemented syncing through Dropbox files. I observe changes in main MOC and save these data to files (each row is saved as gzipped json). If there is an internet connection working, I check if there are any changes on Dropbox (Dropbox gives me delta changes), download them and merge (latest wins), and finally put changed files. Before sync I put lock file on Dropbox to prevent other clients syncing incomplete data. When downloading changes it’s safe that only partial data is downloaded (eg lost internet connection). When downloading is finished (fully or partial) it starts to load files into Core Data. When there are unresolved relations (not all files are downloaded) it stops loading files and tries to finish downloading later. Relations are stored only as GUID, so I can easly check which files to load to have full data integrity.
Syncing is starting after changes to core data are made. If there are no changes, than it checks for changes on Dropbox every few minutes and on app startup. Additionaly when changes are sent to server I send a broadcast to other devices to inform them about changes, so they can sync faster.
Each synced entity has GUID property (guid is used also as a filename for exchange files). I have also Sync database where I store Dropbox revision of each file (I can compare it when Dropbox delta resets it’s state). Files also contain entity name, state (deleted/not deleted), guid (same as filename), database revision (to detect data migrations or to avoid syncing with never app versions) and of course the data (if row is not deleted).
This solution is working for thousands of files and about 30 entities. Instead of Dropbox I could use key/value store as REST web service which I want to do later, but have no time for this :) For now, in my opinion, my solution is more reliable than iCloud and, which is very important, I have full control on how it’s working (mainly because it’s my own code).
Another solution is to save MOC changes as transactions - there will be much less files exchanged with server, but it’s harder to do initial load in proper order into empty core data. iCloud is working this way, and also other syncing solutions have similar approach, eg TICoreDataSync.
--
UPDATE
After a while, I migrated to Ensembles - I recommend this solution over reinventing the wheel.
I am still learning xcode and objective-c. I use to build app for iphone environment only.
However I am in need of realizing an application with an existing prefilled sql database.
For prefilling the database I wouldn't like to use code in the ditributed app, but I would rather prefer to have a separate app for doing that.
The reason is that, the app could only download the updated database, rather than a whole code update .
So, questions are:
is this a possible scenario
if yes, what kind of application
should I build in xcode for
prefilling database ?
thanks
There's no reason that you can't have one app that both uses the database and downloads updates. Keeping the database updated without downloading the whole thing is pretty simple.
If you record the creation and modification timestamps of rows in the database on the server and keep track of those same modification timestamps on the device, updating the database works like this:
The device determines latest modification timestamp it has for a given table. We'll call it latestTimestamp. It sends the latestTimestamp to the server.
The server compares the latestTimestamp to the creation and modification timestamps in the database. The server sends back data based on the comparison result:
If the modification timestamp is earlier than latestTimestamp it doesn't need to send the record, the device already has it;
If the modification timestamp is later than latestTimestamp and the creation timestamp is earlier than latestTimestamp, it sends the record back noting that it is to be updated in the device database;
If the modification timestamp is later than latestTimestamp and the creation timestamp is later than latestTimestamp, it sends the record back noting that it is to be added in the device database.
Lastly, the server database needs to keep track of deleted records and a deletion timestamp for every record recorded. If latestTimestamp is later than the deletion timestamp, it sends back that the record needs to be deleted.
Obviously it gets a bit more complicated when you have a variety of connected tables, but as long as things are sent back in the correct order, it works great.
Use asynchronous data requests (the ASIHTTPRequest library makes it a breeze) and update the data in the background while the user uses the app. If it's essential that the data be updated prior to any interaction with it you can display an activity indicator and have the user wait.
No need at all for a separate app.
I would discourage you from doing that. No matter it is a pre-filled-database-purpose app, or a normal-purpose app, Apple Review Team would treat them with the same procedure, leaving the developer waiting for weeks before that app is finally available on App Store.
Besides, as far as I know, communication between apps is still strictly limited. If the data you would like to transfer between your main app and your db app is larger than a few lines of, let's say, NSString, it might be technically un-plausible.
I am writting a django app and Iphone app, I need to keep them in sync.
Users can delete, update and create new objects in the web app, and in the iphone app.
When they get online with the iphone both app must be in sync.
Is there simple way to do this?
Thanks,
Joaquin
In general: There's no simple way. But I'll outline an approach.
If you don't care about changes being overwritten: Keep a timestamp of the most recent change to each record, and a timestamp of each sync. When syncing, you get a list of all updates on the iPhone since the last sync, and all updates on the server. You write from the iPhone to the server if the iPhone timestamp for that record is newer than the server one, and vice versa.
But you probably care. Say you've edited a note called "Where to meet up on Friday." It started out empty. Now, on the phone, you've written, "My house." Ten minutes later, your friend edits the same note on the server and writes, "The diner." Who wins out? Stack Overflow can't answer that for you; it's application-specific.
OK, so modify the approach above: if both the server version of a record and the local version have been edited since the last sync, then you have to ask the user what to do. That's the basic algorithm.
If you care a lot about changes not being overwritten, to the point that you want to merge changes to different places in the same documents, then your system will begin to approach the complexity of version control systems like Subversion or Git. Not at all simple.
There's no built in way to do this. You need to keep a server data store, and a local data store on the iPhone, and when online, check the differences manually, and see what action you should take on the server and the iPhone side (delete, update, etc.).
Sync is usually hard. I suggest you start laying out the server and iPhone data stores, and think how they relate, and how can the server or the iPhone know the status of their counterpart record, so to keep them in sync.