I am writting a django app and Iphone app, I need to keep them in sync.
Users can delete, update and create new objects in the web app, and in the iphone app.
When they get online with the iphone both app must be in sync.
Is there simple way to do this?
Thanks,
Joaquin
In general: There's no simple way. But I'll outline an approach.
If you don't care about changes being overwritten: Keep a timestamp of the most recent change to each record, and a timestamp of each sync. When syncing, you get a list of all updates on the iPhone since the last sync, and all updates on the server. You write from the iPhone to the server if the iPhone timestamp for that record is newer than the server one, and vice versa.
But you probably care. Say you've edited a note called "Where to meet up on Friday." It started out empty. Now, on the phone, you've written, "My house." Ten minutes later, your friend edits the same note on the server and writes, "The diner." Who wins out? Stack Overflow can't answer that for you; it's application-specific.
OK, so modify the approach above: if both the server version of a record and the local version have been edited since the last sync, then you have to ask the user what to do. That's the basic algorithm.
If you care a lot about changes not being overwritten, to the point that you want to merge changes to different places in the same documents, then your system will begin to approach the complexity of version control systems like Subversion or Git. Not at all simple.
There's no built in way to do this. You need to keep a server data store, and a local data store on the iPhone, and when online, check the differences manually, and see what action you should take on the server and the iPhone side (delete, update, etc.).
Sync is usually hard. I suggest you start laying out the server and iPhone data stores, and think how they relate, and how can the server or the iPhone know the status of their counterpart record, so to keep them in sync.
Related
I am building an internal iOS application (so - it won't ever be in the app store), and I need to keep a directory of content synchronized between a server and each of the instances of the iOS application. This would be easy enough if I just wanted to delete and re-download this content each time, but I would rather use something similar to rsync to only download the elements that have changed.
I haven't found any good way to utilize rsync. I considered looking at Objective-Git as a possibility here, but at a quick glance it looked like there is still a lot of the support for remote repositories that isn't supported yet.
As a final note, while this won't be in the app store, I will not be jailbreaking these devices and I would prefer to not rely on any private API's (although if there was an elegant solution that utilized private API's I might consider it).
Thoughts?
ADDITIONAL NOTE: This needs to be an isolated solution. I won't be relying on outside services (like Dropbox, Box.net, etc...). This needs to work solely between the device and the server (which is on a local network with the device).
Use HTTP to list the contents of each folder on the server.
Compare last modification time of each file with those on the device, and identify added/removed files.
Get added and modified files, remove deleted files.
It sounds like you're maybe asking for a library that already does this, but if you don't find one it's obviously moderately easy to write this from the ground up using stat(2) on the server and the same or a higher-level equivalent on the iOS devices. Have the iPhone send a tree of files with their modification date to the server and get back a list of insert/delete/update operations to do with the url (or whatever) for each one so you can do them incrementally on a background thread. Have the information from the server for new/updated files include the mod date that the server has so you can set it to be the same on the iOS device and send that when asking the server for the status of each file (kind of hack using the file system to store that, but it works).
Why not just set up a RESTful interface and do it across HTTP; that way you could query the modification times easily enough to determine whether client or server files need to be updated. You might also want to keep track of what files on the client have been synced, so you can easily know which files to add or delete. This can be done with a simple .sync file or using a plist / sqlite / etc.
If you'll consider FTP, there are some pretty advanced client libraries available.
For example, the iOS Chilkat bundle includes an FTP client library that supports synchronization in both directions. It's not free, but it's pretty cheap -- and you get a ton of other stuff that will likely prove useful someday. Here's an example of iOS pulling down all additions and changes (mode 2):
http://www.example-code.com/ios/ftp_syncLocalTree.asp
One caveat -- judging solely from the example, it doesn't appear to synchronize deletions. If this is a requirement, you could do it yourself without too much effort immediately following a sync.
acrosync (see https://acrosync.com/library.html) seems like a good fit given the initial question, however I haven't used it myself yet.
Say I have a TODO list iphone app, that can be edited/viewed from both a web application and the iphone application.
When on the iphone, when a user views all his todo lists, or sub-items, I would think that each time the user views a particular list it shouldn't be hitting the web applications API every-time, but rather cache locally the values and only hit the web when things change.
What strategies are there for this type of scenerio?
I agree with you in your dirty-otherwise-do-not-contact-the-server point. And I think this point is pretty straightforward and easy to implement.
However, be careful in this scenario: it gets dirty but at the same time, the device cannot reach the internet. In this scenario, I suggest you check the internet accessibility on a frequent basis (even when your app is in the background), and try to reach your server and update whenever possible.
This is a tricky problem. I'm currently working on an app that needs to perform a similar synchronization, and I haven't decided how I want to handle it yet.
You're right in that you don't want to be hitting the web repeatedly. It would slow the app down considerably. Keeping a local cache is the way to go.
One drawback is that the user could change/add an item on the web and you wouldn't see it on the phone. You'd need to have a refresh button (like in the Mail application, for example) to allow the user to get the changes.
Then you have an issue with conflict resolution. Say the same item is edited on both the phone and on the web. How does the user pick which one to keep, or do they get duplicated?
I think the best way to do this is to replicated your server's schema in CoreData. Then load a given element from the local DB, and in the background go out and check that element for updates if the device has an internet connection. You're hitting the db each time, but the user is not slowed down by the process.
You should not query the internet everytime you view the list.
But when you make updates to it, or edit it, you should update the server as well. That will make your life a whole lot simpler. That way when the user updates an item that he deleted in the web server, the server will just throw that request out...
I'm using MonoTouch and also System.Data to create a DataSet (just xml to those not familiar) for simple data binding. Data on my app is minimal so no need to go all out with SQLLite. The dataset use makes it easy to pass via web services for cloud sync.
I serialize the DataSet to the personal folder on save and of course read this file when the app starts up to load up the user's data. I've had issues where this file is becoming corrupt and I'm not sure why. I assume file I/O may be slow on these devices and that could be the cause, I'm not sure, but it is happening.
I'm also concerned that maybe iTunes is passing this file back and forth between the PC/MAC when the user syncs their devices with iTunes, which may be the cause of the corruption?
I want to prevent this device file from syncing with iTunes and also reliably persist it. I'm using the NSFile.Save option to save it to the device. I'm thinking since it's a text file maybe I could more safely store it in the standard user settings area instead? This would prevent it from being synced by itunes, I presume?
What is the most reliable and safe way to handle this file i/o for the xml dataset storage?
Thank you.
You're using MonoTouch. Isn't it simply a matter of calling DataSet.WriteXml() with a FileStream object ready to write to a document in your Documents folder?
That Documents folder is backed up to iTunes. It's not synced, but it helps if your user is restoring their phone (because they bricked it, lost it, whatever). It doesn't explain why it's corrupt.
The only thing that I can think of why it's corrupt is because it took too long for your app to write it. There's a limited time from the point where the user exits the app until it's closed down, to prevent apps from keeping the system hostage and deteriorate user experience.
If writing the whole dataset takes too long, you want to think about minimizing that. Perhaps you can just store the data, and not the schema. Or you can devise a way to store only the deltas on exit and reconcile when the user has loaded your app again.
You can also prevent complete loss of data by writing to a second file, and when that operation completes delete the old file and rename. That way, the next time you start up if the write operation didn't complete, the old file would still be there and the user would have only lost their more recent changes.
In any case, if your data gets too big for a simple write operation to complete, you should look at different options such as sqlite.
Your best bet is probably to just save the XML as text. It's as simple as File.WriteAllText(...) - there's no reason to go to NSFile for this. That's part of the advantage of MonoTouch :)
Regarding syncing, here's the rule:
If you keep the file in the user's documents folder (Environment.SpecialFolder.MyDocuments and Environment.SpecialFolder.Personal BOTH point to the user's doc folder), then it's going to get backed up whenever the user syncs with iTunes.
There's nothing wrong with this. It persists the data between sessions and makes it recoverable if something goes wrong with the user's phone and they need to restore from a backup. Since your question is about persisting an XML file on the phone, this is what you want.
As for the iTunes question, there's no problem with speed and syncing because your app isn't going to be running while the phone is syncing. The file will either have been saved or it won't. Any corruption that takes place is happening while your app is running.
Reasons for files getting corrupted can include:
Not saving before the user quits. You get a chance to do this.
Not gracefully handling an incoming phone call. The system warns you about this as well.
iTunes definitely isn't corrupting your file. If that were the case, iOS apps would all be broken. It could be happening on your dev machine for whatever reason, but I've never seen this happen elsewhere, and I've done quite a bit of iOS development.
If you'd like a tutorial on reading and writing files, I posted an answer in another question.
It's lengthy, but the point was to answer as many questions as I could so nobody would be left hanging or confused.
A nice thing about iOS devices is that you're back (for most apps) in the one-person-at-a-time world. You're writing apps where you don't have to worry about 5,000 people trying to use your web-based app at the same time (that's not always true, but... you get the point). As a result, you can do things that you might normally consider bad for performance, but you're unlikely to see any performance problems (as long as the file you're saving is either small enough to be saved quickly or you're saving in the background on another thread - you never want to block the main (UI) thread with a heavy IO operation).
If you have any more questions, feel free to ask.
I hope this helps :)
Lots of frameworks are read-only, but I've found that GDataXMLNode from http://code.google.com/p/gdata-objectivec-client/ works very well read/write.
Having said that, on the iPhone you'd do yourself a big favour using Core Data with a SQLLite backend. :-) Apple has done all this for you and optimized it more than any of us every will
Cheers
Nik
Consider using SQLite, I'd go for something like
http://www.taimila.com/entify/ (not tried yet)
catnap orm
or Faks sqlite-net on google code (using this in a few apps)
entify - if it does what it says it can do - looks really good.
persisting XML on the iPhone as a means to store and access data is a nause you dont want to get into. I wrote about it here http://iwayneo.blogspot.com/2010/08/festival-star-history-serialization-as.html
I'm working on an iPhone application that should work in offline and online modes.
In it's online mode it's supposed to feed all the information the user enters to a webservice backed by GWT/GAE.
In it's offline mode it's supposed to store the information locally, and when connection is available sync it up to the web service.
Currently my plan is as follows:
Provide a connection between an app and a webservice using Protobuffers for efficient over-the-wire communication
Work with local DB using Core Data
Poll the network status, and when available sync the database and keep some sort of local-db-to-remote-db key synchronization.
The question is - am I in the right direction? Are the standard patterns for implementing this? Maybe someone can point me to an open-source application that works in a similar fashion?
I am really new to iPhone coding, and would be very glad to hear any suggestions.
Thanks
I think you've blurring the questions together.
If you've got a question about making a GWT web interface, that's one question.
Questions about how to sync an iPhone to a web service are a different question. For that, you don't want to use GWT's RPCs for syncing, as you'd have to fake out the 'browser-side' of the serialization system in your iPhone code, which GWT normally provides for you.
about system design direction:
First if there is no REAL need do not create 2 different apps one GWT and other iPhone
create one but well written GWT app. It will work off line no problem and will manage your data using HTML feature -- offline application cache
If it a must to create 2 separate apps
than at least save yourself effort and do not write server twice as if you go with standard GWT aproach you will almost sertanly fail to talk to server from stand alone app (it is zipped JSON over HTTP with some tricky headers...) or will write things twise so look in to the RestLet library it well supported by the GAE.
About the way to keep sync with offline / online switching:
There are several aproaches to consider and all of them are not perfect. So when you conseder yours think of what youser expects... Do not be Microsoft Word do not try to outsmart the user.
If there at least one scenario in the use cases that demand user intervention to merge changes (And there will be - take it to the bank) - than you will have implement UI for this - than there is a good reason to use it often - user will get used to it. it better than it will see it in a while since he started to use the app because a need fro it is rare because you implemented a super duper merging logic that asks user only in very special cases... Don't do it.
balance the effort. Because the mess that a bug in such code will introduce to user is much more painful than the benefit all together.
so the HOW:
The one way is the Do-UnDo way.
While off line - keep the log of actions user did on data in timed order user did them
as soon as you connected - send to server and execute them. Same from server to client.
Will work fine in most cases as long as you are not writing a Photoshop kind of software with huge amounts of data per operation. Also referred as Action Pattern by the GangOfFour.
Another way is a source control way. - Versions and may be even locks. very application dependent. DBMS internally some times use it for transactions implementations.
And there is always an option to be Read Only when Ofline :-)
Wonder if you have considered using a Sync Framework to manage the synchronization. If that interests you can take a look at the open source project, OpenMobster's Sync service. You can do the following sync operations
two-way
one-way client
one-way device
bootup
Besides that, all modifications are automatically tracked and synced with the Cloud. You can have your app offline when network connection is down. It will track any changes and automatically in the background synchronize it with the cloud when the connection returns. It also provides synchronization like iCloud across multiple devices
Also, modifications in the Cloud are synched using Push notifications, so the data is always current even if it is stored locally.
Here is a link to the open source project: http://openmobster.googlecode.com
Here is a link to iPhone App Sync: http://code.google.com/p/openmobster/wiki/iPhoneSyncApp
I'm looking into building an application which works just as well offline as it does online. Since the application cannot communicate with the server while in offline, there is some level of synchronization which needs to take place.
What are some good tools to read about and start thinking about when planning offline operations with synchronization for your iPhone?
What tools would I have to create on my own, versus tools that apple already provides to help with this particular issue?
I've been working on an app that handles this exact behavior for the last 2 months or so. It has a small subset of functions which are online only and a large set of functionality that is offline/online.
I'm using sqlite for local storage as suggested here with a modified version of the sqlitepersistentobjects library. The base version of sqlitepersistentobjects is not thread safe so watch out if you are using it. (check out objectiverecord in: objectivesync for a thread safe alternative but be prepared to dig into the code). If you are willing to develop for the 3.0 sdk then core data is another possibility for a sqlite library.
The overall architecture is simple enough I have modeled local storage using sqlite and remote interaction using objective resource against a rails app and REST api. It can use either xml or json for data serialization.
When an object is modified locally the change is first saved to the sqlite database record for that object and then added to a queue which is serialized and stored in the local sqlite db as well. (The queue can then be processed at any time)
If there is a connection available any queued local changes are deserialized and added to an NSOperationQueue which then processes them in the background.
In order to make this all work I've subclassed NSOperation so that it can support several types of remote queue operations - create, update, delete essentially using objective resource to make the remote requests.
The nice thing about using NSOperationQueue and NSOperation is that they handle the background threading for you so I'd highly recommend having a look at the apple docs for those classes and also at the apple threading guide.
When the application loads there is a bit of remote checking done and processed in the background to pull down the latest data - although to be honest I am still changing the way this behaves a bit.
That's a quick overview of what I've had to deal with so far...hope it helps a little.
there are plenty of application on the app store which rely on both online as well as offline data
what you should really be doing is on start of your app, run a background thread (which runs silently so your user never sees any delay). this thread downloads the latest data from your server and pushes it into your local database (sqlite is the best choice)
make sure you implement some kind of data versioning so that your app only downloads data which is actually changed since last download - else you would unnecessarily be downloading the entire dataset which can be quite huge (depending upon your app requirements)
also make sure to test for internet connectivity when doing this. if no internet is available, alert the user for sure
this way you get the best of both worlds. users when away from internet can still use your app with their local sqlite data
in iphone os 3.0 apple has introduced push services - where you can simply "PUSH" your data instead of doing a "PULL" however this is not available in the current iPhone OS (2.x.x)
Push is probably not a viable option here, since the amount of data you can push is miniscule, and basically comes back to "tell my app to make a server call". We use an online/offline model in Satchel. Whenever we have to communicate with the server, we bundle that communication (a URL and possibly some POST data) and store it to a database. If we're online, we pull it right back out, send it, and when we get a valid response back, we remove the record from the database. If we're offline, those rows build up, and the next time we ARE online, they get sent out. This is not a workable model in all situations, but can be adapted to most.
In 3.0, you've got access to CoreData, which is a great data management tool. Other than that, the NSURLXXX family is your friend.
I would store all the information I gather while offline in a SQLite database. Then, on user 's request, you can SYNC all the stored information with a server using HTTP or a custom TCP/IP protocol you can come up with.
I have been using this approach on Palm OS applications for almost 10 years now, and they do work very effectively.
As far as I know, the only "tool" you will have to accomplish this is plain old OBJECTIVE-C with Cocoa Touch. Although you could use some TCP/IP C++ libraries that will make your life easier if you decide to implement your own protocol.
Wonder if you have considered using a Sync Framework to manage the synchronization. If that interests you can take a look at the open source project, OpenMobster's Sync service. You can do the following sync operations
two-way
one-way client
one-way device
bootup
Besides that, all modifications are automatically tracked and synced with the Cloud. You can have your app offline when network connection is down. It will track any changes and automatically in the background synchronize it with the cloud when the connection returns. It also provides synchronization like iCloud across multiple devices
Also, modifications in the Cloud are synched using Push notifications, so the data is always current even if it is stored locally.
Here is a link to the open source project: http://openmobster.googlecode.com
Here is a link to iPhone App Sync: http://code.google.com/p/openmobster/wiki/iPhoneSyncApp