I have a mongo database with a collection of objects, lets say orders.
Multiple mobile devices are saving and loading orders to that collection.
When one device saves a new order to the database, all devices should sync and download new data. I want to be able to sync every 3 seconds so everyone can have latest data at every moment.
Now my problem is how to accomplish that efficiently? I do not want to load all the data, and parse it on the client side. I want to download only the missing orders.
What would be the best practice way of achieving this?
One option is to do a differential update.
You'll need to record the time of the last update of each device in the DB and the time of each modification to the DB.
Then you only update with the changes that occurred after the last time the device updated.
Also, one last note, sync'ing every 3 seconds will drain the battery fairly quickly. Perhaps you should consider only updating a device when the device has something to send to the server. ie send a new order, and the server response contains all new orders.
Related
I have a multi-user single-page mobile app developed with Ionic 1, PouchDB and CouchDB. User's management is achieved with SuperLogin.
I would like to add a feature computing a score (something similar to the score in the Waze app) for each user based on his current data, and keeping a track of the former values of that score every past day.
I am wondering about the best way to implement this.
About my app:
it should be able to work offline, and then sync with the server when online (this is why I am using PouchDB and CouchDB, working great so far). So on the server, I have one CouchDB database per user, storing his own data
on the server. The PouchDB database in the app is syncing with the one of the user on the server.
I am considering various options for the score, but none of them really satisfy me, so your advice would be welcome (possibly for yet another option)
Option 1: The score is computed in the app by the Ionic code. The result is stored as a local database object, with a date and a score value. This happens whenever the user changes its data. As the DB is synced with the server, these scores are updated in the server too. However, if some days the user does not use the application, the score won't be computed for these days. More over, if the user runs the app on 2 different devices, and update some data on one of them, this will make the score recomputed locally, then propagated on the server. When the changed data propagate to the server and to the other device, this will trigger a new score computation on this other device, and might lead to conflicts on the score object in the server. Finally, if at some point in time, I want to change the way of computing the score, the value given to each user will depend on whether he has upgraded to the latest app version.
Option 2: have a server-side process that triggers every day, and compute each user's score by connecting to each user's DB on the server, reading its data, computing the corresponding score, and storing it (date+value) back in the server DB. This option looks cleaner to me, but it would require further developments, and an additional process to maintain and keep alive on the server. And if the user inputs data to the application while not connected to the internet, the score will not be updated in the app until he gets connected again (which would cause the server process to recompute the score, and propagate it back to the app through CouchDB sync)
Option 3: have some kind of "stored procedure" in the CouchDB server, triggering every time related data change, in charge of computing the score of each user. But I don't think this is doable with CouchDB.
So how would you do this score computation please??
Many thanks!
I'm developing an application that needs to be syncronized with remote database. The database is connected to the a web-based application that user able to modify some records on the web page.(add/remove/modify) User also able to modify the same records in mobile application. So each side (server - client) must be keep the SAME latest records when an user press the sync button in mobile app. Communication between server and client is provided by Web Serives.(SOAP) and i am not able to change it because of it is strict requirements. (i know this is the worst way that can be used). And another requirement is the clients are not able to delete the server records.
I already be familiar with communicating web service (NSURLConnection), receiving data (NSData) and parsing it. But i could not figure out how the syncronization procedure should be. I have already read this answer which about how i can modify server and client sides with some extra attributes (last_updated_date and is_sync)
Then i could imagine to solve the issue like:
As a first step, client keeps try to modify the server records with sending unsyncronized ones. New recoords are directly added into DB but modified records shoud be compared depending on last_updated_date. At the end of this step, server has the latest data.
But the problem is how can manage to modify the records in mobile app. I thought it in a two way:
is the dummiest way that create a new MOC, download all records into this and change with existing one.
is the getting all modified records which are not in client side, import them into a new MOC and combine these two. But in this point i have some concerns like
There could be two items which are replicated (old version - updated version)
Deleted items could be still located in the main MOCs.
I have to connect multiple relationships among the MOCs. (the new record could have more than 4 relationships with old records)
So i guess you guys can help me to have another ideas which is the best ??
Syncing data is a non-trivial task.
There are several levels of synchronization. Based on your question I am guessing you just need to push changes back to a server. In that case I would suggest catching it during the -save: of the NSManagedObjectContext. Just before the -save: you can query the NSManagedObjectContext and ask it for what objects have been created, updated and deleted. From there you can build a query to post back to your web service.
Dealing with merges, however, is far more complicated and I suggest you deal with them on the server.
As for your relationship question; I suggest you open a second question for that so that there is no confusion.
Update
Once the server has finished the merge it pushes the new "truth" to the client. The client should take these updated records and merge them into its own changes. This merge is fairly simple:
Look for an existing record using a uniqueID.
If the record exists then update it.
If the record does not exist then create it.
Ignoring performance for the moment, this is fairly straight forward:
Set up a loop over the new data coming in.
Set up a NSPredicate to identify the record to be updated/created.
Run your fetch request.
If the record exists update it.
If it doesn't then create it.
Once you get this working with a full round trip then you can start looking at performance, etc. Step one is to get it to work :)
I am still learning xcode and objective-c. I use to build app for iphone environment only.
However I am in need of realizing an application with an existing prefilled sql database.
For prefilling the database I wouldn't like to use code in the ditributed app, but I would rather prefer to have a separate app for doing that.
The reason is that, the app could only download the updated database, rather than a whole code update .
So, questions are:
is this a possible scenario
if yes, what kind of application
should I build in xcode for
prefilling database ?
thanks
There's no reason that you can't have one app that both uses the database and downloads updates. Keeping the database updated without downloading the whole thing is pretty simple.
If you record the creation and modification timestamps of rows in the database on the server and keep track of those same modification timestamps on the device, updating the database works like this:
The device determines latest modification timestamp it has for a given table. We'll call it latestTimestamp. It sends the latestTimestamp to the server.
The server compares the latestTimestamp to the creation and modification timestamps in the database. The server sends back data based on the comparison result:
If the modification timestamp is earlier than latestTimestamp it doesn't need to send the record, the device already has it;
If the modification timestamp is later than latestTimestamp and the creation timestamp is earlier than latestTimestamp, it sends the record back noting that it is to be updated in the device database;
If the modification timestamp is later than latestTimestamp and the creation timestamp is later than latestTimestamp, it sends the record back noting that it is to be added in the device database.
Lastly, the server database needs to keep track of deleted records and a deletion timestamp for every record recorded. If latestTimestamp is later than the deletion timestamp, it sends back that the record needs to be deleted.
Obviously it gets a bit more complicated when you have a variety of connected tables, but as long as things are sent back in the correct order, it works great.
Use asynchronous data requests (the ASIHTTPRequest library makes it a breeze) and update the data in the background while the user uses the app. If it's essential that the data be updated prior to any interaction with it you can display an activity indicator and have the user wait.
No need at all for a separate app.
I would discourage you from doing that. No matter it is a pre-filled-database-purpose app, or a normal-purpose app, Apple Review Team would treat them with the same procedure, leaving the developer waiting for weeks before that app is finally available on App Store.
Besides, as far as I know, communication between apps is still strictly limited. If the data you would like to transfer between your main app and your db app is larger than a few lines of, let's say, NSString, it might be technically un-plausible.
I have an iPhone app syncing with a website using a webservice. The sync procedure I'm using is the following:
Send items that need to sync from iPhone to website
Add items to website MySql database
Send back items that need to sync from website to iPhone
Add items on iPhone Sqlite database
All this is done using one request to the server. Often the user just have a few items (<20) to sync but sometimes a user might have 2000 items to sync. Adding 2000 items to a MySql database can take some time. I'm using transactions and commit on the website when adding the items, something like this:
mysqlTransaction.BeginTransaction();
foreach item sent from iPhone
{
mysql.CommandText = //Sql query
mysql.ExecuteNonQuery();
}
mysqlTransaction.Commit();
Sometimes when a user have a lot of items to add, this can take a veeery long time. My idea is therefore to change the sync procedure a little bit and instead of using one large request I divide the request into chunks, maybe sending and recieiving 50 items at a time. This will cause more but faster requests. What are your ideas on this? How would you solve it? Is it the right thing to use transactions and commit on the server?
I would try to make the update from "iphone to server" and "server to iphone" concurrent and asynchronous if possible. That way the sync will be faster and the iphone app won't block waiting for the whole process to be done.
As well dividing the update into frames if the update is bigger than a given size is a good idea.
I am working on a regular iPhone app which pulls data from a server (XML, JSON, etc...), and I'm wondering what is the best way to implement synching data. Criteria are speed (less network data exchange), robustness (data recovery in case update fails), offline access and flexibility (adaptable when the structure of the database changes slightly, like a new column). I know it varies from app to app, but can you guys share some of your strategy/experience?
For me, I'm thinking of something like this:
1) Store Last Modified Date in iPhone
2) Upon launching, send a message like getNewData.php?lastModifiedDate=...
3) Server will process and send back only modified data from last time.
4) This data is formatted as so:
<+><data id="..."></data></+> // add this to SQLite/CoreData
<-><data id="..."></data></-> // remove this
<%><data id="..."><attribute>newValue</attribute></data></%> // new modified value
I don't want to make <+>, <->, <%>... for each attribute as well, because it would be too complicated, so probably when receive a <%> field, I would just remove the data with the specified id and then add it again (assuming id here is not some automatically auto-incremented field).
5) Once everything is downloaded and updated, I will update the Last Modified Date field.
The main problem with this strategy is: If the network goes down when I am updating something => the Last Modified Date is not yet updated => next time I relaunch the app, I will have to go through the same thing again. Not to mention potential inconsistent data. If I use a temporary table for update and make the whole thing atomic, it would work, but then again, if the update is too long (lots of data change), the user has to wait a long time until new data is available. Should I use Last-Modified-Date for each of the data field and update data gradually?
I would start by making the update routine atomic, since you'll have enough on your hands figuring out how to get the client-server communication working properly.
After that is a good time to consider tweaking it to be incremental, but only after you do some testing to figure out if it's really necessary. If you're tuning your update protocol to be as low bandwidth as possible, you might discover that even a "big" update is downloaded fast enough.
Another way to look at it is to ask yourself, how often is there going to be network trouble when an average user is doing a sync? You probably don't want to tune for unlikely scenarios.
If you are trying to optimize (minimize) the data transfer you may want to consider a different format than XML, since XML is fairly verbose. Or at least you may want to trade in XML readability for space by making each element name and attribute as small as possible, and eliminate all unnecessary whitespace.
Your basic scheme is good. The thing you need to do is to somehow make your updates idempotent so that you can restart a partially-completed transfer without risk. This is a better way to go than to try to implement some sort of true atomic commit (though you could do that too, using, eg, the SQLite database).
In our experience fairly large updates (10s of KB) can be downloaded quite rapidly, if the server is fast enough. No great need to break updates up into tiny bits. But certainly it won't hurt to try to minimize the amount of data transferred by keeping more granular info on "last update".
(And definitely you should use JSON rather than XML as your transmitted data representation.)
Wonder if you have considered using a Sync Framework to manage the synchronization. If that interests you can take a look at the open source project, OpenMobster's Sync service. You can do the following sync operations
two-way
one-way client
one-way device
bootup
Besides that, all modifications are automatically tracked and synced with the Cloud. You can have your app offline when network connection is down. It will track any changes and automatically in the background synchronize it with the cloud when the connection returns. It also provides synchronization like iCloud across multiple devices
Also, modifications in the Cloud are synched using Push notifications, so the data is always current even if it is stored locally.
In your case,
Criteria are speed (less network data exchange), robustness (data recovery in case update fails), offline access
Speed: Only the changes are sent across the network in both directions
Robustness: It stores data in a transactional store like sqlite and any failed updates are communicated in the SyncML payload. Only the successful operations are processed while the failed operations are re-tried during the next sync
Here is a link to the open source project: http://openmobster.googlecode.com
Here is a link to iPhone App Sync: http://code.google.com/p/openmobster/wiki/iPhoneSyncApp