I have a multi-user single-page mobile app developed with Ionic 1, PouchDB and CouchDB. User's management is achieved with SuperLogin.
I would like to add a feature computing a score (something similar to the score in the Waze app) for each user based on his current data, and keeping a track of the former values of that score every past day.
I am wondering about the best way to implement this.
About my app:
it should be able to work offline, and then sync with the server when online (this is why I am using PouchDB and CouchDB, working great so far). So on the server, I have one CouchDB database per user, storing his own data
on the server. The PouchDB database in the app is syncing with the one of the user on the server.
I am considering various options for the score, but none of them really satisfy me, so your advice would be welcome (possibly for yet another option)
Option 1: The score is computed in the app by the Ionic code. The result is stored as a local database object, with a date and a score value. This happens whenever the user changes its data. As the DB is synced with the server, these scores are updated in the server too. However, if some days the user does not use the application, the score won't be computed for these days. More over, if the user runs the app on 2 different devices, and update some data on one of them, this will make the score recomputed locally, then propagated on the server. When the changed data propagate to the server and to the other device, this will trigger a new score computation on this other device, and might lead to conflicts on the score object in the server. Finally, if at some point in time, I want to change the way of computing the score, the value given to each user will depend on whether he has upgraded to the latest app version.
Option 2: have a server-side process that triggers every day, and compute each user's score by connecting to each user's DB on the server, reading its data, computing the corresponding score, and storing it (date+value) back in the server DB. This option looks cleaner to me, but it would require further developments, and an additional process to maintain and keep alive on the server. And if the user inputs data to the application while not connected to the internet, the score will not be updated in the app until he gets connected again (which would cause the server process to recompute the score, and propagate it back to the app through CouchDB sync)
Option 3: have some kind of "stored procedure" in the CouchDB server, triggering every time related data change, in charge of computing the score of each user. But I don't think this is doable with CouchDB.
So how would you do this score computation please??
Many thanks!
Related
Can Firebase get accurate time for offline records which have been stored in device when the device's clock has been changed? The device's clock might be changed when the user rebooted the device or when the user adjusts the date time manually.
My case is quite similar to Get actual DateTime of a device in Offline Mode which I can quote here 1) The application can work in online as well as offline. The app has a feature to create and save the NOTES inside the application. Also, When the app gets internet connection, I need to send the NOTES to backend server. I have a field called 'DATECREATED' in each NOTE (the datetime where the actual NOTE was created) ISSUE: If the User has set the DateTime wrongly in the device, My application sends the incorrect DATETIME to server.
I am going to handle the offline issue on Android and IOS platform, so I came out with another possible solution such as get the GPS time, using some sort of background counter, etc. Based on my experiment, the solution are not reliable enough and some might difficult to do. That's why I decided to get the accurate timestamp using Firebase.
I read a reference https://firebase.google.com/docs/database/flutter/offline-capabilities#clock_skew, but I am not quite sure whether it can help me to get accurate time for offline records.
There is no way you can get a server timestamp while offline. Why? Simply because the timestamp is totally generated on the server. And since you're offline and cannot communicate with the server, you cannot get an estimation. So there is no other system that can generate a timestamp other than the Firebase server.
Besides that, a date that can be generated on the client can be manipulated, so you cannot rely on that.
The option that you have in my opinion would be to have two fields. One is for the real timestamp, which you'll be received when you're back online and the other one is what the client provides. In that way, you make a comparison, if this is what you want.
Edit:
Firestore, is a Cloud hosted NoSQL database. So there is no way you can generate an accurate time while the user is offline. What you can certainly do is calculate the period of time while the user is offline. This means that you have to calculate the number of minutes the user has lost connectivity. Let's say a user losses connectivity for 10 minutes, when it regains connectivity, you receive the Firestore timestamp. Now, all you have to do is to subtract those 10 minutes from the received timestamp in order to have the accurate time that you're looking for.
I have a mongo database with a collection of objects, lets say orders.
Multiple mobile devices are saving and loading orders to that collection.
When one device saves a new order to the database, all devices should sync and download new data. I want to be able to sync every 3 seconds so everyone can have latest data at every moment.
Now my problem is how to accomplish that efficiently? I do not want to load all the data, and parse it on the client side. I want to download only the missing orders.
What would be the best practice way of achieving this?
One option is to do a differential update.
You'll need to record the time of the last update of each device in the DB and the time of each modification to the DB.
Then you only update with the changes that occurred after the last time the device updated.
Also, one last note, sync'ing every 3 seconds will drain the battery fairly quickly. Perhaps you should consider only updating a device when the device has something to send to the server. ie send a new order, and the server response contains all new orders.
Can people give me examples of why they would use coreData in an application?
I ask this because most apps are just clients to a central server where an API of some sort gives you the information you need.
In my case I'm writing a timesheet application for a web app which has an API and I'm debating if there is any value in replicating the data structure on my server in core data(Sqlite)
e.g
Project has many timesheets
employee has many timesheets
It seems to me that I can just connect to the API on every call for lists of projects or existing timesheets for example.
I realize for some kind of offline mode you could store locally in core data but this creates way more problems because you now have a big problem with syncing that data back to the web server when you get connection again.. e.g. the project selected for a timesheet no longer exists.
Can any experienced developer shed some light on there experiences on when core data is best practice approach?
EDIT
I realise of course there is value in storing local persistance but the key value of user defaults seems to cover most applications I can think of.
You shouldn't think of CoreData simply as an SQLite database. It's not JUST an SQLite database. Sure, SQLite is an option, but there are other options as well, such as in-memory and, as of iOS5, a whole slew of custom data stores. The biggest benefit with CoreData is persistence, obviously. But even if you are using an in-memory data store, you get the benefits of a very well structured object graph, and all of the heavy lifting with regards to pulling information out of or putting information into the data store is handled by CoreData for you, without you necessarily needing to concern yourself with what is backing that data store. Sure, today you don't care too much about persistence, so you could use an in-memory data store. What happens if tomorrow, or in a month, or a year, you decide to add a feature that would really benefit from persistence? With CoreData, you simply change or add a persistent data store, and all of your methods to get information out or in remain unchanged. The overhead for that sort of addition is minimal in comparison to if you were trying to access SQLite or some other data store directly. IMHO, that's the biggest benefit: abstraction. And, in essence, abstraction is one of the most powerful things behind OOP. Granted, building the Data Model just for in-memory storage could be overkill for your app, depending on how involved the app is. But, just as a side note, you may want to consider what is faster: Requesting information from your web service every time you want to perform some action, or requesting the information once, storing it in memory, and acting on that stored value for the remainder of the session. An in-memory data store wouldn't persistent beyond that particular session.
Additionally, with CoreData you get a lot of other great features like saving, fetching, and undo-redo.
There are basically two kinds of apps. Those that provide you with local functionality (games, professional applications, navigation systems...) and those that grant access to a remote service.
Your app seems to be in the second category. If you access remote services, your users will want to access new or real-time data (you don't want to read 2 week old Facebook posts) but in some cases, local caching makes sense (e.g. reading your mails when you're on the train with unstable network).
I assume that the value of accessing cached entries when not connected to a network is pretty low for your customers (internal or external) compared to the importance of accessing real-time-data. So local storage might be not necessary at all.
If you don't have hundreds of entries in your timetable, "normal" serialization (NSCoding-protocol) might be enough. If you only access some "dashboard-data", you will be able to get along with simple request/response-caching (NSURLCache can do a lot of things...).
Core Data does make more sense if you have complex data structures which should be synchronized with a server. This adds a lot of synchronization logic to your project as well as complexity from Core Data integration (concurrency, thread-safety, in-app-conflicts...).
If you want to create a "client"-app with a server driven user experience, local storage is not necessary at all so my suggestion is: Keep it as simple as possible unless there is a real need for offline storage.
It's ideal for if you want to store data locally on the phone.
Seriously though, if you can't see a need for it for your timesheet app, then don't worry about it and don't use it.
Solving the sync problems that you would have with an "offline" mode would be detailed in your design of your app. For example - don't allow projects to be deleted. Why would you? Wouldn't you want to go back in time and look at previous data for particular projects? Instead just have a marker on the project to show it as inactive and a date/time that it was made inactive. If the data that is being synced from the device is for that project and is before the date/time that it was marked as inactive, then it's fine to sync. Otherwise display a message and the user will have to sort it.
It depends purely on your application's design whether you need to store some data locally or not, if it is a real problem or a thin GUI client around your web service. Apart from "offline" mode the other reason to cache server data on client side might be to take traffic load from your server. Just think what does it mean for your server to send every time the whole timesheet data to the client, or just the changes. Yes, it means more implementation on both side, but in some cases it has serious advantages.
EDIT: example added
You have 1000 records per user in your timesheet application and one record is cca 1 kbyte. In this case every time a user starts your application, it has to fetch ~1Mbyte data from your server. If you cache the data locally, the server can tell you that let's say two records were updated since your last update, so you'll have to download only 2 kbyte. Now you should scale up this for several tens of thousands of user and you will immediately notice the difference of the server bandwidth and CPU usage.
I am still learning xcode and objective-c. I use to build app for iphone environment only.
However I am in need of realizing an application with an existing prefilled sql database.
For prefilling the database I wouldn't like to use code in the ditributed app, but I would rather prefer to have a separate app for doing that.
The reason is that, the app could only download the updated database, rather than a whole code update .
So, questions are:
is this a possible scenario
if yes, what kind of application
should I build in xcode for
prefilling database ?
thanks
There's no reason that you can't have one app that both uses the database and downloads updates. Keeping the database updated without downloading the whole thing is pretty simple.
If you record the creation and modification timestamps of rows in the database on the server and keep track of those same modification timestamps on the device, updating the database works like this:
The device determines latest modification timestamp it has for a given table. We'll call it latestTimestamp. It sends the latestTimestamp to the server.
The server compares the latestTimestamp to the creation and modification timestamps in the database. The server sends back data based on the comparison result:
If the modification timestamp is earlier than latestTimestamp it doesn't need to send the record, the device already has it;
If the modification timestamp is later than latestTimestamp and the creation timestamp is earlier than latestTimestamp, it sends the record back noting that it is to be updated in the device database;
If the modification timestamp is later than latestTimestamp and the creation timestamp is later than latestTimestamp, it sends the record back noting that it is to be added in the device database.
Lastly, the server database needs to keep track of deleted records and a deletion timestamp for every record recorded. If latestTimestamp is later than the deletion timestamp, it sends back that the record needs to be deleted.
Obviously it gets a bit more complicated when you have a variety of connected tables, but as long as things are sent back in the correct order, it works great.
Use asynchronous data requests (the ASIHTTPRequest library makes it a breeze) and update the data in the background while the user uses the app. If it's essential that the data be updated prior to any interaction with it you can display an activity indicator and have the user wait.
No need at all for a separate app.
I would discourage you from doing that. No matter it is a pre-filled-database-purpose app, or a normal-purpose app, Apple Review Team would treat them with the same procedure, leaving the developer waiting for weeks before that app is finally available on App Store.
Besides, as far as I know, communication between apps is still strictly limited. If the data you would like to transfer between your main app and your db app is larger than a few lines of, let's say, NSString, it might be technically un-plausible.
I am working on a regular iPhone app which pulls data from a server (XML, JSON, etc...), and I'm wondering what is the best way to implement synching data. Criteria are speed (less network data exchange), robustness (data recovery in case update fails), offline access and flexibility (adaptable when the structure of the database changes slightly, like a new column). I know it varies from app to app, but can you guys share some of your strategy/experience?
For me, I'm thinking of something like this:
1) Store Last Modified Date in iPhone
2) Upon launching, send a message like getNewData.php?lastModifiedDate=...
3) Server will process and send back only modified data from last time.
4) This data is formatted as so:
<+><data id="..."></data></+> // add this to SQLite/CoreData
<-><data id="..."></data></-> // remove this
<%><data id="..."><attribute>newValue</attribute></data></%> // new modified value
I don't want to make <+>, <->, <%>... for each attribute as well, because it would be too complicated, so probably when receive a <%> field, I would just remove the data with the specified id and then add it again (assuming id here is not some automatically auto-incremented field).
5) Once everything is downloaded and updated, I will update the Last Modified Date field.
The main problem with this strategy is: If the network goes down when I am updating something => the Last Modified Date is not yet updated => next time I relaunch the app, I will have to go through the same thing again. Not to mention potential inconsistent data. If I use a temporary table for update and make the whole thing atomic, it would work, but then again, if the update is too long (lots of data change), the user has to wait a long time until new data is available. Should I use Last-Modified-Date for each of the data field and update data gradually?
I would start by making the update routine atomic, since you'll have enough on your hands figuring out how to get the client-server communication working properly.
After that is a good time to consider tweaking it to be incremental, but only after you do some testing to figure out if it's really necessary. If you're tuning your update protocol to be as low bandwidth as possible, you might discover that even a "big" update is downloaded fast enough.
Another way to look at it is to ask yourself, how often is there going to be network trouble when an average user is doing a sync? You probably don't want to tune for unlikely scenarios.
If you are trying to optimize (minimize) the data transfer you may want to consider a different format than XML, since XML is fairly verbose. Or at least you may want to trade in XML readability for space by making each element name and attribute as small as possible, and eliminate all unnecessary whitespace.
Your basic scheme is good. The thing you need to do is to somehow make your updates idempotent so that you can restart a partially-completed transfer without risk. This is a better way to go than to try to implement some sort of true atomic commit (though you could do that too, using, eg, the SQLite database).
In our experience fairly large updates (10s of KB) can be downloaded quite rapidly, if the server is fast enough. No great need to break updates up into tiny bits. But certainly it won't hurt to try to minimize the amount of data transferred by keeping more granular info on "last update".
(And definitely you should use JSON rather than XML as your transmitted data representation.)
Wonder if you have considered using a Sync Framework to manage the synchronization. If that interests you can take a look at the open source project, OpenMobster's Sync service. You can do the following sync operations
two-way
one-way client
one-way device
bootup
Besides that, all modifications are automatically tracked and synced with the Cloud. You can have your app offline when network connection is down. It will track any changes and automatically in the background synchronize it with the cloud when the connection returns. It also provides synchronization like iCloud across multiple devices
Also, modifications in the Cloud are synched using Push notifications, so the data is always current even if it is stored locally.
In your case,
Criteria are speed (less network data exchange), robustness (data recovery in case update fails), offline access
Speed: Only the changes are sent across the network in both directions
Robustness: It stores data in a transactional store like sqlite and any failed updates are communicated in the SyncML payload. Only the successful operations are processed while the failed operations are re-tried during the next sync
Here is a link to the open source project: http://openmobster.googlecode.com
Here is a link to iPhone App Sync: http://code.google.com/p/openmobster/wiki/iPhoneSyncApp