I am new to Amplify datastore and decided to use Amplify Datastore with flutter app to provide offline mode feature to the users. I successfully implemented my project with Datastore but I would like to clear the local storage when the user logs out. And if logs back, I want user's data to be there but it seems like datastore sync data from local to cloud out of the box but do not sync cloud to local storage back since I can't see any records (After login back or after reinstalling).
So are we required to retrieve the data from dynamoDB using GraphQL APIs and then save it to the local datastore or is there any out-of-the-box function provided by Amplify to retrieve data from the cloud?
I also notice after a few logins and logouts it somehow fetched the data from the cloud and I was able to see the records on my application. Not sure what's going on :/
Amplify DataStore by default automatically syncs all data from the backend to the local device. It sounds like already found out about DataStore.clear() and use that to clear local storage when the user logs out.
The next time the app starts and the user logs in datastore will connect to the backend. When the app makes the first query data will start syncing from the backend. If you want to cause sync to start earlier use DataStore.start(). In either case, the initial query might not return all the data that you want as sync takes time. You can wait for events that the data has been synced before running queries, or make sure there is a certain amount of data available before the query is run.
More info on start and close methods
The event you can monitor to see if the datastore is ready and all data from the cloud has been synced locally is ready
Related
As per the answer in the link below:
Does Firebase Firestore keep a local copy of snapshot
The Firestore SDK keeps a local copy of:
All data that you have an active listener for.
All pending writes.
In addition, if offline persistence is enabled,
it also keeps a local copy of data that
your code has recently read.
Does this mean that even if offline persistence is disabled on the web, onSnapshot event listeners and pending writes will still be stored locally to cache regardless?
I'm worried about sensitive data on shared computers for these listeners and pending writes that's stored locally (if they did not successfully write to the server, i.e. went offline in the process, etc.). Can malicious users access this data somehow? Is this a flaw of firestore for web apps?
If offline persistence is disabled, only pending writes are stored in the local cache.
You should assume that all data that is entered on the local device can be intercepted by a malicious actor who gets access to that device. If that is not acceptable for your use-case, consider not using a Firebase SDK for Firestore but using the REST API or gRPC API directly.
I’m building a new web application which needs to work seamlessly even when there is no internet connection. I’ve selected Angular and am building a PWA as it comes with built-in functionality to make the application work offline. So far, I have the service worker working perfectly and driven by the manifest file, this very nicely caches the static content and I’ve set it to cache a bunch of API requests which I want to use whilst the application is offline.
In addition to this, I’ve used localStorage to store attempts to invoke put, post and delete API requests when the user is offline. Once the internet connection is re-established, the requests stored in localStorage are sent to the server.
This far in my proof of concept, the user can access content whilst offline, edit data and the data gets synced with the server once the user’s internet connection is re-established. This is where my quandary begins though. There is API request data cached automatically by the service worker as defined in the manifest file, and there is a separate store of data for data edits whilst offline. This leads to a situation where the user edits some data, saves the data, refreshes the page and the data is served by the service worker cached API.
Is there a built in mechanism to update API data cached automatically by the service worker? I don’t fancy trying to unpick this manually as it seems hacky and I can’t imagine it’ll be future proof as service workers evolve.
If there isn’t a standard way to achieve what I need to do, is it common for developers to take full control of offline data by storing it all in IndexedDB/localStorage manually? i.e. I could invoke API requests and write some code which caches the results in a structured format in IndexedDB to form an offline database, then writes back to the offline database whenever the user edits some data, and uploads any data edits when the user is back online. I don’t envisage any technical problems with doing this, it just seems like a lot of effort to achieve something which I was hoping to be standard functionality.
I’m fairly new to developing with Angular, but have many years of other development experience. So please forgive me if I’m asking obvious questions, I just can’t seem to find a good article on best practices for working with data storage with service workers.
Thanks
I have a project where my users can edit local data when they are offline and I use Cloud Firestore to have a local database cached available. If I understood you correctly, this would be exactly your requirement.
The benefit of this solution is that with just one line of code, you get not only a local db, but also all the changes made offline are automatically synchronised with the server once the client gets online again.
firebase.firestore().enablePersistence()
.catch(function(err) {
// log the error
});
// Subsequent queries will use persistence, if it was enabled successfully
If using this NoSQL database is an option for you I would go with it, otherwise you need to implement the local updates by yourself as there is not a built in solution for that.
a question about local and remote storage of user data. Is there a best practices for the common situation where a user accesses data from an API and can favourite or otherwise personalise the data.
I have seen tutorials, e.g. a movie browsing app, where the use can make a list of favourite movies, where this personalised data is stored locally (e.g. in sqflite) and other tutorials where this data is stored remotely, eg. firebase. And firebase has an offline mode, so that data can be synced later. In that case, is it a common use case to set up local storage as well as cloud storage? Is there a common practice for this situation?
Thanks for any insights.
This is not specifically a Flutter question, more of a general app development question. It's very common to have both local and cloud "storage" but I wouldn't think of it that way. If you're interacting with an API backend I wouldn't consider it as the cloud storage for your app. Instead look at it as a different component within your applications overall architecture. You API/Backend component, this way it's not apart of your app instead it's something your app interacts with.
I assume you know the purpose of your API. Returns your data you want to see, keeps track of user profile information and other sensitive information.
When it comes to local storage I'd say the most common scenarios for local storage is results caching and storing information that the API requires on every session to make the user experience a bit better. See some examples below for both:
On instagram they store your "Feed watermark" which is a string value that is linked to a specific set of results so that when you open the app and request again they return that set of results, plus anything new - Local storage
They also "store locally" (better referred to as caching) a small set of your feeds from your posts, a list of user profiles that has stories on them and your DM's for instant and offline access. This way when the app loads up it has something to show while performing the action to get the new information. - Caching
They also store your login token, that never expires. - Local storage
tl;dr: Yes. If you need data on every session to use your API store that locally in a secure way and use that to interact with your "Cloud storage".
I'm creating a multi page application where i need to create and store transactions even when the users are offline. How do i achieve this using firestore ? Also i need some idea on how to persist the data received from the firestore locally.
You can't run transactions when offline,but if you think that your data is not changed while you are offline you can get the data from cache and update it it there using dbRef.addSnapshotListener(MetadataChanges.INCLUDE) and dbRef.update()
How to perform transactions in Firestore when the user is offline?
You cannot! Transactions are not supported for offline use, they can't be cached or saved for later. This is because a transaction absolutely requires round trip communications with the server in order to ensure that the code inside the transaction completes successfully. So you can use transaction only while online because the transactions are network dependent.
Also I need some idea on how to persist the data received from the Firestore locally.
According to the official documentation of Cloud Firestore regarding offline persistence:
For Android and iOS, offline persistence is enabled by default. To disable persistence, set the PersistenceEnabled option to false.
For the web, offline persistence is disabled by default. To enable persistence, call the enablePersistence method. Cloud Firestore's cache isn't automatically cleared between sessions. Consequently, if your web app handles sensitive information, make sure to ask the user if they're on a trusted device before enabling persistence.
Important: For the web, offline persistence is an experimental feature that is supported only by Chrome, Safari, and Firefox web browsers. Also, if a user opens multiple browser tabs that point to the same Cloud Firestore database, and offline persistence is enabled, Cloud Firestore will work correctly only in the first tab.
Edit:
The Firestore SDK for Android has a local cache that's enabled by default. So all read operations will come from the cache when there is no connectivity. So Firestore provides this feature to handle offline data. This means that if the user tries to add/delete documents while offline, every operation is added to a queue. Once the user regains the connection, every change that is made while offline will be updated on Firebase servers. In other words, all queries will be committed on the server.
Please also note that when you are offline, pending writes that have not yet been synced to the server are held in a queue. If you do too many write operations without going online to sync them, that queue will grow fast and it will not slow down only the write operations it will also slow down your read operations. So I suggest using this database for its online capabilities.
My app parses XML from remote server and store the objects in Core Data(SQLite storage). So that user can browse the material when OFFLINE by reading from local storage.
User may make changes to objects when browsing offline which gets stored locally in Core Data SQLite store. Another User makes some changes to object on Remote server and it is stored there. Now when I detect internet connection, my app should sync my local storage with remote server. Which means remote server is updated with changes I made to my Core Data(SQLite storage) when I was offline and my local storage - Core Data(SQLite storage) needs to be updated with what ever changes other user made to remote server.
For example there is a forum and it is stored in my local storage so that I can read and reply when I am traveling. When later on internet is accessible. My app should automatically put all my replies stored in core data to remote server and also bring other posts on remote server into my local storage.
Remote server is sending XML which I'm parsing and storing in Coredata. My problem is how to sync it ?
How both ways communication happens when there is a change?
How to sync only data which has changed and not to IMPORT whole remote server DB and vice-versa ?
I know one of the way to do it..
add one more field to your local and server database. i.e. Timestamp.
when user change data on the local database change the Timestamp to current time.
do same on the server..i.e. When someone edit data on the server change Timestamp to current time.
When user connects to internet... check local Timestamp to Server timestamp..
case 1 Both are same - Nothing to do
case 2 local time > server time - use sql to get all the data having timestamp greater than server timestamp.. and upload it on the server...
case 3 local < server .... get all the records greater than the local timestamp and add it to the local database..
I am not sure if there is any better way... but this surely works...
One solution could be
iphone syncs the changes to the server
server merges the new and old stuff
iphone gets the new changes (from the merge) from the server
So let the server be the master which should know how to merge stuff and the clients should only download the data incrementally after some changes.
Transactions. You want to follow the ACID rules for transactions. Essentially you have to make sure that data has not be updated that you have not refreshed locally before altering your local write to update.
So the easiest way is have a user get the most recent update from the server, then overwrite that and make sure that with timestamps no othe update happens during that process. Even better yet, use blocking with threads to insure that nothing else happens.
If you Google transactions or ACID there will be a lot of info out there. It's a big area in RDBMS environments where many users can corrupt the data and locks must be held between writes and updates.