I am building a stock portfolio app and am facing an issue with data flow management. I have 3 collections for users, transactions and accounts in firebase. I use a stream provider to fetch data for transactions from firebase into the flutter app. In addition, I have a service that fetches and saves the current stock data(price, percentage change, volume, etc) from a REST API into a list within the flutter app. I wanted to understand what is the best way to manage data from both these endpoints and be able to show them on the screen, update them whenever they get updated.
If
I would suggest having a proxy server in the middle to manage your communications for :
Managing Firebase collections
Fetching and save current stock price data
Load balancing and proxying.
The reasons for suggesting the same are:
Decoupling of data flow into microservices. Your Flutter app should just manage UI and hit the necessary api endpoints. Your logic for resolving client requests should only exist on the server. This will reduce your app bundle size and help with horizontal-scaling should the traffic increase.
Can cache common responses by applying Redis on this server.
Better application of security rules for both Firebase and REST API compared to Flutter application
Say tomorrow you wish to add more API services, this will just mean adding those endpoints on this server and hitting them with your Frontend application.
Having such an architecture will reduce the dependency of the entire stack on each independent service (be it your Flutter app or your Firebase backend).
Related
I am a rather fresh Flutter programmer so please excuse any flaws in the questions below…
I am struggling with a structural/ architecture dilemma. Here is the background:
App rationale:
my app allows its users to check little jobs available in their area and if they find time and are in a proper location to execute the job for a remuneration,
the app uses standard REST API (not Firebase) so that the server cannot be relied on sending status change notifications to trigger re-fetching of data,
the critical elements are (1) up-to date list of jobs for a given address - other user may have already taken on a job in an address (timed refresh of list e.g. every 5 mins), and (2) the app needs to keep track of the user’s location and accordingly ask the server for jobs if the user relocates by more than 2km in less than the refresh time,
The challenge:
I guess that on the basic level the app should have the following providers: (1) auth – providing the authToken, (2) geolocation – regularly checking user’s location, (3) jobList - for particular location (fetches high level job descriptions and addresses(, (4) jobDetails – fetches exact instructions for carrying a particular job,
as you can see: (2) geolocation and (3) jobList – need to refresh programmatically (at interval or on some change of geolocation), while (1) auth, (4) jobDetails are triggered by the user.
The Big Question ;) is … what is the proper architecture for the above type of app? More specifically:
should I use services for connecting to the server API and these would in turn be used by the providers?
how to ensure programmatic refetch of jobList on timer and relocation event from geolocation?
how to continually listen to location changes to detect a relocation but not overwhelming the app with processing?
should I store the (quickly outdating) jobLIst data just in its object class or should I use settings provider or a local db or maybe there is an easy way of storing the latest JSON response not to have to build the settings provider or db mapping?
in all my call to Auth api I need to provide the deviceId - how to make it available accross the app - this is pretty static but is needed in authentication so should checking it be a part of the auth provider?
If you could comment on the above or suggest a source of relevant examples I would be really grateful.
Thanks and cheers!
Here are my thoughts:
how to continually listen to location changes to detect a relocation but not overwhelming the app with processing?
You can rely on third party to do this for you. Such as: geolocator. With this, you can specify the amount of distance the user must have moved before the package notifies you of the change in user location.
should I store the (quickly outdating) jobLIst data just in its object class or ...
Since it is likely for a job listing app to use this data often and in various places, I would prefer to use db. It would be helpful in the long run too, if you plan to have some sort of analytics done on the mobile end or to gather any insights.
in all my call to Auth api I need to provide the deviceId - how to make it available accross the app ...
When you app is initialized, you could fetch the deviceID and store it in shared_preferences. Then in auth api, you could just retrieve it before making the API call.
should I use services for connecting to the server API and these would in turn be used by the providers?
As for geo location, geolocator can update you about the change in location and you could make an API call based on that.
However, if you plan to have a timer based approach to refresh your job listing, then you must realize that your users are likely to face issues arising from your inconsistent data. If you have plans to tackle it, then this implementation here might help. But I strongly feel that server supporting push notifications or maybe a web socket approach would be ideal here.
I'm a bit jealous of services like Google Cloud Firestore which achieve realtime sync between mobile app (or web app) and back-end. It makes me see plain-old HTTP GET back-ends as prehistoric.
I'm wondering if, in 2O19, it exists quite simple/scalable solutions/frameworks to achieve this on my own back-end.
I've heard about sockets but it looks costly and quite difficult to setup (maybe I'm wrong). Silent notifications maybe? (but again it adds a layer of complexity of managing that and we don't have 100% confidence that every notification will reach its target).
I understand conflicts is the most sensitive topic so even a readonly solution (only back-end can modify entities) would be great.
There are solutions for real-time synchronization of data. Most of them use WebSockets or SSE (Server Side Events) for transportation, which also goes for Google Cloud Firestore. And many of the existing solutions are databases with synchronization capabilities, just like Firestore. But there are other options too.
Existing solutions
Assuming you are not looking for push solutions, such as PubNub or Pusher, but rather seamless synchronization of data, then I can mention a few to get you started:
Resgate
CouchDB
RethinkDB
DeepStream
Example using Resgate
Realtime API gateways such as Resgate (which I am clearly baised towards :) ), lets you write (micro)services in the language of your choice in a similar fashion as you would write an ordinary HTTP webservice. The gateway then exposes the resources in a REST and realtime API, and keeps the clients synchronized.
C# service using ResgateIO.Service
ResService service = new ResService("example");
service.AddHandler("mymodel", new DynamicHandler()
.SetGet(r => r.Model(new {
message = "Hello, World!",
}))
.SetAccess(r => r.AccessGranted()));
service.Serve("nats://127.0.0.1:4222");
But with the addition that you can send events to update and synchronize all clients:
service.With("example.mymodel", r => r.ChangeEvent(new Dictionary<string, object> {
{ "message", "Hello, StackOverflow!" }
}));
The client can then fetch the data, and listen for updates:
Javascript client using ResClient
let client = new ResClient('ws://localhost:8080');
client.get('example.mymodel').then(model => {
console.log(model.message); // Hello, World!
model.on('change', () => {
console.log(model.message); // Updated to: Hello, StackOverflow!
});
});
Considerations
All of the solutions mentioned above (and there are more for those who seek) have with their strengths and weaknesses in areas such as:
Resilience - handling of lost messages and connections
Access control - granting access and withdrawing access to a subscription
Live queries - fetching partial or filtered data that is updated live
Offline support - working with the data while offline
Scaling
Database requirements
Simplicity of usage
Just look around and see which solution suits your needs best.
There are many solutions, the more I search the more I find.
I use the couchbase lite stack which consists of:
Front end cross platform Couchbase Lite (CBL) database which runs with an invisible synchronizer
Backend Couchbase database cluster
Couchbase sync gateway service which synchronizes data between FE and BE on WebSocket basis
More details: Couchbase Mobile
I’m building a new web application which needs to work seamlessly even when there is no internet connection. I’ve selected Angular and am building a PWA as it comes with built-in functionality to make the application work offline. So far, I have the service worker working perfectly and driven by the manifest file, this very nicely caches the static content and I’ve set it to cache a bunch of API requests which I want to use whilst the application is offline.
In addition to this, I’ve used localStorage to store attempts to invoke put, post and delete API requests when the user is offline. Once the internet connection is re-established, the requests stored in localStorage are sent to the server.
This far in my proof of concept, the user can access content whilst offline, edit data and the data gets synced with the server once the user’s internet connection is re-established. This is where my quandary begins though. There is API request data cached automatically by the service worker as defined in the manifest file, and there is a separate store of data for data edits whilst offline. This leads to a situation where the user edits some data, saves the data, refreshes the page and the data is served by the service worker cached API.
Is there a built in mechanism to update API data cached automatically by the service worker? I don’t fancy trying to unpick this manually as it seems hacky and I can’t imagine it’ll be future proof as service workers evolve.
If there isn’t a standard way to achieve what I need to do, is it common for developers to take full control of offline data by storing it all in IndexedDB/localStorage manually? i.e. I could invoke API requests and write some code which caches the results in a structured format in IndexedDB to form an offline database, then writes back to the offline database whenever the user edits some data, and uploads any data edits when the user is back online. I don’t envisage any technical problems with doing this, it just seems like a lot of effort to achieve something which I was hoping to be standard functionality.
I’m fairly new to developing with Angular, but have many years of other development experience. So please forgive me if I’m asking obvious questions, I just can’t seem to find a good article on best practices for working with data storage with service workers.
Thanks
I have a project where my users can edit local data when they are offline and I use Cloud Firestore to have a local database cached available. If I understood you correctly, this would be exactly your requirement.
The benefit of this solution is that with just one line of code, you get not only a local db, but also all the changes made offline are automatically synchronised with the server once the client gets online again.
firebase.firestore().enablePersistence()
.catch(function(err) {
// log the error
});
// Subsequent queries will use persistence, if it was enabled successfully
If using this NoSQL database is an option for you I would go with it, otherwise you need to implement the local updates by yourself as there is not a built in solution for that.
We are developing an application having many screens. Each screen data is coming from Rest API. What the best practice to store the session data(screen data) at backed?
For example- I need data of screen two(includes the screen input data and response from rest API) in 4th screen. For this I want to store the rest response of screen two in server side.
I came of with two scenarios for this, if anyone has experience please help:
1) Session management using REDIS-- but this is mostly used in clustering environment.
2) Session management using spring security and spring session management.
Please suggest the better way of doing it.
Details:
Spring-boot application will be hosted in cloud
Also the question is not related to security, authentication, authorization.
Kindly help me with best practice to move the data to different screens.
For example- I need data of screen two(includes the screen input data and response from rest API) in 4th screen. For this I want to store the rest response of screen two in server side.
What you are describing there is a violation of the Stateless architectural constraint of REST.
The "right" answer is to take one of two approaches; one is to store the "session data" on the client -- the server sends the data back to the client (for example, as fixed/hidden fields in the form) in the response. The other is to use the client actions to modify a resource (think shopping cart).
The core problem is this: the stateless constraint means that the server is only operating on the current request; the server only ever sees requests, not state changes (ex: the client can hit the back button, or otherwise jump to some other state in its history, or fetch additional state from somewhere else).
If you use the "modify a resource" approach, you may want to review RFC 7232: Conditional Requests, and think about whether or not your use case needs to worry about the "lost update problem".
I will build a dashboard system for my apps, where a page will have several widgets that draw charts, tables and glyphs representing potentially unrelated data.
The client will be HTML5 and I can push for only modern web browser.
My big problem is what backend use for this. I want to store "tables" for use in the charts and in real-time update the widgets.
For example, a invoicing widget will show how much $$ have been collected today. In the "table" will have a row for each total of the invoice:
inv = 1; total = 50
Total: 50
and the widget will draw that. When new data is pushed:
inv = 2; total = 100
Total: 150
The widget will show in realtime the total to the end-user.
The data is private for the user company. Eventually I will need to purge too old data (ie: I only need to keep as much data is necessary to proper evaluation of the info need for the end-user. For example, only keep 1 month of invoicing totals).
I'm thinking in use something like http://www.firebase.com/ or http://pusher.com/ but I suspect only solve the "notify in realtime" part of the equation. As far as I understand, they not let me get past data (ie: If the data is update in the weekend and the user open his dashboard to see what happened)
Then I see http://derbyjs.com/ and the possibility to use mongodb.
I wonder which backend/platform will bring me closer to the build of this system. I have experience with python/django/.net/postgress but could accept the use of something else if solve best this kind of app behavior.
Firebase offers both the "notify in relatime" part that you mention, as well as persistent data storage. Take a look at the tutorial, which walks you through building a real-time persisted chat app (the past chat messages are stored in Firebase and are sent back to the client every time you reload). And you can do much more complicated stuff like the real-time charts / widgets that you mention as well.
The big limitation with Firebase right now is that we're in closed beta and the data is currently unprotected (anybody can read and write your data). The security features are coming soon though.
Some other backend platforms you may want to evaluate are: Meteor and Simperium. Firebase and Simperium are cloud services where your data is stored in the cloud and you don't have to manage any servers of your own, while Meteor and DerbyJS are platforms that you have to install and run on your own server.
I would recommend signalR. It's amazing and you can literally do anything with it. Check it out: www.signalr.net and if you have any problems simply go to www.jabbr.net You will find a very helpful community there. I implemented a notification mechanism similar to facebook together with real time monitoring and a small chat in the same web site.