I am building a stock portfolio app and am facing an issue with data flow management. I have 3 collections for users, transactions and accounts in firebase. I use a stream provider to fetch data for transactions from firebase into the flutter app. In addition, I have a service that fetches and saves the current stock data(price, percentage change, volume, etc) from a REST API into a list within the flutter app. I wanted to understand what is the best way to manage data from both these endpoints and be able to show them on the screen, update them whenever they get updated.
If
I would suggest having a proxy server in the middle to manage your communications for :
Managing Firebase collections
Fetching and save current stock price data
Load balancing and proxying.
The reasons for suggesting the same are:
Decoupling of data flow into microservices. Your Flutter app should just manage UI and hit the necessary api endpoints. Your logic for resolving client requests should only exist on the server. This will reduce your app bundle size and help with horizontal-scaling should the traffic increase.
Can cache common responses by applying Redis on this server.
Better application of security rules for both Firebase and REST API compared to Flutter application
Say tomorrow you wish to add more API services, this will just mean adding those endpoints on this server and hitting them with your Frontend application.
Having such an architecture will reduce the dependency of the entire stack on each independent service (be it your Flutter app or your Firebase backend).
I’m building a new web application which needs to work seamlessly even when there is no internet connection. I’ve selected Angular and am building a PWA as it comes with built-in functionality to make the application work offline. So far, I have the service worker working perfectly and driven by the manifest file, this very nicely caches the static content and I’ve set it to cache a bunch of API requests which I want to use whilst the application is offline.
In addition to this, I’ve used localStorage to store attempts to invoke put, post and delete API requests when the user is offline. Once the internet connection is re-established, the requests stored in localStorage are sent to the server.
This far in my proof of concept, the user can access content whilst offline, edit data and the data gets synced with the server once the user’s internet connection is re-established. This is where my quandary begins though. There is API request data cached automatically by the service worker as defined in the manifest file, and there is a separate store of data for data edits whilst offline. This leads to a situation where the user edits some data, saves the data, refreshes the page and the data is served by the service worker cached API.
Is there a built in mechanism to update API data cached automatically by the service worker? I don’t fancy trying to unpick this manually as it seems hacky and I can’t imagine it’ll be future proof as service workers evolve.
If there isn’t a standard way to achieve what I need to do, is it common for developers to take full control of offline data by storing it all in IndexedDB/localStorage manually? i.e. I could invoke API requests and write some code which caches the results in a structured format in IndexedDB to form an offline database, then writes back to the offline database whenever the user edits some data, and uploads any data edits when the user is back online. I don’t envisage any technical problems with doing this, it just seems like a lot of effort to achieve something which I was hoping to be standard functionality.
I’m fairly new to developing with Angular, but have many years of other development experience. So please forgive me if I’m asking obvious questions, I just can’t seem to find a good article on best practices for working with data storage with service workers.
Thanks
I have a project where my users can edit local data when they are offline and I use Cloud Firestore to have a local database cached available. If I understood you correctly, this would be exactly your requirement.
The benefit of this solution is that with just one line of code, you get not only a local db, but also all the changes made offline are automatically synchronised with the server once the client gets online again.
firebase.firestore().enablePersistence()
.catch(function(err) {
// log the error
});
// Subsequent queries will use persistence, if it was enabled successfully
If using this NoSQL database is an option for you I would go with it, otherwise you need to implement the local updates by yourself as there is not a built in solution for that.
Context
Greetings,
One day I randomly found RethinkDB and I was really fascinated by the whole real-time changes thing. In order to learn how to use this tool I quickly spinned up a container running RethinkDB and i started making a small project. I wanted to make something very simple therefore i thought about creating a service in which speakers can create room and the audience can ask questions. Other users can upvote questions in order to let the speaker know which one are the best. Obviously this project has a lot of realtime needs that i believe are best satisfied by using RethinkDB.
Design
I wanted to use a vary specific set of tools for this. The backend would be made in Laravel Lumen, the frontend in Vue.JS and the database of course would be RethinkDB.
The problem
RethinkDB as it seems is not designed to be exposed to the end user directly despite the fact that no security concern exists.
Assuming that the user only needs to see the questions and the upvoted in real time, no write permissions are needed and if a user changed the room ID nothing bad will happen since the rooms are all publicly accessible.
Therefore something is needed in order to await data updates and push it through a socket to the client (socket.io for example or pusher).
Given the fact that the backend is written in PHP i cannot tell Lumen to stay awake and wait for data updates. From what i have seen from the online tutorials a secondary system should be used that should listen for changes and then push them. (lets say a node.js service for example)
This is understandable however i strongly believe that this way of transferring the data to the user is inefficient and it defeats the purpose of RethinkDB.
If I have to send the action from the client's computer (user asks a question), save it to the database, have a script that listens for changes, then push the changes to socket.io and finally have the client (vue.js) act when a new event arrives, what is the point of having a real-time database in the first place?
I could avoid all this headache simply by having the Lumen app push the event directly to socket.io and user any other database system instead.
I really cant understand the point of all this. I am not experienced with no-sql databases by any means but i really want to experiment with them.
Thank you.
This is understandable however i strongly believe that this way of transferring the data to the user is inefficient and it defeats the purpose of RethinkDB.
RethinkDB has no built in mechanism to transfer data to end-users. It has no access control (in the conventional sense) as well. The common way, like you said, is to spin up one / multiple node instance(s) running socket.io. On each instance you can listen on your RethinkDB change streams and use socket.io's broadcast functionality. This would be a common way, but as RethinkDB's streams are pretty optimized, you could also open a change stream for every incoming socket.io connection.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I am thinking about buildning a REST API with both websockets and http where I use websockets to tell the client that new data is available or provide the new data to the client directly.
Here are some different ideas of how it could work:
ws = websocket
Idea A:
David get all users with GET /users
Jacob add a user with POST /users
A ws message is sent to all clients with info that a new user exist
David recive a message by ws and calls GET /users
Idea B:
David get all users with GET /users
David register to get ws updates when a change is done to /users
Jacob add a user with POST /users
The new user is sent to David by ws
Idea C:
David get all users with GET /users
David register to get ws updates when a change is done to /users
Jacob add a user with POST /users and it gets the id 4
David receive the id 4 of the new user by ws
David get the new user with GET /users/4
Idea D:
David get all users with GET /users
David register to get ws updates when changes is done to /users.
Jacob add a user with POST /users
David receive a ws message that changes is done to /users
David get only the delta by calling GET /users?lastcall='time of step one'
Which alternative is the best and what are the pros and cons?
Is it another better 'Idea E'?
Do we even need to use REST or is ws enought for all data?
Edit
To solve problems with data getting out of sync we could provide the header"If-Unmodified-Since"https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Unmodified-Sinceor "E-Tag" https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag or both with PUT requests.
Idea B is for me the best, because the client specifically subscribes for changes in a resource, and gets the incremental updates from that moment.
Do we even need to use REST or is ws enought for all data?
Please check: WebSocket/REST: Client connections?
I don't know Java, but I worked with both Ruby and C on these designs...
Funny enough, I think the easiest solution is to use JSON, where the REST API simply adds the method data (i.e. method: "POST") to the JSON and forwards the request to the same handler the Websocket uses.
The underlying API's response (the response from the API handling JSON requests) can be translated to any format you need, such as HTML rendering... though I would consider simply returning JSON for most use cases.
This helps encapsulate the code and keep it DRY while accessing the same API using both REST and Websockets.
As you might infer, this design makes testing easier, since the underlying API that handles the JSON can be tested locally without the need to emulate a server.
Good Luck!
P.S. (Pub/Sub)
As for the Pub/Sub, I find it best to have a "hook" for any update API calls (a callback) and a separate Pub/Sub module that handles these things.
I also find it more resource friendly to write the whole data to the Pub/Sub service (option B) instead of just a reference number (option C) or an "update available" message (options A and D).
In general, I also believe that sending the whole user list isn't effective for larger systems. Unless you have 10-15 users, the database call might be a bust. Consider the Amazon admin calling for a list of all users... Brrr....
Instead, I would consider dividing this to pages, say 10-50 users a page. These tables can be filled using multiple requests (Websocket / REST, doesn't matter) and easily updated using live Pub/Sub messages or reloaded if a connection was lost and reestablished.
EDIT (REST vs. Websockets)
As For REST vs. Websockets... I find the question of need is mostly a subset of the question "who's the client?"...
However, once the logic is separated from the transport layer, than supporting both is very easy and often it makes more sense to support both.
I should note that Websockets often have a slight edge when it comes to authentication (credentials are exchanged once per connection instead of once per request). I don't know if this is a concern.
For the same reason (as well as others), Websockets usually have an edge with regards to performance... how big an edge over REST depends on the REST transport layer (HTTP/1.1, HTTP/2, etc').
Usually these things are negligible when it comes time to offer a public API access point and I believe implementing both is probably the way to go for now.
To summarize your ideas:
A: Send a message to all clients when a user edits data on the server. All users then request an update of all data.
-This system may make a lot of unnecessary server calls on behalf of clients who are not using the data. I don't recommend producing all of that extra traffic as processing and sending those updates could become costly.
B: After a user pulls data from the server, they then subscribe to updates from the server which sends them information about what has changed.
-This saves a lot of server traffic, but if you ever get out of sync, you're going to be posting incorrect data to your users.
C: Users who subscribe to data updates are sent information about which data has been updated, then fetch it again themselves.
-This is the worst of A and B in that you'll have extra round trips between your users and servers just to notify them that they need to make a request for information which may be out of sync.
D: Users who subscribe to updates are notified when any changes are made and then request the last change made to the server.
-This presents all of the problems with C, but includes the possibility that, once out of sync, you may send data that will be nonsense to your users which might just crash the client side app for all we know.
I think that this option E would be best:
Every time data changes on the server, send the contents of all the data to the clients who have subscribed to it. This limits the traffic between your users and the server while also giving them the least chance of having out of sync data. They might get stale data if their connection drops, but at least you wouldn't be sending them something like Delete entry 4 when you aren't sure whether or not they got the message that entry 5 just moved into slot 4.
Some Considerations:
How often does the data get updated?
How many users need to be updated each time an update occurs?
What are your transmission
costs? If you have users on mobile devices with slow connections, that will affect how often and how much you can afford to send to them.
How much data gets updated in a given update?
What happens if a user sees stale data?
What happens if a user gets data out of sync?
Your worst case scenario would be something like this: Lots of users, with slow connections who are frequently updating large amounts of data that should never be stale and, if it gets out of sync, becomes misleading.
I personally have used Idea B in production and am very satisfied with the results. We use http://www.axonframework.org/, so every change or creation of an entity is published as an event throughout the application. These events are then used to update several read models, which are basically simple Mysql tables backing one or more queries. I added some interceptors to the event processors that update these read models so that they publish the events they just processed after the data is committed to the DB.
Publishing of events is done through STOMP over web sockets. It is made very simple is you use Spring's Web Socket support (https://docs.spring.io/spring/docs/current/spring-framework-reference/html/websocket.html). This is how I wrote it:
#Override
protected void dispatch(Object serializedEvent, String topic, Class eventClass) {
Map<String, Object> headers = new HashMap<>();
headers.put("eventType", eventClass.getName());
messagingTemplate.convertAndSend("/topic" + topic, serializedEvent, headers);
}
I wrote a little configurer that uses Springs bean factory API so that I can annotate my Axon event handlers like this:
#PublishToTopics({
#PublishToTopic(value = "/salary-table/{agreementId}/{salaryTableId}", eventClass = SalaryTableChanged.class),
#PublishToTopic(
value = "/salary-table-replacement/{agreementId}/{activatedTable}/{deactivatedTable}",
eventClass = ActiveSalaryTableReplaced.class
)
})
Of course, that is just one way to do it. Connecting on the client side may look something like this:
var connectedClient = $.Deferred();
function initialize() {
var basePath = ApplicationContext.cataDirectBaseUrl().replace(/^https/, 'wss');
var accessToken = ApplicationContext.accessToken();
var socket = new WebSocket(basePath + '/wss/query-events?access_token=' + accessToken);
var stompClient = Stomp.over(socket);
stompClient.connect({}, function () {
connectedClient.resolve(stompClient);
});
}
this.subscribe = function (topic, callBack) {
connectedClient.then(function (stompClient) {
stompClient.subscribe('/topic' + topic, function (frame) {
callBack(frame.headers.eventType, JSON.parse(frame.body));
});
});
};
initialize();
Another option is to use Firebase Cloud Messaging:
Using FCM, you can notify a client app that new email or other data is
available to sync.
How does it work?
An FCM implementation includes two main components for sending and
receiving:
A trusted environment such as Cloud Functions for Firebase or an app server on which to build, target and send messages.
An iOS, Android, or Web (JavaScript) client app that receives messages.
Client registers its Firebase key to a server. When updates are available, server sends push notification to the Firebase key associated with the client. Client may receive data in notification structure or sync it with a server after receiving a notification.
Generally you might have a look at current "realtime" web frameworks like MeteorJS which tackle exactly this problem.
Meteor in specific works more or less like your example D with subscriptions on certain data and deltas being sent out after changes only to the affected clients. Their protocol used is called DDP which additionally sends the deltas not as overhead prone HTML but raw data.
If websockets are not available fallbacks like long polling or server sent events can be used.
If you plan to implement it yourself i hope these sources are some kind of inspiration how this problem has been approached. As already stated the specific use case is important
The answer depends on your use case. For the most part though I've found that you can implement everything you need with sockets. As long as you are only trying to access your server with clients who can support sockets. Also, scale can be an issue when you're using only sockets. Here are some examples of how you could use just sockets.
Server side:
socket.on('getUsers', () => {
// Get users from db or data model (save as user_list).
socket.emit('users', user_list );
})
socket.on('createUser', (user_info) => {
// Create user in db or data model (save created user as user_data).
io.sockets.emit('newUser', user_data);
})
Client side:
socket.on('newUser', () => {
// Get users from db or data model (save as user_list).
socket.emit('getUsers');
})
socket.on('users', (users) => {
// Do something with users
})
This uses socket.io for node. I'm not sure what your exact scenario is but this would work for that case. If you need to include REST endpoints that would be fine too.
With all great information all the great people added before me.
I found that eventually there is no right or wrong, its simply goes down to what suits your needs:
lets take CRUD in this scenario:
WS Only Approach:
Create/Read/Update/Deleted information goes all through the websocket.
--> e.g If you have critical performance considerations ,that is not
acceptable that the web client will do successive REST request to fetch
information,or if you know that you want the whole data to be seen in
the client no matter what was the event , so just send the CRUD events
AND DATA inside the websocket.
WS TO SEND EVENT INFO + REST TO CONSUME THE DATA ITSELF
Create/Read/Update/Deleted , Event information is sent in the Websocket,
giving the web client information that is necessary to send the proper
REST request to fetch exactly the thing the CRUD that happend in server.
e.g. WS sends UsersListChangedEvent {"ListChangedTrigger: "ItemModified" , "IdOfItem":"XXXX#3232" , "UserExtrainformation":" Enough info to let the client decide if it relevant for it to fetch the changed data"}
I found that using WS [Only for using Event Data] and REST
[To consume the data ]is better because:
[1] Separation between reading and writing model, Imagine you want to add some runtime information when your data is retrieved when its read from REST , that is now achieved because you are not mixing Write & Read models like in 1.
[2] Lets say other platform , not necessarily web client will consume this data.
so you just change the Event trigger from WS to the new way, and use REST to
consume the data.
[3] Client do not need to write 2 ways to read the new/modified data.
usually there is also code that reads the data when the page loads , and not
through the websocket , this code now can be used twice , once when page
loads , and second when WS triggered the specific event.
[4] Maybe the client do not want to fetch the new User because its showing currently only a view of old Data[E.g. users] , and new data changes is not in its interest to fetch ?
i prefer the A, it allows client the flexibility whether or not to update the existing data.
also with this method, implementation and access control becomes much more easier.
for example you can simply broadcast the userUpdated event to all users, this saves having a client list for do specific broadcasts and the Access Controls and Authentications applied for your REST Route wont have to change to reapplied again because the client is gonna make a GET request again.
Many things depends on the what kind of application you are making.
I have a database (MySQL) to which data is being written to. I need to push new records and changed records to UI. A few constraints here: I do not have control on the code which writes to this database and I cannot modify it to write to a queue.
So far, I am reading the DB periodically for changes and new additions (using a last update timestamp) and pushing that data to a mongo db (as I do not want to hit main MySQL server for every request). Then I push these changes to frontend using cramp (ruby framework) and server sent events. To maintain per user queue, I have redis in the mix.
I realize that this is a convulated way of doing realtime push. I was wondering if there is a more neat solution to this mess.
If you want to push data realtime from the server, then make use of technologies that provide real time access. I would recommend you to make use of Websockets.
The only issue is websockets is not supported by all the browsers, to take care of this you can use the available frameworks built over websockets that provide fallback to protocols supported by the browsers such as long polling, streaming etc. Following are the frameworks which I would suggest to use:
Atmosphere framework - https://github.com/Atmosphere/atmosphere
Play framework! - http://www.playframework.org/