RN Web + Firebase: snapshot listeners unsubscribe in unmount vs in global unmount (using context) - google-cloud-firestore

In short: which is most memory + cost efficient way to use Firestore snapshot listeners, unmount them always at screen unmount or have the unsubscribe function in context and unmount when whole site "unmounts"?
Lets say in home screen I use snapshot listener for collection "events" which has 100 documents. Now I navigate through the site and return to home screen 2 more times during using the site. In this case which is better memory and cost efficiently wise (is there also other things to consider) and is there drawbacks?
to mount and unmount the listener on each mount and unmount of the home screen.
to mount on home screen and to unmount in whole site "unmount" (for example using window.addEventListener('beforeunload', handleSiteClose).
The usage of first is probably familiar with most but usage of the second could be done with something like this:
-Saving listener unsubscribe function in context with collection name as key:
const { listenerHolder, setListenerHolder } = DataContext();
useEffect(() => {
const newListeners = anyDeepCopyFunction(listenerHolder);
const collection = 'events';
if (listenerHolder[collection] === undefined) {
//listenerBaseComponent would be function to establish listener and return unsubscribe function
const unSub = listenerBaseComponent();
if (unSub)
newListeners[collection] = unSub;
}
if (Object.entries(newListeners).length !== Object.entries(listenerHolder).length) {
setListenerHolder(newListeners);
}
}, []);
-Unmounting all listeners (in component that holds inside of it all screens and is unmounted only when whole site is closed):
const { listenerHolder, setListenerHolder } = DataContext();
const handleTabClosing = () => {
Object.entries(listenerHolder).forEach(item => {
const [key, value] = item;
if (typeof value === 'function')
value();
});
setListenerHolder({});
}
useEffect(() => {
window.addEventListener('beforeunload', handleTabClosing)
return () => {
window.removeEventListener('beforeunload', handleTabClosing)
}
})
In both cases the home screen is showing most recent from "events" collection, but in my understanding...
-The first approach creates listener 3 times to collection "events" and so 3 x 100 read operations are done.
-The second approach creates listener 1 time to collection "events" and so 1 x 100 read operations are done.
If storing the unsubscribe function to context is possible and all listener unsubscribtions are handled at once in site unmount or in logout, doesn't this make using it this way super easy, more maintainable and more cost efficient? If I would need to see data from "events" collection in any other screen I would not have to do get call / create a new listener, because I would always have latest data from "events" when site is used. Just check if there is (in this case) collection name as key in global state "listenerHolder", and if there is, there would be most up to date data always for events.

Since there wasn't information from others about this use case I made some testing myself jumping from this "homescreen" to another screen back and forth multiple times. "Homescreen" has about 150 items and second screen 65.
The results are from Firebase, Cloud Firestore usage tab:
This is the result of reads from that jumping: 654(1.52pm-1.53pm) + 597(1.53pm-1.54pm) = 1251 reads
Now I tested the same jumping back and forth when using global context listeners: 61(1.59pm-2.00pm) + 165(2.00pm-2.01pm) = 226 reads
So using listeners in global context will result significantly less reads. This is depending how many times new listeners (in normal use case) would need to be recreated.
I have not yet tested well enough memory usage comparing these two cases. But if I test it, I will add the results here for others to benefit.

Related

how to sync firebase and local data model in flutter

Pretty positive I'm just totally overthinking this or approaching it from an illogical angle.
I'm separating my logic from my ui where button presses call a method located in the userModel which has a change notifier (getting passed into MyApp with a provider). I'm trying to implement firebase but have never called firebase directly from the ui (always just had the requests in the ui code, never used a model).
import 'package:cloud_firestore/cloud_firestore.dart';
import 'package:flutter/material.dart';
import 'package:tuple/tuple.dart';
//TODO: firebase has been installed now I need to figure out how to implement it
// TODO: add firebase read and write
class UserModel with ChangeNotifier {
//index = session number
List session = [];
//create an instance of firebase (this might need to go higher in the tree)
var db = FirebaseFirestore.instance;
//TODO:
//convert incoming firebase to JSON
//convert outgoing json to firebase
//track session
// takes in current session data and adds the new chunk
// already recorded (new item in list but time, breaks etc. are adding from the last one)
// IF ADDING TO EXISTING, ALL PARAMETRS MUST BE SET
addSessionChunk(
{required String intention,
int workTime = 0,
String progress = "null",
int breakTime = 0}) {
session.add({
"intention": intention,
"workTime": workTime,
"progress": progress,
"breakTime": breakTime,
});
//firebase update?
}
//TODO: when returning to a previous intention, add to the numbers that were
//TODO: currently only works for 1 call per chunk (no going back to the same intention)
//get previous data from this intention for returning to a task (do
//these update functions updadate the LAST CHUNK in the session
updateChunkWorkTime({required int workTime}) {
//this later)
session.last["workTime"] = workTime;
}
//takes in inputed progress and updates the latest chunk with it
updateChunkProgress({required String progress}) {
session.last["progress"] = progress;
}
//takes inputed breaktime and updates the lastest chunk with it
updateChunkBreakTime({required int breakTime}) {
session.last["breakTime"] = breakTime;
}
//returns tuple of the total time spent working and breaking in the current session
calculateSessionTimeTotal() {
int totalWorkTime = 0;
int totalBreakTime = 0;
for (var chunk in session) {
totalWorkTime += chunk["workTime"] as int;
totalBreakTime += chunk["breakTime"] as int;
}
return Tuple2(totalWorkTime, totalBreakTime);
}
//firebase functions
pushDataUp() {
db.collection("sessions").doc().set({
"currentSession": session,
"total": calculateSessionTimeTotal().toString()
});
}
pullDataDown() {}
}
You can see at the bottom there I started to try and come up with a way to sync the local data state with firebase but am confused. Seems weird for the user to send their data up to firebase then back down into the model which is already holding that data?
Whats the best approach to local model and cloud database interaction? Any guidance in the right direction is greatly appreciated.
What seems weird to you, is actually a quite well defined patterns known as command query responsibility segregation, and is the basic pattern behind most modern UI frameworks. By separating the command (the writing of the data here) from the query (the reading of the data here) each remains simpler, and the total app becomes much easier to reason about.
With Firestore in Flutter, this usually translates into:
The user takes some action.
Your code writes to the database.
Your onSnapshot listener gets triggered with the updated data.
Your code updates the data model/state with the new data.
Which then renders the updated UI.
All of this happens pretty instantly, as Firebase actually handles it locally before even sending the data to the server and handles any exception that may occur during the synchronization with the server.

react-query: How to process a queue, one item at a time, and remove the original data after processing?

I'm using react-query 4 to get some data from my server via JSON:API and create some objects:
export type QueryReturnQueue = QueueObject[] | false;
const getQueryQueue = async (query: string): Promise<QueryReturnQueue> => {
const data = await fetchAuth(query);
const returnData = [] as QueueObject[];
if (data) {
data.map((queueItem) => returnData.push(new QueueObject(queueItem)));
return returnData;
}
return false;
};
function useMyQueue(
queueType: QueueType,
): UseQueryResult<QueryReturnQueue, Error> {
const queryKey = ['getQueue', queueType];
return useQuery<QueryReturnQueue, Error>(
queryKey,
async () => {
const query = getUrl(queueType);
return getQueryQueue(query);
},
);
}
Then I have a component that displays the objects one at a time and the user is asked to make a choice (for example, "swipe left" or "swipe right"). This queue only goes in one direction-- the user sees a queueObject, processes the object, and then goes to the next one. The user cannot go back to a previous object, and the user cannot skip ahead.
So far, I've been using useContext() to track the index in the queue as state. However, I've been running into several bugs with this when the queue gets refreshed, which happens a lot, so I thought it would be easier to directly manipulate the data returned by useQuery().
How can I remove items as they are processed from the locally cached query results?
My current flow:
Fetch the queue data and generation objects with useQuery().
Display the queue objects one at a time using useContext().
Mutate the displayed object with useMutation() to modify useContext() and then show the next object in the cached data from useQuery().
My desired flow:
Fetch the queue data and generation objects with useQuery().
Mutate the displayed object with useMutation(), somehow removing the mutated item from the cached data from useQuery() (like what shift() does for arrays).
Sources I consulted
Best practices for editing data after useQuery call (couldn't find an answer relevant to my case)
Optimistic updates (don't know how to apply it to my case)
My desired flow:
Fetch the queue data and generation objects with useQuery().
Mutate the displayed object with useMutation(), somehow removing the mutated item from the cached data from useQuery() (like what shift() does for arrays).
This is the correct way to think about the data flow. But mutations shouldn't be updating the cache with data, they should be invalidating existing cache data.
You have defined your query correctly. Now you simply have to instruct your mutation function (which should be making an API call that updates the records queue) to invalidate all existing queries for the data in the onSuccess handler.
e.g.
function useMyMutation(recordId, queueType) {
const queryClient = useQueryClient();
return useMutation({
mutationFn: ({id, swipeDirection}) =>
asyncAPICall(`/swipes/${id}`, { swipeDirection }),
onSuccess: () => queryClient.invalidateQueries(['getQueue', queueType]);
});
}
As suggested by #Jakub Kotrs:
shift the first item from the list + only ever display the first
I was able to implement this in my useMutation() hook:
onMutate: async (queueObjectRemoved) => {
const queryKey = ['getQueue', queueType];
// Cancel any outgoing refetches
// (so they don't overwrite our optimistic update).
await queryClient.cancelQueries({
queryKey,
});
if (data?.[0]?.id === queueObjectRemvoed.data.id) {
// Optimistically update the data by removing the first item.
data.shift();
queryClient.setQueryData(queryKey, () => data);
} else {
throw new Error('Unable to set queue!');
}
},
onError: () => {
const queryKey = ['getQueue', queueType];
setShowErrorToast(true);
queryClient.invalidateQueries(
queryKey,
);
},
This way users can process all the items in the current queue before needing to refetch.

Is a single Firestore write operation guaranteed to be atomic?

I have a Chat document that represents a chat between two users. It starts out empty, and eventually looks like this:
// chats/CHAT_ID
{
users: {
USER_ID1: true,
USER_ID2: true
},
lastAddedUser: USER_ID2
}
Each user is connected to a different Cloud Run container via websockets.
I would like to send a welcome message to both users once the second user connected. This message must be sent exactly once.
When a user sends a "connected" message to its websocket, the container performs something like the following:
// Return boolean reflecting whether the current container should emit the welcome message to both users
async addUserToChat(userId) {
// Write operation
await this.chatDocRef.set({ activeUsers: { [userId]: true }, lastAddedUser: userId, { merge: true })
// Read operation
const chatSnap = await this.chatDocRef.get();
const chatData = chatSnap.data();
return chatData.users.length === 2 && chatData.lastAddedUser === userId;
}
And there is a working mechanism that allows container A to send a message to a user connected to container B.
The issue is that sometimes, each container ends up concluding that it is the one that should send the welcome message to both users.
I am unclear as to why that would happen given Firestore's "immediately consistency model" (per this). The only explanation I can think of that allows racing condition is that write operations involving multiple fields are not guaranteed to be atomic. So this:
await this.chatDocRef.set({ activeUsers: { [userId]: true }, lastAddedUser: userId, { merge: true })
actually performs two separate updates for activeUsers and lastAddedUser, opening the possibility for a scenario where after partial update of activeUsers by container A, container B completes the write and read operations before container A overwrites lastAddedUser.
But this sounds wrong.
Can anyone shed light on why racing conditions might occur?
I no longer have racing conditions if I base the logic on the server timestamps instead of the lastAddedUser field.
The document is now simpler:
// chats/CHAT_ID
{
users: {
USER_ID1: true,
USER_ID2: true
}
}
And the function looks like this:
// Return boolean reflecting whether the current container should emit the welcome message to both users
async addUserToChat(userId) {
// Write operation
const writeResult = await this.chatDocRef.set({ activeUsers: { [userId]: true }, { merge: true })
// Read operation
const chatSnap = await this.chatDocRef.get();
const chatData = chatSnap.data();
return chatData.users.length === 2 && writeResult.writeTime.isEqual(chatSnap.updateTime);
}
In other words, the condition for sending the welcome message now becomes: the executing container is the container responsible for the update that resulted in having two users.
While the problem is solved, I am still unclear as to why relying on document data (instead of server metadata) opens up the possibility for racing conditions to occur. If anyone knows the explanation behind this phenomenon, please add an answer and I'll accept it as the solution to this question.

Mongoose how to listen for collection changes

I need to build a mongo updater process to dowload mongodb data to local IoT devices (configuration data, etc.)
My goal is to watch for some mongo collections in a fixed interval (1 minute, for example). If I have changed a collection (deletion, insertion or update) I will download the full collection to my device. The collections will have no more than a few hundred simple records, so it´s gonna not be a lot of data to download.
Is there any mechanism to find out a collection has changed since last pool ? What mongo features should be used in that case ?
To listen for changes to your MongoDB collection, set up a Mongoose Model.watch.
const PersonModel = require('./models/person')
const personEventEmitter = PersonModel.watch()
personEventEmitter.on('change', change => console.log(JSON.stringify(change)))
const person = new PersonModel({name: 'Thabo'})
person.save()
// Triggers console log on change stream
// {_id: '...', operationType: 'insert', ...}
Note: This functionality is only available on a MongoDB Replicaset
See Mongoose Model Docs for more:
If you want to listen for changes to your DB, use Connection.watch.
See Mongoose Connection Docs for more
These functions listen for Change Events from MongoDB Change Streams as of v3.6
I think best solution would be using post update middleware.
You can read more about that here
http://mongoosejs.com/docs/middleware.html
I have the same demand on an embedded that works quite autonomously, and it is always necessary to auto adjust your operating parameters without having to reboot your system.
For this I created a configuration manager class, and in its constructor I coded a "parameter monitor", which checks the database only the parameters that are flagged for it, of course if a new configuration needs to be monitored, I inform the config -manager in another part of the code to reload such an update.
As you can see the process is very simple, and of course can be improved to avoid overloading the config-manager with many updates and also prevent them from overlapping with a very small interval.
Since there are many settings to be read, I open a cursor for a query as soon as the database is connected and opened. As data streaming sends me new data, I create a proxy for it so that it can be manipulated according to the type and internal details of Config-manager. I then check if the property needs to be monitored, if so, I call an inner-function called watch that I created to handle this, and it queries the subproject of the same name to see what default time it takes to check in the database by updates, and thus registers a timeout for that task, and each check recreates the timeout with the updated time or interrupts the update if watch no longer exists.
this.connection.once('open', () => {
let cursor = Config.find({}).cursor();
cursor.on('data', (doc) => {
this.config[doc.parametro] = criarProxy(doc.parametro, doc.valor);
if (doc.watch) {
console.log(sprintf("Preparando para Monitorar %s", doc.parametro));
function watch(configManager, doc) {
console.log("Monitorando parametro: %s", doc.parametro);
if (doc.watch) setTimeout(() => {
Config.findOne({
parametro: doc.parametro
}).then((doc) => {
console.dir(doc);
if (doc) {
if (doc.valor != configManager.config[doc.parametro]) {
console.log("Parametro monitorado: %(parametro)s, foi alterado!", doc);
configManager.config[doc.parametro] = criarProxy(doc.parametro, doc.valor);
} else
console.log("Parametro monitorado %{parametro}s, não foi alterado", doc);
watch(configManager, doc);
} else
console.log("Verifique o parametro: %s")
})
},
doc.watch)
}
watch(this, doc);
}
});
cursor.on('close', () => {
if (process.env.DEBUG_DETAIL > 2) console.log("ConfigManager closed cursor data");
resolv();
});
cursor.on('end', () => {
if (process.env.DEBUG_DETAIL > 2) console.log("ConfigManager end data");
});
As you can see the code can improve a lot, if you want to give suggestions for improvements according to your environment or generics please use the gist: https://gist.github.com/carlosdelfino/929d7918e3d3a6172fdd47a59d25b150

RXJS : Idiomatic way to create an observable stream from a paged interface

I have paged interface. Given a starting point a request will produce a list of results and a continuation indicator.
I've created an observable that is built by constructing and flat mapping an observable that reads the page. The result of this observable contains both the data for the page and a value to continue with. I pluck the data and flat map it to the subscriber. Producing a stream of values.
To handle the paging I've created a subject for the next page values. It's seeded with an initial value then each time I receive a response with a valid next page I push to the pages subject and trigger another read until such time as there is no more to read.
Is there a more idiomatic way of doing this?
function records(start = 'LATEST', limit = 1000) {
let pages = new rx.Subject();
this.connect(start)
.subscribe(page => pages.onNext(page));
let records = pages
.flatMap(page => {
return this.read(page, limit)
.doOnNext(result => {
let next = result.next;
if (next === undefined) {
pages.onCompleted();
} else {
pages.onNext(next);
}
});
})
.pluck('data')
.flatMap(data => data);
return records;
}
That's a reasonable way to do it. It has a couple of potential flaws in it (that may or may not impact you depending upon your use case):
You provide no way to observe any errors that occur in this.connect(start)
Your observable is effectively hot. If the caller does not immediately subscribe to the observable (perhaps they store it and subscribe later), then they'll miss the completion of this.connect(start) and the observable will appear to never produce anything.
You provide no way to unsubscribe from the initial connect call if the caller changes its mind and unsubscribes early. Not a real big deal, but usually when one constructs an observable, one should try to chain the disposables together so it call cleans up properly if the caller unsubscribes.
Here's a modified version:
It passes errors from this.connect to the observer.
It uses Observable.create to create a cold observable that only starts is business when the caller actually subscribes so there is no chance of missing the initial page value and stalling the stream.
It combines the this.connect subscription disposable with the overall subscription disposable
Code:
function records(start = 'LATEST', limit = 1000) {
return Rx.Observable.create(observer => {
let pages = new Rx.Subject();
let connectSub = new Rx.SingleAssignmentDisposable();
let resultsSub = new Rx.SingleAssignmentDisposable();
let sub = new Rx.CompositeDisposable(connectSub, resultsSub);
// Make sure we subscribe to pages before we issue this.connect()
// just in case this.connect() finishes synchronously (possible if it caches values or something?)
let results = pages
.flatMap(page => this.read(page, limit))
.doOnNext(r => this.next !== undefined ? pages.onNext(this.next) : pages.onCompleted())
.flatMap(r => r.data);
resultsSub.setDisposable(results.subscribe(observer));
// now query the first page
connectSub.setDisposable(this.connect(start)
.subscribe(p => pages.onNext(p), e => observer.onError(e)));
return sub;
});
}
Note: I've not used the ES6 syntax before, so hopefully I didn't mess anything up here.