Client-side, I'm using a listener to detect if the "notifications" collection of the user changes. The App calls a Cloud Function that retrieves the last three unread notifications and the total number of unread notifications.
In my App, I have this:
Listener
firestore.collection("users")
.doc(uid)
.collection("notifications")
.snapshots().listen((QuerySnapshot querySnapshot) {
NotificationsPreviewModel notificationsPreview =
await _cloudFunctionsService.getNotificationsPreview(doctor.id)
})
Cloud Function
exports.getNotificationsPreview = functions.https.onCall(async (data, context) => {
const userId = data.userId;
let notifications = [];
const notificationsDocuments = await db
.collection("users")
.doc(userId)
.collection("notifications")
.orderBy("dateTime", "desc")
.get();
notifications = notificationsDocuments.docs.map((rawNotification) =>
rawNotification.data()).filter((element) => element.unread == true);
const notificationsNumber = notifications.length;
notifications = notifications.slice(0, 3);
return { "notifications": notifications, "notificationsNumber": notificationsNumber };
});
The Cloud Function gets called only when a change is detected, so it shouldn't return old data.
The error appears only the first time the Cloud Function is called from the App's start, but not always. The following calls don't generate the error.
How can I solve this? For now, I've added a delay of 500ms, and it works perfectly, but it's not a real solution.
Based on your description, it sounds like you see some form of latency while collecting the data from Firestore. Retrieving data from the Cloud takes time, and a delay of 500ms is not excessive.
I am not familiar with Flutter enough to comment on your code. However, according to the documentation for Java:
By default, get() attempts to provide up-to-date data when possible by waiting for data from the server, but it may return cached data or fail if you are offline and the server cannot be reached. This behavior can be altered via the Source parameter.
Source:
By providing a Source value, these methods can be configured to fetch results only from the server, only from the local cache, or attempt to fetch results from the server and fall back to the cache (which is the default).
If you are online, get() checks the server for the latest data, which can take between 300ms and 1500ms depending on several factors. For example, where is your Firestore instance located in comparison to your Cloud Function and client? Try adjusting the delay and see if you can identify the timing.
There are also some soft limits you should be aware of as this might also impact your timings for how quickly you can retrieve the data. There is a maximum sustained write rate to a document of 1 per second. Sustaining a write rate above once per second increases latency and causes contention errors.
As for the documentation:
When you set a listener, Cloud Firestore sends your listener an initial snapshot of the data, and then another snapshot each time the document changes.
It seems that you are initially receiving the snapshot of the data, and then the following updates, as expected.
You can check some possible solutions to this in this post.
Related
I have the following definition of useQuery that i use in a couple of React components:
useQuery("myStuff", getMyStuffQuery().queryFn);
Where getMyStuffQuery looks like this:
export const getMyStuffQuery = () => {
return {
queryFn: () => makeSomeApiCall(),
}}
I would expect that although all of those components render, makeSomeApiCall() would only make an API call once, and the rest of the time will use the cache resulted in from this first call.
However, it seems like it keeps calling makeSomeApiCall() again and again, whenever any of said components renders.
Why is React Query not using the cache? Am i doing something wrong?
React Query will cache the data of the query by default, but that does not affect whether or not it thinks that data is stale. If it thinks data is stale, it will call the query function (hit the API) every time useQuery() is called. This means it will read the data from the cache if it has it, but since it thinks that data is stale, will still hit the API in the background to fetch any updated data.
Fortunately, you have complete control over whether or not React Query considers data to be stale. You can set a staleTime config option to control how long specific data should be considered fresh. You can even set it to Infinity to say that as long as your app is open, it should only ever call the query function (hit the API) one time. By default this value is 0, which is why you are seeing the behavior you are - React Query will refetch the data in the background every time useQuery is called because it immediately thinks that data is stale (even though it's still cached).
In your example, if you truly ever only wanted an API to be called one time, you could simply set the staleTime option to Infinity.
useQuery("myStuff", getMyStuffQuery().queryFn, { staleTime: Infinity });
This option, along with all others, can be read about in the docs here https://react-query.tanstack.com/reference/useQuery
React Query has a slightly different model of request caching.
A request can have its results cached, and those results can go stale.
Cached results are returned immediately, but if stale they are re-fetched in the background and the cache is updated.
The default configuration caches results for 5 minutes and makes them stale immediately.
See: https://tanstack.com/query/v4/docs/guides/caching
The cacheTime and staleTime can be set as part of the useQuery options object as shown here for a 5 minute cache time and 1 minute stale time.
useQuery({ queryKey: ['todos'], queryFn: fetchTodos, staleTime: 1 * 60 * 1000, cacheTime: 5 * 60 * 1000 });
The refetching cached results strategy can be changed with options like refetchOnWindowFocus
See: https://tanstack.com/query/v4/docs/guides/important-defaults
You can stop this refetching of stale values on the window focus event like this:
const client = new QueryClient({
defaultOptions: {
queries: {
refetchOnWindowFocus: false,
},
},
});
To reduce the number of reads it is a general technique to maintain timestamp of last edits in documents and comparing timestamp to load only modified documents.
Here is an example from firebase docs:
db.collection('groups')
.where('participants', 'array-contains', 'user123')
.where('lastUpdated', '>', lastFetchTimestamp)
.orderBy('lastUpdated', 'desc')
.limit(25)
They claim this would reduce the reads.
I tried implementing the use-case, I have a document as shown below:
I have sections in my app where I use scorecards to list top scorers, My query is as follows
private void loadFriendScores(UserScorecard scorecard) {
Query friendScoreQuery=scorecardRef.whereIn("uid", scorecard.getFriendsList())
.whereGreaterThan("lastActive", scorecard.getLastActive()).limit(5);
FirestoreRecyclerOptions<UserScorecard> friends = new FirestoreRecyclerOptions
.Builder<UserScorecard>()
.setQuery(friendScoreQuery, UserScorecard.class)
.setLifecycleOwner(getViewLifecycleOwner())
.build();
TopScoresAdapter friendsAdapter = new TopScoresAdapter(friends, getContext(), this);
binding.topScorersFriendsRcv.setAdapter(friendsAdapter);
binding.topScorersFriendsRcv.setLayoutManager(new LinearLayoutManager(getContext()));
}
I assumed the query to load all modified changes along with others (from cache):
The screen on android is as follows:
While I expected it to load all of my friendlist (as I understood from docs).
I suppose they did not mention that we need to fetch the cached list, there is a way to do a cached request in firestore.
but I'm not sure if this is reliable perhaps the cache will be cleaned and the last request would be empty ,
then, you should save the last response using the localstorage library
#react-native-async-storage/async-storage
I'm struggling myself with the costs issue. The reads are way higher then 50 reads and I'm not sure how to count them properly. so I upvoted the issue
I have been using onSnapshot successfully to alert my code to changes in underlying data, as in
// Set up to listen for changes to the "figures" collection, that is,
// someone has created a new figure that we will want to list on the screen.
setFiguresListener: function () {
// `figuresCR` is a collection reference defined elsewhere
return this.figuresCR.onSnapshot((iFigs) => {
iFigs.forEach((fSnap) => {
const aFigure = figureConverter.fromFirestore(fSnap, null);
const dbid = aFigure.guts.dbid; // ID of the "figure" in the database
nos2.theFigures[dbid] = aFigure; // update the local copy of the data
});
nos2.ui.update();
console.log(` Listener gets ${iFigs.size} figures`);
});
But I now read about on in the docs. It explains:
[The on() function] Listens for data changes at a particular location.
This is the primary way to read data from a Database. Your callback
will be triggered for the initial data and again whenever the data
changes. Use off( )to stop receiving updates. See Retrieve Data on
the Web for more details.
The syntax is a bit different, and on() seems to do much the same as onSnapshot().
So what is the real difference? Should we be using on() instead of onSnapshot()?
on() is an operation for reading from Firebase Realtime Database. That's a completely different database with different APIs than Firestore. They have essentially no overlap. There is no on() operation with Firestore.
If you're working with Firestore, ignore all the documentation about Realtime Database, and stick to using onSnapshot() for getting realtime updates.
Other tyros who fall into this tar pit: in the API doc pages, you might think that since firestore is a database under firebase, you could look for help under firebase.database. But no: look only in the next section, firebase.firestore.
What am I trying to accomblish?
I am currently facing a bunch of problems implementing a real time updated infinite scrolling list with the firestore backend.
In my application I want to display comments (like in e.g. YouTube or other social media sites) to the user. Since the number of comments in a collection might be quite big, I see an option to paginate the collection, while receiving real time updates based on snapshots. So I initially load x comments with the option to load up to x more items whenever the user presses a button. In the image below x = 3.
The standard solution
Based on other SO questions I figured out that one is supposed to use the .limit() and the .startAfter() methods to implement such behaviour.
So the first page is loaded as:
query = this
.collection
.orderBy('date', descending: true)
.limit(pageSize);
query.snapshots().map((QuerySnapshot snap) {
lastVisible = snap.documents.last;
// convert the DocumentSnapshot into model object
});
All additional pages are loaded with the following code:
query = this.collection
.orderBy('date', descending: true)
.startAfterDocument(lastVisible)
.limit(pageSize);
Furthermore, I'd like to add that this code is located in a repository class which is used with the BLoC pattern similar to the code shown in Felix Angelov's Flutter Todos Tutorial.
While Felix uses a simple flutter list to show the items, I have a list of pages showing comments based on the data provided by their BLoCs. Note that each BLoC accesses a shared repository (parts of the repository code is shown below).
The Problem with the standard solution
With the code shown above I see multiple problems:
If a comment is inserted in the middle of the ordered collection (how is not of importance), the added comment is shown because of the Stream provided by the snapshot. However, another comment that already existed is not longer shown because of the .limit() operator in the query. One could increase the limit by one but I'm not sure how to edit a snapshot query. In the case that editing a snapshot query is not possible, one could create a new (and bigger) query, but that would cost additional reads.
Similar to 1., if a comment in the middle is deleted, the snapshot will return a list which does not longer contain the deleted comment, however another comment (which is already covered by a different page) appears. E.g., in the scenario shown in the image above 5 comments are loaded. Assuming that comment 3 is deleted, comment 2 will show twice.
Improving the standard solution
Based on these two problems discussed above, I decided that the solution is not sufficient and I implemented a solution which first loads x items by obtaining two "interval" documents. Then a query which fetches the required items in an interval using .startAtDocument() and .endAtDocument() is created, which eliminates the .limit() operator.
DocumentSnapshot pageStartDocument;
DocumentSnapshot pageEndDocument;
Future<Stream<List<Comment>>> comments() async {
// This fetches the first and next Document as initialization
// (maybe should be implemented in constructor)
if (pageStartDocument == null) {
Query query = collection
.orderBy('date', descending: true)
.limit(pageSize);
QuerySnapshot snap = await query.getDocuments();
pageStartDocument = snap.documents.first;
pageEndDocument = snap.documents.last;
} else {
Query query = collection
.orderBy('date', descending: true)
.startAfterDocument(pageEndDocument)
.limit(pageSize);
QuerySnapshot snap = await query.getDocuments();
pageStartDocument = snap.documents.first;
pageEndDocument = snap.documents.last;
}
// This fetches a subcollection of elements from the collection
// with the tradeof of double the reads
Query query = this
.collection
.orderBy('date', descending: true)
.startAtDocument(pageStartDocument)
.endAtDocument(pageEndDocument);
return query.snapshots().asyncMap((QuerySnapshot snap) async {
// convert the QuerySnapshot into model objects
});
As commented in the code, this solution has the following drawback:
Since a query is required to obtain the pageStartDocument and pageEndDocument, the number of reads is doubled, because all the data is read again when the second query is created. The performance impact might be neglectable because I believe the data is cashed, however having 2x database read cost can be significant.
Question:
Since I am not only implementing pagination but also real time updates (with collection insertions), the .limit() operator seems to be not working in my case.
How does one implement a pagination with real time updates (without double reads)?
Side Notes:
I watched how Todd Kerpelman devoures a massive gummy bear while explaining pagination, but in the video it seems to be not so trivial (and a point was made that a tradeoff might be necessary).
If further code from my side is required please say so in the comments.
For the scenario of comments it does not really makes sense that an item is inserted into the middle of the (sorted) collection. However I would like to understand how it should be implemented if the scenario requires such a feature.
this may come as a very late answer. The OP probably won't need help anymore, however for anyone who should stumble on this I wrote a tutorial with a solution that partly solve this:
the Bloc keep a list of stream subscription to keep trace of realtime updates to the list.
however concerning the insertion problem, since when you will have paginated streams based on a document cursor, upon insertion or deletion you necessarily need to reset your pagination stream subscriptions unless it is the last page.
Hence my solution around it was to update the list when modifications occur but reset it when insertions or deletions occur.
Here is the link to the tutorial :
https://link.medium.com/2SPf2Qsbsgb
first,I added an animation, after animation completed I called below code line but it's consuming too much time.
getData()async{
DateTime now = DateTime.now();
await Firestore.instance.collection('Table_Name').getDocuments().then((QuerySnapshot snapshot){
print("length ${snapshot.documents.length}");
for(int i = 0 ; i< snapshot.documents.length; i++){
bool isToday = snapshot.documents[i].data['CreatedBy'].toString().split(" ")[0]==DateFormat("yyyy-MM-dd").format(now);
print("isTOday $isToday");
if(isToday && snapshot.documents[i].data['GainStatus']== "0"){
setState(() {
giftDocumentID = snapshot.documents[i].documentID;
});
break;
}
}
winOrLose(now);
print("giftDocumentID: $giftDocumentID");
});
}
This Above codes taking approx 10 min to retrieve data from firestore, Why is it too lazy?
It got solved by testing it on another device and running smoothly without any much time consumption.
Actually my first device's memory has filled, so that's why, taking too much time.
But I'm still confuse, what the relation between device's memory space and firestorm database..!
The relation between firestore data and device's memory is cache. When data is loaded for the first time, it's from firebase's servers (and thus costs you read requests), then firestore saves the data in the cache memory for future use. So slow retrieval of the data must be a Device OS problem (or a power issue).
You can remove caching feature from firebase data but I won't recommend that.
Also as suggested earlier you should not load all data at once especially if the data is going to be fetched a lot or data gets changed very often. The solution for that is pagination.
This Video explains Flutter Pagination nicely:
https://www.youtube.com/watch?v=coR4Y-DkrLc