I've been working on a chat feature in my app and wanted to handle displaying messages with a paginated stream from firestore.
Currently I'm using this code which is going through a provider. When the user scrolls, the provider adds pageNumber to the limit and then calls getMessages again. My main concern is that when I do this its reading all the docs again. For example on load the limit is 10, so it reads the first 10 docs, then after scrolling the limit is changed to 20, does it read the first 20, or does it just read the next 10 since it already has the first? I have looked though the firestore documentation and haven't been able to find a definitive answer to this. If anyone knows please let me know
If its the case it does re-read all docs, I will need to implement a startAfter, which isn't a big deal. I am just very curious about this and want to know definitively.
#override
Stream<List<Message>> getMessages({
required int limit,
required String conversationID,
}) {
return firestore
.collection(chatsCollectionPath)
.doc(conversationID)
.collection('messages')
.orderBy('date', descending: true)
.limit(limit)
.snapshots()
.map((event) => event.docs.map((e) => Message.fromDoc(e)).toList());
}
Related
I have Flutter mobile application in the front end and Firebase in the backend. I am currently running Firebase emulator. I am using Firestore for storing data.
I have a QUERY that is supposed to fetch a document and a WRITE that is supposed to update the document. When I run both the QUERY and WRITE in the following order I see a bug. Is there a solution.
Run the QUERY to read the document from Firestore.
Run the WRITE to update the document in Firestore. The WRITE fails because of a security rule which is what I expect. The document in Firestore is not updated also as expected.
Run the QUERY to read the document from Firestore. This time, the value read is the value in step 2 even though the write has failed. When printing the Metadata over here, I find that isFromCache is false but hasPendingWrites is true.
It seems to me that the write in step 2 updated the cache and the read in step 3 read the doc from the cache giving me wrong results.
Any idea if this is a bug in Firestore emulator or if I should be doing something else?
This is the WRITE code:
await _firestore.doc(PathService.boosts(uid)).update({
'boosting': true,
'boosts': FieldValue.increment(-1),
'endedAt': Timestamp.fromDate(
DateTime.now().add(const Duration(hours: 1)),
),
'ttl': Timestamp.fromDate(
DateTime.now().add(const Duration(days: 7 * 365)),
),
});
Here is the code for the READ. I tried to use Source.server hoping to force the READ from server instead from cache but did not work:
_firestore
.doc(PathService.boosts(uid))
.withConverter<MyProfileBoosts>(
fromFirestore: (snapshot, _) {
return MyProfileBoosts.fromJson(snapshot.data()!);
},
toFirestore: (myProfileBoosts, _) => myProfileBoosts.toJson(),
)
.get(const GetOptions(source: Source.server));
As for the Firestore security rules, all you need is simple rule that makes the write fails.
I have a streambuilder where I'm loading products
StreamBuilder<QuerySnapshot>(
stream: _ref.collection("products").where("category", isEqualTo:
widget.selectedCategory.nameLabel).orderby("standing",descending:false).limit(10)
builder: (context, snapshot) {
if(snapshot.hasData){
if(snapshot.data!.docs.isNotEmpty) {
for (var changes in snapshot.data!.docChanges) {
//adding data to list<Products> products for pagination
}
}
return ProductsList.vertical(
scrollController: _scrollProducts,
products: products,
scroll: true,
);
}
The problem is I'm receiving more than 10 documents. If there is cache data, it will load that first then load the actual data ordered by 'standing'.
This problem exists mostly on the first load when cache data is not correct. How can we fix this?
What you're describing sounds like the expected behavior. The Firebase SDK for Firestore will:
immediately give you the documents from the local cache that meet the query conditions,
then reach out to the server to get latest content, with which it then updates its local cache.
finally fire an additional event on the stream with this latest content.
I don't think there's a way to bypass the first event on your stream in Flutter, which you can detect that snapshots come from the cache (and may not be up to date with the server) by checking their document.metadata.isFromCache property.
I am using Firestore stream in my flutter app. The data is user profiles so I need to filter it as needed. My initial data stream is as follows
Stream s = FirebaseFirestore.instance
.collection('users')
.snapshots(includeMetadataChanges: true);
and when I have to filter data, I do
setState((){
s = FirebaseFirestore.instance
.collection('users')
.where('name',isEqualTo:"MyName")
.snapshots(includeMetadataChanges: true);
});
This works fine but my concern is reads, I know that data is cached and is read from server only if something changes but I don't know of streams.
I tried monitoring the usage on firebase but due to the app being in production I couldn't measure anything properly.
And if filtering by changing stream does cause reads on firebase then what else can I do to filter data from cache.
One thing I thought off was storing everything locally but storing data of thousands of users locally is impractical.
A stream is a sequence of asynchronous events. It is like an
asynchronous Iterable—where, instead of getting the next event when
you ask for it, the stream tells you that there is an event when it is
ready.
From Dart (https://dart.dev/tutorials/language/streams)
A stream is a continues datastream that closes when the class is terminated or by canceling the stream object:
streamSub.cancel();
You can use a stream in a streambuilder. Every time a change is applied to any of the documents, it rebuilds the builder.
StreamBuilder(
stream: stream,
builder: (BuildContext content, AsyncSnapshot snapshot) {
}
)
If 1 document changes in query results, you will be billed 1 read. You are not billed for documents that are unchanged. The query snapshot object merely reuses the document data that was in memory from the prior set of results. By Doug Stevenson (https://stackoverflow.com/a/60062304/9142870)
Client-side, I'm using a listener to detect if the "notifications" collection of the user changes. The App calls a Cloud Function that retrieves the last three unread notifications and the total number of unread notifications.
In my App, I have this:
Listener
firestore.collection("users")
.doc(uid)
.collection("notifications")
.snapshots().listen((QuerySnapshot querySnapshot) {
NotificationsPreviewModel notificationsPreview =
await _cloudFunctionsService.getNotificationsPreview(doctor.id)
})
Cloud Function
exports.getNotificationsPreview = functions.https.onCall(async (data, context) => {
const userId = data.userId;
let notifications = [];
const notificationsDocuments = await db
.collection("users")
.doc(userId)
.collection("notifications")
.orderBy("dateTime", "desc")
.get();
notifications = notificationsDocuments.docs.map((rawNotification) =>
rawNotification.data()).filter((element) => element.unread == true);
const notificationsNumber = notifications.length;
notifications = notifications.slice(0, 3);
return { "notifications": notifications, "notificationsNumber": notificationsNumber };
});
The Cloud Function gets called only when a change is detected, so it shouldn't return old data.
The error appears only the first time the Cloud Function is called from the App's start, but not always. The following calls don't generate the error.
How can I solve this? For now, I've added a delay of 500ms, and it works perfectly, but it's not a real solution.
Based on your description, it sounds like you see some form of latency while collecting the data from Firestore. Retrieving data from the Cloud takes time, and a delay of 500ms is not excessive.
I am not familiar with Flutter enough to comment on your code. However, according to the documentation for Java:
By default, get() attempts to provide up-to-date data when possible by waiting for data from the server, but it may return cached data or fail if you are offline and the server cannot be reached. This behavior can be altered via the Source parameter.
Source:
By providing a Source value, these methods can be configured to fetch results only from the server, only from the local cache, or attempt to fetch results from the server and fall back to the cache (which is the default).
If you are online, get() checks the server for the latest data, which can take between 300ms and 1500ms depending on several factors. For example, where is your Firestore instance located in comparison to your Cloud Function and client? Try adjusting the delay and see if you can identify the timing.
There are also some soft limits you should be aware of as this might also impact your timings for how quickly you can retrieve the data. There is a maximum sustained write rate to a document of 1 per second. Sustaining a write rate above once per second increases latency and causes contention errors.
As for the documentation:
When you set a listener, Cloud Firestore sends your listener an initial snapshot of the data, and then another snapshot each time the document changes.
It seems that you are initially receiving the snapshot of the data, and then the following updates, as expected.
You can check some possible solutions to this in this post.
What am I trying to accomblish?
I am currently facing a bunch of problems implementing a real time updated infinite scrolling list with the firestore backend.
In my application I want to display comments (like in e.g. YouTube or other social media sites) to the user. Since the number of comments in a collection might be quite big, I see an option to paginate the collection, while receiving real time updates based on snapshots. So I initially load x comments with the option to load up to x more items whenever the user presses a button. In the image below x = 3.
The standard solution
Based on other SO questions I figured out that one is supposed to use the .limit() and the .startAfter() methods to implement such behaviour.
So the first page is loaded as:
query = this
.collection
.orderBy('date', descending: true)
.limit(pageSize);
query.snapshots().map((QuerySnapshot snap) {
lastVisible = snap.documents.last;
// convert the DocumentSnapshot into model object
});
All additional pages are loaded with the following code:
query = this.collection
.orderBy('date', descending: true)
.startAfterDocument(lastVisible)
.limit(pageSize);
Furthermore, I'd like to add that this code is located in a repository class which is used with the BLoC pattern similar to the code shown in Felix Angelov's Flutter Todos Tutorial.
While Felix uses a simple flutter list to show the items, I have a list of pages showing comments based on the data provided by their BLoCs. Note that each BLoC accesses a shared repository (parts of the repository code is shown below).
The Problem with the standard solution
With the code shown above I see multiple problems:
If a comment is inserted in the middle of the ordered collection (how is not of importance), the added comment is shown because of the Stream provided by the snapshot. However, another comment that already existed is not longer shown because of the .limit() operator in the query. One could increase the limit by one but I'm not sure how to edit a snapshot query. In the case that editing a snapshot query is not possible, one could create a new (and bigger) query, but that would cost additional reads.
Similar to 1., if a comment in the middle is deleted, the snapshot will return a list which does not longer contain the deleted comment, however another comment (which is already covered by a different page) appears. E.g., in the scenario shown in the image above 5 comments are loaded. Assuming that comment 3 is deleted, comment 2 will show twice.
Improving the standard solution
Based on these two problems discussed above, I decided that the solution is not sufficient and I implemented a solution which first loads x items by obtaining two "interval" documents. Then a query which fetches the required items in an interval using .startAtDocument() and .endAtDocument() is created, which eliminates the .limit() operator.
DocumentSnapshot pageStartDocument;
DocumentSnapshot pageEndDocument;
Future<Stream<List<Comment>>> comments() async {
// This fetches the first and next Document as initialization
// (maybe should be implemented in constructor)
if (pageStartDocument == null) {
Query query = collection
.orderBy('date', descending: true)
.limit(pageSize);
QuerySnapshot snap = await query.getDocuments();
pageStartDocument = snap.documents.first;
pageEndDocument = snap.documents.last;
} else {
Query query = collection
.orderBy('date', descending: true)
.startAfterDocument(pageEndDocument)
.limit(pageSize);
QuerySnapshot snap = await query.getDocuments();
pageStartDocument = snap.documents.first;
pageEndDocument = snap.documents.last;
}
// This fetches a subcollection of elements from the collection
// with the tradeof of double the reads
Query query = this
.collection
.orderBy('date', descending: true)
.startAtDocument(pageStartDocument)
.endAtDocument(pageEndDocument);
return query.snapshots().asyncMap((QuerySnapshot snap) async {
// convert the QuerySnapshot into model objects
});
As commented in the code, this solution has the following drawback:
Since a query is required to obtain the pageStartDocument and pageEndDocument, the number of reads is doubled, because all the data is read again when the second query is created. The performance impact might be neglectable because I believe the data is cashed, however having 2x database read cost can be significant.
Question:
Since I am not only implementing pagination but also real time updates (with collection insertions), the .limit() operator seems to be not working in my case.
How does one implement a pagination with real time updates (without double reads)?
Side Notes:
I watched how Todd Kerpelman devoures a massive gummy bear while explaining pagination, but in the video it seems to be not so trivial (and a point was made that a tradeoff might be necessary).
If further code from my side is required please say so in the comments.
For the scenario of comments it does not really makes sense that an item is inserted into the middle of the (sorted) collection. However I would like to understand how it should be implemented if the scenario requires such a feature.
this may come as a very late answer. The OP probably won't need help anymore, however for anyone who should stumble on this I wrote a tutorial with a solution that partly solve this:
the Bloc keep a list of stream subscription to keep trace of realtime updates to the list.
however concerning the insertion problem, since when you will have paginated streams based on a document cursor, upon insertion or deletion you necessarily need to reset your pagination stream subscriptions unless it is the last page.
Hence my solution around it was to update the list when modifications occur but reset it when insertions or deletions occur.
Here is the link to the tutorial :
https://link.medium.com/2SPf2Qsbsgb