I have Flutter mobile application in the front end and Firebase in the backend. I am currently running Firebase emulator. I am using Firestore for storing data.
I have a QUERY that is supposed to fetch a document and a WRITE that is supposed to update the document. When I run both the QUERY and WRITE in the following order I see a bug. Is there a solution.
Run the QUERY to read the document from Firestore.
Run the WRITE to update the document in Firestore. The WRITE fails because of a security rule which is what I expect. The document in Firestore is not updated also as expected.
Run the QUERY to read the document from Firestore. This time, the value read is the value in step 2 even though the write has failed. When printing the Metadata over here, I find that isFromCache is false but hasPendingWrites is true.
It seems to me that the write in step 2 updated the cache and the read in step 3 read the doc from the cache giving me wrong results.
Any idea if this is a bug in Firestore emulator or if I should be doing something else?
This is the WRITE code:
await _firestore.doc(PathService.boosts(uid)).update({
'boosting': true,
'boosts': FieldValue.increment(-1),
'endedAt': Timestamp.fromDate(
DateTime.now().add(const Duration(hours: 1)),
),
'ttl': Timestamp.fromDate(
DateTime.now().add(const Duration(days: 7 * 365)),
),
});
Here is the code for the READ. I tried to use Source.server hoping to force the READ from server instead from cache but did not work:
_firestore
.doc(PathService.boosts(uid))
.withConverter<MyProfileBoosts>(
fromFirestore: (snapshot, _) {
return MyProfileBoosts.fromJson(snapshot.data()!);
},
toFirestore: (myProfileBoosts, _) => myProfileBoosts.toJson(),
)
.get(const GetOptions(source: Source.server));
As for the Firestore security rules, all you need is simple rule that makes the write fails.
Related
I've been working on a chat feature in my app and wanted to handle displaying messages with a paginated stream from firestore.
Currently I'm using this code which is going through a provider. When the user scrolls, the provider adds pageNumber to the limit and then calls getMessages again. My main concern is that when I do this its reading all the docs again. For example on load the limit is 10, so it reads the first 10 docs, then after scrolling the limit is changed to 20, does it read the first 20, or does it just read the next 10 since it already has the first? I have looked though the firestore documentation and haven't been able to find a definitive answer to this. If anyone knows please let me know
If its the case it does re-read all docs, I will need to implement a startAfter, which isn't a big deal. I am just very curious about this and want to know definitively.
#override
Stream<List<Message>> getMessages({
required int limit,
required String conversationID,
}) {
return firestore
.collection(chatsCollectionPath)
.doc(conversationID)
.collection('messages')
.orderBy('date', descending: true)
.limit(limit)
.snapshots()
.map((event) => event.docs.map((e) => Message.fromDoc(e)).toList());
}
I am running a Flutter mobile app that queries data points from Firestore. Until very recently, I have been running the following query:
return firestore
.collection('organisations/$organisationId/alerts/$alertId/deviceTrails/$deviceTrailId/markers')
.where('deviceCreatedUtc', isGreaterThanOrEqualTo: timestamp)
.snapshots().handleError(handleFirestoreError);
I found, while running this query, that it would work well and provide a certain number of snapshots, but that it would stop generating snapshots without throwing any errors after a period of a few minutes. Changing the query to the following seemed to resolve this error (snapshots became more reliable):
return firestore
.collection('organisations/$organisationId/alerts/$alertId/deviceTrails/$deviceTrailId/markers')
.orderBy('deviceCreatedUtc')
.startAt([timestamp])
.snapshots().handleError(handleFirestoreError);
Other than the ordering (which is not strictly necessary in my case, since I am adding the points to my on-device database instead of using them directly), there does not appear to be much in the way of functional difference between these queries. But the former fails silently, while the latter is more reliable.
Is there any reason why this would happen? And is one of the queries intrinsically more efficient than the other?
As mentioned by Frank van Puffelen, two snippets should do exactly the same (outside of ordering as you said). You can fill a bug on the repo here :
For more information you can refer to the Documentation (listen a document with onsnapshot() method )and Documentation(working with list of data in flutter firebase.)
index.js
const query = db.collection('cities').where('state', '==', 'CA');
const observer = query.onSnapshot(querySnapshot => {
console.log(`Received query snapshot of size ${querySnapshot.size}`);
// ...
}, err => {
console.log(`Encountered error: ${err}`);
});
Client-side, I'm using a listener to detect if the "notifications" collection of the user changes. The App calls a Cloud Function that retrieves the last three unread notifications and the total number of unread notifications.
In my App, I have this:
Listener
firestore.collection("users")
.doc(uid)
.collection("notifications")
.snapshots().listen((QuerySnapshot querySnapshot) {
NotificationsPreviewModel notificationsPreview =
await _cloudFunctionsService.getNotificationsPreview(doctor.id)
})
Cloud Function
exports.getNotificationsPreview = functions.https.onCall(async (data, context) => {
const userId = data.userId;
let notifications = [];
const notificationsDocuments = await db
.collection("users")
.doc(userId)
.collection("notifications")
.orderBy("dateTime", "desc")
.get();
notifications = notificationsDocuments.docs.map((rawNotification) =>
rawNotification.data()).filter((element) => element.unread == true);
const notificationsNumber = notifications.length;
notifications = notifications.slice(0, 3);
return { "notifications": notifications, "notificationsNumber": notificationsNumber };
});
The Cloud Function gets called only when a change is detected, so it shouldn't return old data.
The error appears only the first time the Cloud Function is called from the App's start, but not always. The following calls don't generate the error.
How can I solve this? For now, I've added a delay of 500ms, and it works perfectly, but it's not a real solution.
Based on your description, it sounds like you see some form of latency while collecting the data from Firestore. Retrieving data from the Cloud takes time, and a delay of 500ms is not excessive.
I am not familiar with Flutter enough to comment on your code. However, according to the documentation for Java:
By default, get() attempts to provide up-to-date data when possible by waiting for data from the server, but it may return cached data or fail if you are offline and the server cannot be reached. This behavior can be altered via the Source parameter.
Source:
By providing a Source value, these methods can be configured to fetch results only from the server, only from the local cache, or attempt to fetch results from the server and fall back to the cache (which is the default).
If you are online, get() checks the server for the latest data, which can take between 300ms and 1500ms depending on several factors. For example, where is your Firestore instance located in comparison to your Cloud Function and client? Try adjusting the delay and see if you can identify the timing.
There are also some soft limits you should be aware of as this might also impact your timings for how quickly you can retrieve the data. There is a maximum sustained write rate to a document of 1 per second. Sustaining a write rate above once per second increases latency and causes contention errors.
As for the documentation:
When you set a listener, Cloud Firestore sends your listener an initial snapshot of the data, and then another snapshot each time the document changes.
It seems that you are initially receiving the snapshot of the data, and then the following updates, as expected.
You can check some possible solutions to this in this post.
I have 100 documents
Document id
0001
0002
0003
....
....
0100
and if we load 5 documents with id 001,002,004,005,006
then firestore charge for 5 document reads and then we again load(call the read operation query) documents with id 004,005,006,007,008,001,002
then firestore will charge for 7 document reads
here on first time we already loaded document with ids 001,002,004,005,006 and in second time or refresh time we are loading documents already loaded and some new documents
Here we need to avoid multiple times reading document from server and read it from cash and need to avoid the firestore over document read charges How to do it?
Firestore have cash loading option but it will only load from cash and not from server here what we need is load exiting data from cash and load remaning data form server.Here now what firestore doing is it will load from server and if it is failed then it will read from cash that is ok but i need in revise order
Now what happening is if non internet all data load faster with out showing progress and if there is internet it will take few sec to load and it will shows loader When we do it without fireabase our app will shows loading only one time then first it will shows the data from sqlite then when ever the api call resoppnce reached we will update in ui, so users will not face any loader but with firestore user need to wait for a progress bar to finish
From a first glance it seems that you may use firebase firestore caching for this use case. This can be done easily for example in JS:
documentRef.get({ source: 'server' }) // or 'cache'
this will indeed reduce costs however it may read always from your local cache and never reach the server for new changes in your document. This might be what you want but it seems practical only if your documents (immutable) and never change. so you will be able to read new documents but if they change you might not see the changes. Please, read more about this here
A better suggestion is to change your app logic. So, rather than reading the documents this way:
001,002,004,005,006
004,005,006,007,008,001,002
it's better to read them in a paginated way like this:
001,002,003, 004,005,006
007,008,009,010,011,012
You can achieve that easily by using the concept of pagination in Firestore:
var first = db.collection("cities")
.orderBy("population")
.limit(25);
return first.get().then(function (documentSnapshots) {
// Get the last visible document
var lastVisible = documentSnapshots.docs[documentSnapshots.docs.length-1];
console.log("last", lastVisible);
// Construct a new query starting at this document,
// get the next 25 cities.
var next = db.collection("cities")
.orderBy("population")
.startAfter(lastVisible)
.limit(25);
});
check Firestore documentation for more details
What am I trying to accomblish?
I am currently facing a bunch of problems implementing a real time updated infinite scrolling list with the firestore backend.
In my application I want to display comments (like in e.g. YouTube or other social media sites) to the user. Since the number of comments in a collection might be quite big, I see an option to paginate the collection, while receiving real time updates based on snapshots. So I initially load x comments with the option to load up to x more items whenever the user presses a button. In the image below x = 3.
The standard solution
Based on other SO questions I figured out that one is supposed to use the .limit() and the .startAfter() methods to implement such behaviour.
So the first page is loaded as:
query = this
.collection
.orderBy('date', descending: true)
.limit(pageSize);
query.snapshots().map((QuerySnapshot snap) {
lastVisible = snap.documents.last;
// convert the DocumentSnapshot into model object
});
All additional pages are loaded with the following code:
query = this.collection
.orderBy('date', descending: true)
.startAfterDocument(lastVisible)
.limit(pageSize);
Furthermore, I'd like to add that this code is located in a repository class which is used with the BLoC pattern similar to the code shown in Felix Angelov's Flutter Todos Tutorial.
While Felix uses a simple flutter list to show the items, I have a list of pages showing comments based on the data provided by their BLoCs. Note that each BLoC accesses a shared repository (parts of the repository code is shown below).
The Problem with the standard solution
With the code shown above I see multiple problems:
If a comment is inserted in the middle of the ordered collection (how is not of importance), the added comment is shown because of the Stream provided by the snapshot. However, another comment that already existed is not longer shown because of the .limit() operator in the query. One could increase the limit by one but I'm not sure how to edit a snapshot query. In the case that editing a snapshot query is not possible, one could create a new (and bigger) query, but that would cost additional reads.
Similar to 1., if a comment in the middle is deleted, the snapshot will return a list which does not longer contain the deleted comment, however another comment (which is already covered by a different page) appears. E.g., in the scenario shown in the image above 5 comments are loaded. Assuming that comment 3 is deleted, comment 2 will show twice.
Improving the standard solution
Based on these two problems discussed above, I decided that the solution is not sufficient and I implemented a solution which first loads x items by obtaining two "interval" documents. Then a query which fetches the required items in an interval using .startAtDocument() and .endAtDocument() is created, which eliminates the .limit() operator.
DocumentSnapshot pageStartDocument;
DocumentSnapshot pageEndDocument;
Future<Stream<List<Comment>>> comments() async {
// This fetches the first and next Document as initialization
// (maybe should be implemented in constructor)
if (pageStartDocument == null) {
Query query = collection
.orderBy('date', descending: true)
.limit(pageSize);
QuerySnapshot snap = await query.getDocuments();
pageStartDocument = snap.documents.first;
pageEndDocument = snap.documents.last;
} else {
Query query = collection
.orderBy('date', descending: true)
.startAfterDocument(pageEndDocument)
.limit(pageSize);
QuerySnapshot snap = await query.getDocuments();
pageStartDocument = snap.documents.first;
pageEndDocument = snap.documents.last;
}
// This fetches a subcollection of elements from the collection
// with the tradeof of double the reads
Query query = this
.collection
.orderBy('date', descending: true)
.startAtDocument(pageStartDocument)
.endAtDocument(pageEndDocument);
return query.snapshots().asyncMap((QuerySnapshot snap) async {
// convert the QuerySnapshot into model objects
});
As commented in the code, this solution has the following drawback:
Since a query is required to obtain the pageStartDocument and pageEndDocument, the number of reads is doubled, because all the data is read again when the second query is created. The performance impact might be neglectable because I believe the data is cashed, however having 2x database read cost can be significant.
Question:
Since I am not only implementing pagination but also real time updates (with collection insertions), the .limit() operator seems to be not working in my case.
How does one implement a pagination with real time updates (without double reads)?
Side Notes:
I watched how Todd Kerpelman devoures a massive gummy bear while explaining pagination, but in the video it seems to be not so trivial (and a point was made that a tradeoff might be necessary).
If further code from my side is required please say so in the comments.
For the scenario of comments it does not really makes sense that an item is inserted into the middle of the (sorted) collection. However I would like to understand how it should be implemented if the scenario requires such a feature.
this may come as a very late answer. The OP probably won't need help anymore, however for anyone who should stumble on this I wrote a tutorial with a solution that partly solve this:
the Bloc keep a list of stream subscription to keep trace of realtime updates to the list.
however concerning the insertion problem, since when you will have paginated streams based on a document cursor, upon insertion or deletion you necessarily need to reset your pagination stream subscriptions unless it is the last page.
Hence my solution around it was to update the list when modifications occur but reset it when insertions or deletions occur.
Here is the link to the tutorial :
https://link.medium.com/2SPf2Qsbsgb