Google Cloud Firestore console reading of all documents and charges - google-cloud-firestore

I am new to Firestore so I have a Profiles and Users collections. In the Cloud Firestore Console when I click on Database > Firestore > Data tab > Profiles or > Users the console is reading ALL the documents in each collection. These reads are counted in the Usage tab. So my question is if I have lets say 500K documents in the Profiles collection and clicked on Data I will be charged for reading 500K docs just to view the first 25 docs only. Just clicking on Data tab the console reads all the docs of the first Collection.
I tried using the filter but to use it you will have to click on the Collection and read all the docs first then you can edit the filter.
Is this the way it works or is it my misunderstanding?

I faced the same confusion a while ago and upon digging down to the issue I learnt that all the data which gets loaded in the 'Data' tab of Firestore page does count towards the overall Firestore usage.
However, I was concerned with the same question as yours thus I contacted Firebase support. They reverted back confirming the first instinct of mine(Document reads in 'Data' tab does count) BUT initially it reads only the first 300 documents of ANY selected collection, so even if your collection has over 1 million docs, it will still load only the first 300 documents.
They suggested a way around it until the Firebase team finds a legit solution
Bookmarking the Usage tab of the Firestore page. (So you basically 'Skip' the Data Tab and the useless 300 reads)
Adding a dummy collection in a certain way that ensures it is the first collection(alphabetically) which gets loaded by default on the Firestore page.

So my question is if I have lets say 500K documents in the Profiles collection and clicked on Data I will be charged for reading 500K docs
Absolutely not! You are only charged with read operations for the documents that are apart of the query. I don't know the exact number at which the first query is limited but the data is loaded in smaller chunks, with other words a pagination system is implemented. So once you scroll down other elements are loaded and so on.
If you intend to create an app that uses Firestore as backend, please also note that, offline persistence:
For Android and iOS, offline persistence is enabled by default.
For the web, offline persistence is disabled by default. To enable persistence, call the enablePersistence method.
For the web, offline persistence is an experimental feature that is supported only by the Chrome, Safari, and Firefox web browsers. Also, if a user opens multiple browser tabs that point to the same Cloud Firestore database, and offline persistence is enabled, Cloud Firestore will work correctly only in the first tab.
This means, that once you get a document(s) from the Firebase servers, you are not charched anymore since the results are coming from the local cache.

I just have a similar Question like and I find it redundant to post it as a Question so I'm posting it as asnswer,sorry for that,
Question:
this.db.collection(this.collectionName.usersCollection)
.doc(currentUserId).collection(this.collectionName.friendsCollection)
.where("friendID","==",id)
.get().then(snapshot =>{
if(snapshot.empty)
{
this.db.collection(this.collectionName.chatsCollection).add({
user1 : currentUserId,
user2 : id
}).then(docRef =>{
docId = docRef.id;
this.db.collection(this.collectionName.usersCollection)
.doc(currentUserId)
.collection(this.collectionName.friendsCollection).doc(docId)
.set({
friendID: id,
})
this.db.collection(this.collectionName.usersCollection).doc(id)
.collection(this.collectionName.friendsCollection).doc(docId)
.set({
friendID:currentUserId,
})
resolve(docRef.id);
})
}
else{
snapshot.forEach(document =>{
// console.log("friend id",document.data().friendID, " docid: ",document.data().docId);
resolve(document.id);
})
}
})
so here I'm writing and reading the docId ,so will it affect the counts? I cannot have a check properly because with this many other operations are happening.
Thanks .

Related

Firestore, pagination, loaded data are called again?

I saw the best reply after watching this video(https://www.youtube.com/watch?v=poqTHxtDXwU&feature=emb_title) and there was a comment like this.
"So there is a read charge even clients app cached the same document data" (currently 24 thumbs up)
And in the comment of another comment, Todd Kerpelman wrote this comment.
"Great question! The answer is that yes, you really will fetch those first 20 documents all the time. Note that this is different than when you have a realtime listener set up and a document changes in a query you're currently listening to - in that case, only the changed doc will be sent up. But if you're making a series of separate get calls that just happen to overlap, the database will send up the entire data set each time."
I am confused now. My question is, when you load the next list with startAfter, do you load the lists that have already been loaded again? Will you be paid?
when you load the next list with startAfter, do you load the lists that have already been loaded again?
No, for pagination, each query is completely different than the last. It does not re-fetch all the prior documents again, and you will not be charged for those prior documents. The query uses the document specified in startAfter() to determine exactly where the query should start reading results, and you will be charged for only the documents that are returned by the query.

My firestore account has A LOT of reads. How can I monitore from where?

My page is quite small it has around 300-1000 visits each day. But at some point I started to accumulate HUGE firestore read requests:
Till the 8th date it was somewhere around 50K each day. I am pushing new code all the time so I'm not sure what I did. Looking at the page I don't see anything out of ordinary. Is there some sort of log in google or firestore I could look at?
The Firebase documentation indicates that each time you create a project, it also creates a project in Google Cloud Platform, therefore you can track daily Cloud Firestore usage like writes, deletes, etc. This usage information is shown in the GCP's console in the App Engine Quotas page .You can see more details in the link. https://firebase.google.com/docs/firestore/monitor-usage#google-cloud-platform-console
There is currently no way to track the origin of reads. What you're looking at now is the best indicator you have available.
Bear in mind that the Firebase and Cloud consoles show updates to documents in real time, and each document update costs a read. If you leave the console open on a busy collection, it will rack up reads.

addSnapshotListener swift firestore behaviour

I have a document in Cloud firestore to which I listen for updates. It has 2 fields, it has a field description and a field for a picture. The picture is approximately 0.2 mb and description is a few words. I wanted to know what would happen if I made changes to the description in the document, I wanted to know if addSnapshotListener actually downloads a fresh new copy of the document or just the field that has been changed.
I indeed see, by looking at how much data is being downloaded in Xcode, a new fresh copy of the document is downloaded.
This is not efficient at all, since the picture field is rarely changed, only the description might change in my application.
Is there a way to optimize this?
Is there a way to optimize this?
Yes! Don't do that.
Firestore (and the realtime database) is not intended to store images or large datasets per field.
You should explore Storage and keep a reference (url) to the item stored in storage in your Firebase.
Cloud Storage is built for app developers who need to store and serve
user-generated content, such as photos or videos.
By leveraging storage if you need to update or change a field in Firestore, you're only working with a small amount of data instead of an entire image worth.
To answer the question; if you read a document from Firebase, it does read the Document and it's child data.
Here's a link to the Storage Docs which shows how to capture a reference to the item uploaded to storage.
https://firebase.google.com/docs/storage/ios/upload-files
If you want to automatically sync the images to all clients and have them available offline, just put them in a separate document.
// Store your small, frequently changing text here:
db.collection('users').doc(userId).set({email: vince#example.com})
// Store your image here:
db.collection('user_profile_pic').doc(userId).set({data: <imagedata>})

Is Firebase appropriate in pwa with offline mode?

I'm developing a PWA with Ionic/angular and Firebase/Firestore.
It manage a collection of categorized lyrics. Admins can add/delete/modify lyrics etc. Firestore is very usefull for that.
I want my PWA fully accessible offline after installation. To do that i need to read all collection (300 documents) when app loads in order to cache it. It is an important amount of reads by user, Google charge 300 reads each time app is loaded, and even more with listeners with database changes. I think, my use case is not very adapted to Firestore way to get data.
const lyricsCollection: AngularFirestoreCollection<Lyric> = this.afStore.collection<Lyric>('lyrics', ref => ref.orderBy('title'));
this.lyrics$ = lyricsCollection.snapshotChanges();
My question is: How can i improve my db or app architecture to be more adapted to my needs ?
Thanks

MongoDB Stitch vs Firebase Firestore AND/OR operations

I mainly want to compare using firestore vs mongoDB Stitch queries.
I'm building an app (angular 5 + ios + android). The app is mainly of 2 views profile and feeds.
The profile contains an array of tags(all user post tags) and a timeline of his posts.
every post got its own array of tags and timestamp. so I can filter them.
filters are either AND / OR operation.
In firestore I plan to use object maps so the AND operation can work
In firestore in order to create OR operation, I need to loop on all the possibilities fetch them individually, aggregate the data and remove duplicates. This can be done on the frontend or inside a cloud function.
I also need to run analytics to know the trending tags every day.
If a user got 100s of posts. How will firestore compare to mongoDB Stitch regarding?
Speed
Cost