If I create multiple onSnapshot listeners for the same document in different places in my code, will I be charged once (one document) or multiple times (for each listener).
Does it make sense to write a wrapper around Firestore that does this or is this built-in?
As per documentation:
Cloud Firestore allows you to listen to the results of a query and get
realtime updates when the query results change.
When you listen to the results of a query, you are charged for a read
each time a document in the result set is added or updated. You are
also charged for a read when a document is removed from the result set
because the document has changed. (In contrast, when a document is
deleted, you are not charged for a read.)
Also, if the listener is disconnected for more than 30 minutes (for
example, if the user goes offline), you will be charged for reads as
if you had issued a brand-new query.
What you decide to do afterwards will heavily depend on your use case and your application needs.
Related
I am using Google Cloud Functions and read, write operations are performed on the Firestore thru these Cloud Functions. We are seeing unexpectedly high number of read operations on Firestore, the source of which I am unable to figure out.
Not more than 20K documents are generated on a daily basis. But the daily read count is usually more than 25,000,000
What I am looking for, is ways to identify the root cause of these high number of reads in the Cloud Functions.
To start with, I have captured the size of the results of all the Firestore get() methods in Cloud Functions. But the sum total of all the sizes is much much much lower than the read count I mentioned above.
Need suggestions on ways/practices to identify the source from where these high reads are generating.
You can use a SnapshotListener as a workaround, which allows us to listen for changes in real-time.
You will be charged for readings as if we had sent a new query if the listener is disconnected for more than 30 minutes. If the listener is disconnected every 31 minutes in the worst-case scenario, we will be charged 50 reads each time.
As a result, this technique is only practicable when the listener is not frequently disconnected.
According to the documentation, I found you can reduce the number of reads using get().
In each document in the collection, you must add a new property named lastModified of type Date. When you create a new document or edit an existing one, you must use FieldValue.serverTimestamp() to set or update the field's value.
Reviewing Firebase documentation, I found that high read or write rates to lexicographically close documents need to be avoided to avoid contention faults in your application. Hotspotting is the term for this problem, and it can occur if your program does any of the following:
Creates new documents at a rapid rate and assigns its own IDs that
are monotonically rising.
A scatter algorithm is used by Cloud Firestore to assign document
IDs.
If you use automated document IDs to create new documents, you must
not see hotspotting on writes.
In a collection with few documents, creates new documents at a fast
rate.
Creates new documents at a rapid rate with a monotonically growing
field, such as a timestamp.
Deletes a large number of documents from a collection.
Writes to the database at a rapid rate without growing traffic
gradually.
I recently realized that even if a Firestore query doesn't match any document, I will still be charged for 1 read.
In my case, there could be lots of queries for non-existing docs, and I want to avoid this cost.
In my case, the client already has (or can generate locally) the relevant document Id beforehand, but the client still doesn't know if this document exists or not.
So instead of querying and receiving the doc, I can do get(docId)
Question: Does the Firestore charge for replying error to a get() request of the non-existing document?
A get() call for a document that requires the server to read data is charged as a document read. Since the server needs to check whether the document exists, that is a charged read operation (as far as i know).
The documentation on Firestore pricing says:
Minimum charge for queries
There is a minimum charge of one document read for each query that you
perform, even if the query returns no results.
So it sounds like you will be charged. The important thing to realize is that the indexes the Firestore uses to manage your documents do take time and space to maintain, so if you make use of an index, it's reasonable to expect that it's going cost money because of resources consumed.
We have large amount of data where we need to update documents based on a status. We will writing in batch of 500 chunks and wondering how many max records we can commit with a single trigger invocation?
Our trigger is a pubsub trigger in firebase cloud functions.
We see there is a limit of 540secs per invocation so would like to know how many max documents we can write in batches?
Update : Adding usecase
I have an event collection(Events) where users can subscribe for each event happening in a country.
Users have an api to see how many events they have subscribed to. They have query flags like is the event Live/Finished/Upcoming.
As I can't save list of user array who subscribed for an event in the event document(assuming subscribers can go beyond the document limit when stored), I maintained a separate sub-collection under users collection.
Ex : users/user-id/subscribedevents
Event document's status (Live/Finished/Upcoming), i'm updating from a cron job which will be running every minute. This is because I can't apply filters with two different fields (startDate & endDate).
When ever an event's status changes, i need to update in subscribedevents subcollection (which is under user's collection).
As I will be updating all the subscribedevents subcollection entries, I want to do it in batches.
Hope the usecase gives some clarity on where it is applied. As firestore is designed for scale, wondering how others are handling this scenario as its very common.
Each transaction or batch of writes can write to a maximum of 500 documents.
And 20 maximum batched writes.
References
Firestore Write Documents
Transaction and batches in firestore
Do document updates, where the update from a client, has no difference with the Firestore server copy still cost in terms of number of writes?
Also would a cloud function that is listening for updates also be invoked?
According to the Cloud Firestore Pricing documentation, each set() or update() operation will count as a write. In this case, you will still be charged with the cost of 1 write when called with new changes or not.
Document updates all incur billing for a write. But Cloud Functions and realtime listeners will not be invoked if data doesn't actually change in the document. The only situation where I suspect a listener might be invoked is if the client itself attempts to make an update to the same document it's listening to, but I'm not sure about that.
I am using swift and Firestore and in my application I have a snapshotlistener which retrieves data every time some documents are changed. As I expect this to happen many times a second, I would like to limit the snapshotlistener to retrieve data once every 2 seconds, say. Is this possible? I looked everywhere but could not find anything.
Cloud Firestore stores your data in multiple data centers, and only confirms the write operations once it's written to all of those. For this reason the maximum update frequency of a single document in Cloud Firestore is roughly once per second. So if your plan is to update a document many times per second, that won't work anyway.
There is no way to set a limit on how frequently Firestore broadcasts out updates to the underlying data. If the data gets updated, it is broadcast out to all active listeners.
The typical solution would be to limit how frequently you update the data. If nobody is going to see a significant chunk of the updates, you might as well not write them to the database. This sort of logic if often accomplished with a client side throttle/debounce (see 1, 2).