My page is quite small it has around 300-1000 visits each day. But at some point I started to accumulate HUGE firestore read requests:
Till the 8th date it was somewhere around 50K each day. I am pushing new code all the time so I'm not sure what I did. Looking at the page I don't see anything out of ordinary. Is there some sort of log in google or firestore I could look at?
The Firebase documentation indicates that each time you create a project, it also creates a project in Google Cloud Platform, therefore you can track daily Cloud Firestore usage like writes, deletes, etc. This usage information is shown in the GCP's console in the App Engine Quotas page .You can see more details in the link. https://firebase.google.com/docs/firestore/monitor-usage#google-cloud-platform-console
There is currently no way to track the origin of reads. What you're looking at now is the best indicator you have available.
Bear in mind that the Firebase and Cloud consoles show updates to documents in real time, and each document update costs a read. If you leave the console open on a busy collection, it will rack up reads.
Related
As Firestore charges by the read/write, it would be super helpful to keep the changes in memory during the session and only commit them when the user exists either the entire app or a specific section. Is there a way to do that in a Flutter web application?
I think one problem with this approach is that the user might just close the tab including your app. In this case, you have no time to send your data to Firestore.
This aside, you could use packages like Hive to store your documents offline and later run a function to add the data to Firestore later.
You also have 50k reads and 20k writes for free with Firebase, which is sufficient for smaller apps. If you exceed this limit, your app is probably big enough to earn money with it anyway.
So I'm making an app that involves streaming audio(radio-like) from the Google Cloud Storage and was looking into the costs. It seems it would be much too expensive as is.
e.g. Lets say I have 10MB audio files, a user listens to 20 files a day and I have 2000 active users. That's 400GBs or $48/day. i.e. ~$1440/month just for that.
I then looked into putting a CDN in front of it, to minimize direct reads from the Storage. Now initially that made sense to me. The CDN would cache the audio files and the clients would be getting the files from the cache most of the time. However, as I was looking at Fastly's pricing (Fastly is a Google partner and seems like a good fit) I noticed that they seem to be pricing bandwidth usage to their cache at the exact same rate as Google cloud does ($0.12/GB). So unless I'm reading this wrong, putting up the CDN would not save me ANY money. Now I get that there are other reasons why putting a CDN in front of it could be a good idea, but am I really reading this right?
Also, if you have any other tips on how I should set this up, I'm all ears.
Estimating the invoice of such a service is a complex matter. To get an informed answer and tips regarding possible implementation paths I would suggest reaching out to a GCP Sales representative. Similarly you should contact the Fastly team to get a precise picture of their pricing model.
Moreover, any estimate we could make here would be outdated as soon as any of the respective pricing model changes, which would invalidate the answer and probably drive future readers to wrong conclusions.
I'm currently facing to a huge pick of Read on Cloud Firestore and no way to go up stream to find the issue. I saw first this increase on Cloudflare. From 1million request to 175millions in 3 days with no correlation with user activity.
Cloudflare Dashboard before
Cloudflare Dashboard after
Diving into Statistics from GCP and Firebase is even more confusing had they are reflecting different reality.
GCP Dashboard Cloud Firestore Read and Write
Firebase Dashboard Firestore Read and Write
I verified if it was correlated to a new development or new security rule, but nothing.
I was thinking for a time about an hacking but Write seems to follow Read, but sure of nothing.
Can anyone had a previous experience like that, or a hint of where to find more infos on GCP.
Thanks for reading guys
I am getting very high counts of Entity Writes in my firestore database.
Write permission in most of the paths are restricted, done from back-end server using admin SDK. Only a very few paths have write access- specifically only to the users who are (authenticated & registered & joined and approved in a specific group), so even though the ways to abuse are apparently thin, yet hard to specifically identify.
Only way I see- is to execute Cloud Functions on every write, and have the function log the paths somewhere to analyze. But that introduces further costs and complexity.
Is there any way/recommendation to monitor/profile where (i.e.- path) and who (UID or any identity) are performing the writes? There are tools to do such for RTDB, bu't can't find anything for Firestore.
I am also wondering if there is any way to restrict ip/users automatically in case of abuse (i.e.- high rate of read/write)?
What I'm currently doing is going to firestore console => menu usage => view usage
and I see something like this:
It's not the same as the profiler, but better than nothing.
I'm also keeping an eye on the video on the link below to see if someone provides an answer. People are asking for the profiler too.
https://www.youtube.com/watch?v=9CObBsjk6Tc
I will build a dashboard system for my apps, where a page will have several widgets that draw charts, tables and glyphs representing potentially unrelated data.
The client will be HTML5 and I can push for only modern web browser.
My big problem is what backend use for this. I want to store "tables" for use in the charts and in real-time update the widgets.
For example, a invoicing widget will show how much $$ have been collected today. In the "table" will have a row for each total of the invoice:
inv = 1; total = 50
Total: 50
and the widget will draw that. When new data is pushed:
inv = 2; total = 100
Total: 150
The widget will show in realtime the total to the end-user.
The data is private for the user company. Eventually I will need to purge too old data (ie: I only need to keep as much data is necessary to proper evaluation of the info need for the end-user. For example, only keep 1 month of invoicing totals).
I'm thinking in use something like http://www.firebase.com/ or http://pusher.com/ but I suspect only solve the "notify in realtime" part of the equation. As far as I understand, they not let me get past data (ie: If the data is update in the weekend and the user open his dashboard to see what happened)
Then I see http://derbyjs.com/ and the possibility to use mongodb.
I wonder which backend/platform will bring me closer to the build of this system. I have experience with python/django/.net/postgress but could accept the use of something else if solve best this kind of app behavior.
Firebase offers both the "notify in relatime" part that you mention, as well as persistent data storage. Take a look at the tutorial, which walks you through building a real-time persisted chat app (the past chat messages are stored in Firebase and are sent back to the client every time you reload). And you can do much more complicated stuff like the real-time charts / widgets that you mention as well.
The big limitation with Firebase right now is that we're in closed beta and the data is currently unprotected (anybody can read and write your data). The security features are coming soon though.
Some other backend platforms you may want to evaluate are: Meteor and Simperium. Firebase and Simperium are cloud services where your data is stored in the cloud and you don't have to manage any servers of your own, while Meteor and DerbyJS are platforms that you have to install and run on your own server.
I would recommend signalR. It's amazing and you can literally do anything with it. Check it out: www.signalr.net and if you have any problems simply go to www.jabbr.net You will find a very helpful community there. I implemented a notification mechanism similar to facebook together with real time monitoring and a small chat in the same web site.