Is there a way to easily debug the read and write requests a Flutter app makes to Firestore? I'm getting a high number of reads, but battling to find where those are originating from in the app.
Have you tried using the StackDriver logging User Interface?
It offers plenty of log analysis tools that you can use to monitor resources that are writing into your Firestore DataBase.
You can read more about this here [1].
Once you have created logs-based metrics, you can create charts and alerts on said metrics.
[1] https://firebase.google.com/docs/functions/writing-and-viewing-logs
Related
Is there a way to quantify how many Firestore reads come from clients and how many from Google Cloud Functions?
I'd like to reduce my project reads costs.
Firebase currently does not provide tools to track the origin of document reads. All the reads fall under the same bucket, which is: "Read happened". If you need to measure specific reads from your app, you will have to track that yourself somehow. You can add a logger to your app which will track if the request came from the client or just the Cloud Function itself.
This documentation may come in handy.
Firestore audit logging information
Google Cloud services write audit logs to help you answer the
questions, "Who did what, where, and when?" within your Google Cloud
resources.
Data Access audit logs
Includes "admin read" operations that read metadata or configuration
information. Also includes "data read" and "data write" operations
that read or write user-provided data.
To receive Data Access audit logs, you must explicitly enable them.
https://cloud.google.com/firestore/docs/audit-logging
I am building a stock portfolio app and am facing an issue with data flow management. I have 3 collections for users, transactions and accounts in firebase. I use a stream provider to fetch data for transactions from firebase into the flutter app. In addition, I have a service that fetches and saves the current stock data(price, percentage change, volume, etc) from a REST API into a list within the flutter app. I wanted to understand what is the best way to manage data from both these endpoints and be able to show them on the screen, update them whenever they get updated.
If
I would suggest having a proxy server in the middle to manage your communications for :
Managing Firebase collections
Fetching and save current stock price data
Load balancing and proxying.
The reasons for suggesting the same are:
Decoupling of data flow into microservices. Your Flutter app should just manage UI and hit the necessary api endpoints. Your logic for resolving client requests should only exist on the server. This will reduce your app bundle size and help with horizontal-scaling should the traffic increase.
Can cache common responses by applying Redis on this server.
Better application of security rules for both Firebase and REST API compared to Flutter application
Say tomorrow you wish to add more API services, this will just mean adding those endpoints on this server and hitting them with your Frontend application.
Having such an architecture will reduce the dependency of the entire stack on each independent service (be it your Flutter app or your Firebase backend).
I would like to achieve a real time change data capture (log-based preferred) pipeline from Google Cloud Spanner to PubSub/Kafka for my downstream real time applications. Could you please let me know if there is a great and cost-effective way to achieve that? I will appreciate any advice and recommendations.
In addition, for Cloud Data Fusion from google, I noticed that it could achieve real time from mysql/postgresql to cloud spanner, but I did not find the way go from cloud spanner to pubsub/kafka in real time.
Also, I found another two ways, which to be listed here for any comments or suggestions.
Use Debezium, a log-based change data capture Kafka connector from the link https://cloud.google.com/architecture/capturing-change-logs-with-debezium#deploying_debezium_on_gke_on_google_cloud
Create a polling service (which may miss some data) to poll data from cloud spanner from the link: https://cloud.google.com/architecture/deploying-event-sourced-systems-with-cloud-spanner
If you have any suggestion or comment on this, I will be really grateful.
There's a open source implementation of a polling service for Cloud Spanner that can also automatically push changes to PubSub here: https://github.com/cloudspannerecosystem/spanner-change-watcher
It is however not log-based. It has some inherent limitations:
It can miss updates if the same record is updated twice within the polling interval. In that case, only the last value will be reported.
It only supports soft deletes.
You could have a look at the samples to see if it is something that might suit your needs at least to some degree: https://github.com/cloudspannerecosystem/spanner-change-watcher/tree/master/samples
Cloud Spanner has a new feature called Change Streams that would allow building a downstream pipeline from Spanner to PubSub/Kafka.
At this time, there's not a pre-packaged Spanner to PubSub/Kafka connector.
The way to read change streams currently is to use the SpannerIO Apache Beam connector that would allow building the pipeline with Dataflow, or also directly querying the API.
Disclaimer: I'm a Developer Advocate that works with the Cloud Spanner team.
I'm currently facing to a huge pick of Read on Cloud Firestore and no way to go up stream to find the issue. I saw first this increase on Cloudflare. From 1million request to 175millions in 3 days with no correlation with user activity.
Cloudflare Dashboard before
Cloudflare Dashboard after
Diving into Statistics from GCP and Firebase is even more confusing had they are reflecting different reality.
GCP Dashboard Cloud Firestore Read and Write
Firebase Dashboard Firestore Read and Write
I verified if it was correlated to a new development or new security rule, but nothing.
I was thinking for a time about an hacking but Write seems to follow Read, but sure of nothing.
Can anyone had a previous experience like that, or a hint of where to find more infos on GCP.
Thanks for reading guys
I want to build an internal dashboard to show the key metrics of a startup.
All data is stored in a mongodb database on Mongolab (SaaS on top of AWS).
Queries to aggregate datas from all documents take 1-10minutes.
What is the best practice to cache such data and make it immediately available?
Should I run a worker thread once a day and store the result somewhere?
I want to build an internal dashboard to show the key metrics of a startup. All data is stored in a mongodb database on Mongolab (SaaS on top of AWS). Queries to aggregate datas from all documents take 1-10minutes.
Generally users aren't happy to wait on the order of minutes to interact with dashboard metrics, so it is common to pre-aggregate using suitable formats to support more realtime interaction.
For example, with time series data you might want to present summaries with different granularity for charts (minute, hour, day, week, ...). The MongoDB manual includes some examples of Pre-Aggregated Reports using time-based metrics from web server access logs.
What is the best practice to cache such data and make it immediately available?
Should I run a worker thread once a day and store the result somewhere?
How and when to pre-aggregate will depend on the source and interdependency of your metrics as well as your use case requirements. Since use cases vary wildly, the only general best practice that comes to mind is that a startup probably doesn't want to be investing too much developer time in building their own dashboard tool (unless that's a core part of the product offering).
There are plenty of dashboard frameworks and reporting/BI tools available in both open source and commercial flavours.