I get logs for every call to my API. From these logs, I can retrieve interesting information that I use in a reporting dashboard.
My problem is that the number of those logs is getting bigger and bigger, and now I am looking for a new solution to store them.
Do I store logs or I store only the information I get from these logs?
Which database do I choose for storage (MySQL, HBase, MongoDB,Cassandra) ?
Related
Is there a way to quantify how many Firestore reads come from clients and how many from Google Cloud Functions?
I'd like to reduce my project reads costs.
Firebase currently does not provide tools to track the origin of document reads. All the reads fall under the same bucket, which is: "Read happened". If you need to measure specific reads from your app, you will have to track that yourself somehow. You can add a logger to your app which will track if the request came from the client or just the Cloud Function itself.
This documentation may come in handy.
Firestore audit logging information
Google Cloud services write audit logs to help you answer the
questions, "Who did what, where, and when?" within your Google Cloud
resources.
Data Access audit logs
Includes "admin read" operations that read metadata or configuration
information. Also includes "data read" and "data write" operations
that read or write user-provided data.
To receive Data Access audit logs, you must explicitly enable them.
https://cloud.google.com/firestore/docs/audit-logging
I've been looking through the docs and grafana community but can't seem to find a definitive answer to this.
I have Grafana configured witha PostgreSQL datasource and created a dashboard to monitor the number of new sessions being created in my database. This works, and I can see a graph of sessions being generated over time.
My question is regarding where the numbers are recorded, if anywhere? If I have this graph on my dashboard, does it go away and make the query every single time the page is loaded? My main concern is that a user can change the time period, going back potentially years, which would hammer the database grouping all those sessions into time intervals.
If they are not stored anywhere with Grafana, how are people managing this? Would we need to use another 'middle man' to receive all of the stats, and use this as the datasource instead of the PostgreSQL database?
When you configured the datasource, you are telling grafana to pick data from there. In your case it is your postgres db.
Grafana does not copy data anywhere else.
If you want that then you should store that data somewhere else. You can setup monitoring for postgresql and store metrics in some engines like prometheus.
I am using a web application for doing data entry which has a mechanism for storing the data entry form (which is an html form) in the browser cache IndexDB.
I am able to see the form in the browser dev tool like so :
I want to know for how long the Index DB will be able to store the form in the browser? Is it possible that it is months since the browser cache was same? Will closing the browser clear the keys? or is this persistent enough storage to last for a few months?
Is it possible to find out when(the exact date or time) the cache entry was made in the IndexDB?
I am asking this because I suspect some discperancy in the form for some of our users as the data being sent is a little different than expected.
Any help would is appreciated.
Thanks
DHIS2, the application you are referring to, has an application you and other users can use to clear any cached data. This app is named "Browser Cache Cleaner", and gives you a list of different things to clear. I would try this app and see if your users still have these issues.
Databases don't expose the timestamp of when the database record was last modified. That's something the developer needs make the application to store in the database records. For example, one could have created_at and modified_at columns to track when the record was created and when was it last modified.
IndexedDB is a persistent client storage API, so yes, data will stay permanently unless the user clears the browser's cache.
If there is some discrepancy in the form being sent, I would look at the caching strategy. Offline data caching is a pretty broad topic (also I don't know much about your application), but Google's Offline Cookbook is a good place to start digging in this topic, as long as caching strategies for your use.
We would like to collect some interesting user-related metrics on our website (e.g. "user edited profile", or "user clicked on downloaded file", etc.) and are thinking about using the ELK stack for this.
Is it a good idea to use Elasticsearch to store such events? Or would it make more sense to log them in our RDBMS?
What would be the advantages of using either of those?
(Side note: We already use Elasticsearch and PostgreSQL in our our stack.
You could save your logs in any persistent solution out there and later decide what tool to use for analyzing them.
If you want to do some queries (manage your data on the fly/real-time) you could just directly parse/pipe the logs generated by your applications and send them to elastic search, the flow would be something like:
(your app) --> filebeat --> elasticsearch <-- Kibana
Just keep in mind that the elk stack is not "cheap" and based on your setup could become more expensive to maintain in long term.
At the end depends on your use case, both solutions you mention can be used to store data, but the way you extract/query data is the one that makes the difference.
I'm at the point of giving Zumero a shot. I have downloaded the Zumero Client SDK for the cloud and was going through the getting started and zumero_core.
I am wondering if my impression of Zumero is correct and if it is not recommended to use Zumero in my use case.
My impression: Zumero makes an exact replicate of the SQLite database on the Client in the Cloud. It keeps these two in sync. (with zumero_sync())
My Use-Case: Reservation app, so there is one big database and every consumer will only have her records locally. So a small part of the big database, the consumer can't have access to the other data. So I wanted to provide certain columns and rows with sync capability.
Is it true that Zumero replicates and by this can not help me to achieve my use case, or am I missing something?
With Zumero-in-the-cloud, no, you can't directly do that. You'd need to create separate DBFiles per customer.
Zumero for SQL Server does allow server-side filtering, so you could filter per-user, choose a subset of columns that the user can see, etc.