I've been looking through the docs and grafana community but can't seem to find a definitive answer to this.
I have Grafana configured witha PostgreSQL datasource and created a dashboard to monitor the number of new sessions being created in my database. This works, and I can see a graph of sessions being generated over time.
My question is regarding where the numbers are recorded, if anywhere? If I have this graph on my dashboard, does it go away and make the query every single time the page is loaded? My main concern is that a user can change the time period, going back potentially years, which would hammer the database grouping all those sessions into time intervals.
If they are not stored anywhere with Grafana, how are people managing this? Would we need to use another 'middle man' to receive all of the stats, and use this as the datasource instead of the PostgreSQL database?
When you configured the datasource, you are telling grafana to pick data from there. In your case it is your postgres db.
Grafana does not copy data anywhere else.
If you want that then you should store that data somewhere else. You can setup monitoring for postgresql and store metrics in some engines like prometheus.
Related
I am using grafana to visualize some data from my InfluxDB-database, that collects data from some sensors. That said, it's the first time for my working with both grafana and InfluxDB, also I'm pretty new to coding so my knowledge is very limited.
As I scroll through threads and forums on the web trying to find guidance, I find a lot of tutorials mostly 2-4 years old that seem to use features in grafana that are simply not available vor me.
For example I tried to set an alert which tells me when my sensor is delivering flawed values (values that I my case cannot physically be true) too often. But when I'm using avg() from the classic condition operations, I can't select a time frame in which I want the average value monitored.
My expression part of the alert settings
Is it a problem that has to be configured via grafana.ini? Is it because these features cannot be used with InfluxDB?
For some background information, I'm using a Ubuntu Server via VirtualBox to run both the database and the grafana server. I'm using a little python script to distribute the sensor data into the database.
If someone could help me out soon that would be great!
I'm wondering how Grafana fires it's queries (to the datasource).
Does it only fire queries when the dashboard/panel is visible or does it keep on firing queries based on the default time period of all dashboards ?
My assumption was that since Grafana itself doesn't store data, the queries would be on a need basis, but I've been seeing http requests occur periodically from my VM.
This is relevant for metrics such as CloudWatch etc, where each API call can be charged.
Yes, you are correct. Grafana dashboards/panels fire queries only when they are visible (loaded in the browser).
But you may have alert rules in the Grafana and they should be evaluated also periodically. So I guess alerts are source of your queries.
We would like to collect some interesting user-related metrics on our website (e.g. "user edited profile", or "user clicked on downloaded file", etc.) and are thinking about using the ELK stack for this.
Is it a good idea to use Elasticsearch to store such events? Or would it make more sense to log them in our RDBMS?
What would be the advantages of using either of those?
(Side note: We already use Elasticsearch and PostgreSQL in our our stack.
You could save your logs in any persistent solution out there and later decide what tool to use for analyzing them.
If you want to do some queries (manage your data on the fly/real-time) you could just directly parse/pipe the logs generated by your applications and send them to elastic search, the flow would be something like:
(your app) --> filebeat --> elasticsearch <-- Kibana
Just keep in mind that the elk stack is not "cheap" and based on your setup could become more expensive to maintain in long term.
At the end depends on your use case, both solutions you mention can be used to store data, but the way you extract/query data is the one that makes the difference.
I want to build an internal dashboard to show the key metrics of a startup.
All data is stored in a mongodb database on Mongolab (SaaS on top of AWS).
Queries to aggregate datas from all documents take 1-10minutes.
What is the best practice to cache such data and make it immediately available?
Should I run a worker thread once a day and store the result somewhere?
I want to build an internal dashboard to show the key metrics of a startup. All data is stored in a mongodb database on Mongolab (SaaS on top of AWS). Queries to aggregate datas from all documents take 1-10minutes.
Generally users aren't happy to wait on the order of minutes to interact with dashboard metrics, so it is common to pre-aggregate using suitable formats to support more realtime interaction.
For example, with time series data you might want to present summaries with different granularity for charts (minute, hour, day, week, ...). The MongoDB manual includes some examples of Pre-Aggregated Reports using time-based metrics from web server access logs.
What is the best practice to cache such data and make it immediately available?
Should I run a worker thread once a day and store the result somewhere?
How and when to pre-aggregate will depend on the source and interdependency of your metrics as well as your use case requirements. Since use cases vary wildly, the only general best practice that comes to mind is that a startup probably doesn't want to be investing too much developer time in building their own dashboard tool (unless that's a core part of the product offering).
There are plenty of dashboard frameworks and reporting/BI tools available in both open source and commercial flavours.
I am using Tableau Desktop 8.2 and Tableau server 8.2 (Licensed versions) , the workbook created in Tableau are successfully published to Tableau server.
But when the user want to see the views or workbooks it takes a very long time to preview or open?
The Workbooks are created with Amazon RedShift Database having (>5 million records)
Could somebody guide me on this? like what is it taking a long to preview or open even after being published to Tableau server?
First question, are the views performant when opened using only Tableau Desktop? Get them working well on Desktop before introducing Server into the mix.
then look at the logs in My Tableau Repository which include query strings and timing info to see if you can narrow down the cause. You can also try the Performance Recorder feature.
A typical problem is an overly expensive query just to display a dashboard. In that case, simplify. Start with a simple high level summary viz and the introduce complexity testing the impact on performance. If one viz is too slow, there are usually alternative approaches available
Completely agree with Alex; I had a similar issue with HP Vertica. I had lot of action set on the dashboard. Considering the database structure is final, I did created the tableau extract and used the Online tableau extract in place of live connection. Vola! that solved my problem and the users are happy with the response time as well. Hope this helps you too..
Tableau provides two mode of data refreshes:
Live : Tableau will execute underlying queries every time the
dashboard is referred / refreshed. Apart from badly formulated
queries, this is one of the reason why your dashboard on Tableau Online
might take forever to load.
Extract : Query will be executed once, according to (your) specified
schedule and same data will reflect everytime the dashboard is
refreshed.
In extract mode, the time is taken only when the extract is being refreshed. However, if the extract is not refreshed periodically, the same, stale data will reflect on the dashboard. Thus, extract view is not recommended for representations of live data.
You can toggle Live <--> Extract from the Data Source pane of Tableau Desktop. (refer top right of the snapshot).