what is the best way to handle Line Chart data to send in API?
we have chart like this It has Hours, Days, Weeks, Months Data so how Will I manage with easy way?
I tired normal X,Y values but there is bulk data in every category so it is hard to handle.
my question about what data should I get from the server so I will show these bulk data easily.
As i can figure out, you are facing data processing issue while converting all seconds data to Hour, Month and other required formats every second.
If this is the issue then you can follow these steps to overcome processing data every second.
Process all data only once when you receive data for the first time
Next second when you receive data, do not process all data. Just get latest data and show in Heart Rate.
You can process all data again after sometime, based on your requirements. (this can take time)
Because as per my knowledge you are receiving all data every second, but your chart's minimum time parameter is in hours, so you have to process same data every second, which can consume little more time and results in delay.
If this is not an exact issue you are facing then please update your question with exact issue you are facing.
Related
I have a data source that ingests new data within first 30mins of every hour. The rules need to be run such that once there is new data, they should evaluate and fire if the threshold is exceeded. So roughly, 45th min of every hour.
We are not able to figure out how to do that. Also, on what basis/database column does Grafana decide when to start evaluating? I went through Grafana postgres database, it has the table alert_rule and alert_instance among others.
alert_rule has a column called updated. Is that the basis?
alert_instance has a column called last_eval_time. How is this time decided?
Grafana version: 9.2.2
Current configuration: Evaluate every 1h for 0s.
They are all firing around 31st minute of the hour. Want to understand on what basis is this happening.
Also, if there is a data point that got populated at 25th min, and the rule has fired at 31st min, will this new data point be part of the calculation?
How does Grafana behave when there is no data point available for a defined time window? For ex, consider an alert rule configured to compare between two different time windows, if one of the time window data is not found in the data source, does Grafana looks for the last data point available and pick that? We have been observing some inconsistencies around this. Our understanding is that in this case the rule should not fire.
Thanks!
Kind of related, as allowing infinitely late data would also solve my problem.
I have a pipeline linked to a pub/sub topic, from which data can come very late (days) as well as nearly real time: I have to make simple parsing and aggregation tasks on it, based on event time.
As each element holds a timestamp of the event time, my first idea when looking at the programming guide was to use windowing by setting the timestamp on event time.
However, as the processing can happen a long time after the event, I would have to allow very late data, which based on the answer of the first link I posted seems to not be such a good idea.
An alternative solution I have come up with would be to simply group the data based on event time, without marking it as a timestamp, and then calculate my aggregates based on such a key. As the data for a given even time tend to arrive at the same time in the pipeline, I could then just provide windows of arbitrary length based on processing time so that my data is ingested as soon as it is available.
This got me wondering: does that mean that windowing is useless as soon as your data comes a bit later than real time ? Would it be possible to start a window when the first element is received and close it a bit afterwards ?
I'm developing a sensor monitoring application using Thingsboard CE and PostgreSQL.
Contex:
We collect data every second, such that we can have a real time view of the sensors measurements.
This however is very exhaustive on storage and does not constitute a requirement other than enabling real time monitoring. For example there is no need to check measurements made last week with such granularity (1 sec intervals), hence no need to keep such large volumes of data occupying resources. The average value for every 5 minutes would be perfectly fine when consulting the history for values from previous days.
Question:
This poses the question on how to delete existing rows from the database while aggregating the data being deleted and inserting a new row that would average the deleted data for a given interval. For example I would like to keep raw data (measurements every second) for the present day and aggregated data (average every 5 minutes) for the present month, etc.
What would be the best course of action to tackle this problem?
I checked to see if PostgreSQL had anything resembling this functionality but didn't find anything. My main ideia is to use a cron job to periodically perform the aggregations/deletions from raw data to aggregated data. Can anyone think of a better option? I very much welcome any suggestions and input.
As a background to this question: I've been using Tableau for some time now, but I've been using code (Python, Swift, etc) as a crutch for getting some of the more complicated things done. My employer is now making me move what I can away from custom code and into retail software packages because it will make things easier to maintain if I get hit by a bus or something.
The scenario: With code, I find it very easy to deal with constantly changing/growing data by using recursion. I know that this isn't something I can do with Tableau, but I've found that for many problems so far there is a "Tableau way" of thinking/doing that can solve a lot of problems. And, I'm not allowed to use Rserve/TabPy.
I have a batch of transactional data that grows every month by about 1.6mil records. What I would like to do is build something in Tableau that can let me track a complicated rolling total across the data without having to do it manually. In my code of choice, it would have been something like:
Import the data into a frame
For every unique date value in the 'transaction date' field, create a new column with that name
Total the number of transaction in each account for that day
Write the data to the applicable column
Move on to the next day
Then create new columns that store the sum total of transactions for that account over all of the 30 day periods available (date through date + 29 days)
Select the max value of the accounts for a customer for those 30-day sums
Dump all of that 30-day data into a new table based on the customer identifier
It's a lot of steps, but with a couple of nice recursive functions, it's done in a snap with a bit of code. Plus, it can handle the data as it changes.
The actual question: How in the world do I approach problems like this within Tableau since my brain goes straight to recursive function land? I can do this manually with Tableau Prep, but it takes manual tweaking every time the data changes. Is there a better way, or is this just not within the realm of what Tableau really does?
*** Edit 10/1/2020: Minor typo fix. ***
so i am constructing a table within grafana with prometheus as a data source. right now, my queries are set to instant, and thus it's showing scrape data from the instant that the query is made (in my case, shows data from the past 2 days)
however, i want to see data from the past 14 days. i know that you can adjust time shift in grafana as well as use the offset <timerange> command to shift the moment when the query is run, however these only adjust query execution points.
using a range vector such as go_info[10h] does indeed go back that range, however the scrapes are done in 15s intervals and as such produce duplicate data in addition to producing query results for a query done in that instant
(and not an offset timepoint), which I don't want
I am wondering if there's a way to gather data from two weeks ago until today, essentially aggregating data from multiple offset time points.
i've tried writing multiple queries on my table to perform this,
e.g:
go_info offset 2d
go_info offset 3d
and so on..
however this doesn't seem very efficient and the values from each query end up in different columns (a problem i could probably alleviate with an alteration to the query, however that doesn't solve the issue of complexity in queries)
is there a more efficient, simpler way to do this? i understand that the latest version of Prometheus offers subquerys as a feature, but i am currently not able to upgrade Prometheus (at least in a simple manner with the way it's currently set up) and am also not sure it would solve my problem. if it is indeed the answer to my question, it'll be worth the upgrade. i just haven't had the environment to test it out
thanks for any help anyone can provide :)
figured it out;
it's not pretty but i had to use offset <#>d for each query in a single metric.
e.g.:
something_metric offset 1d
something_metric offset 2d