So, I have grafana, where my underlying database is AWS Timestream.
I have around 20 widgets (panes) in my dashboard, which makes 20 calls to the datasource.
AWS prices 10mb minimum per query, which causes me to pay 200mb worth of queries instead of few MB which is the amount of data scanned.
What I want to do, is to run the a unified query (take the 20 queries and run them once with outer joins), then perform the queries of the panes against this result.
Is this even possible?
You can try reuse query result https://grafana.com/blog/2020/10/14/learn-grafana-share-query-results-between-panels-to-reduce-load-time/ and apply transformations.
yes, in Grafana there is a special built-in datasource called dashboard. Select this option and you can re-use the query from another panel. Very useful for reducing the number of queries you make to a backend for more or less the same data. As Jan says, you can then use transformations to fiddle with the data:
https://grafana.com/docs/grafana/latest/datasources/
Related
I am looking to support a use case that returns kdb datasets back to users. The users connects to kdb using the Java API, runs the query synchronously and retrieves results.
However, issues are coming up when returning larger datasets and therefore I would like to return the data from kdb to the java process in pages/slices. Unfortunately users need to be able to run queries that return millions of rows and it would be easier to handle if they were passed back in slices of say 100,000 rows (Cassandra and other DBs do this sort of thing).
The potential approaches I have come up with are as follows:
Run the "where" part of the query on the database and return only the indices/date partitions (if applicable) of the data required. The java process would then use these indices to select the data required slice by slice . This approach would control memory usage on the kdb side as it would not have to load all HDB data required at once. However, overall this would increase the run time of the query as data would have to be searched/queried multiple times. This could work well for simple selects but complicated queries may need to go through an "onboarding" process which I want to avoid.
Store results of the query in a global variable in kdb which the java process can then query slice by slice. This simpler method could support any query but could potentially hit limits on the kdb side (memory/timeout) if too large a dataset is queried.
Other points to consider:
It should support users running queries on any type of process - gateway, hdb, rdb etc
It should support more than just simple selects e.g.
((1!select sym, price from trade where sym=`AAA) uj
1!select sym,price from order where sym=`AAA)
lj select avgBid:avg bid by sym from quote where sym=`AAA
The paging functionality should be removed from the end user
Does anyone have any views on if there are there any options available other than the ones listed above? Essentially I am looking for a select[m n] type approach that supports any query.
Is it possible to add a large amount of SELECT fields(in my example) to the dashboard?
Lets say I have 96 processors on an ESXi host I want to monitor metrics on, is there a less tedious way other than manually adding each field in the "Metrics" tab?
Use single query with proper GROUP BY part, so this single query returns multiple time series, which will be visualized by Grafana.
I have a couple of millions entries in a table which start and end timestamps. I want to implement an analysis tool which determines unique entries for a specific interval. Let's say between yesterday and 2 month before yesterday.
Depending on the interval the queries take between a couple of seconds and 30 minutes. How would I implement an analysis tool for a web front-end which would allow to quite quickly query this data, similar to Google Analytics.
I was thinking of moving the data into Redis and do something clever with interval and sorted sets etc. but I was wondering if there's something in PostgreSQL which would allow to execute aggregated queries, re-use old queries, so that for instance, after querying the first couple of days it does not start from scratch again when looking at different interval.
If not, what should I do? Export the data to something like Apache Spark or Dynamo DB and analysis in there to fill Redis for retrieving it quicker?
Either will do.
Aggregation is a basic task they all can do, and your data is smll enough to fit into main memory. So you don't even need a database (but the aggregation functions of a database may still be better implemented than if you rewrite them; and SQL is quite convenient to use.
Jusr do it. Give it a try.
P.S. make sure to enable data indexing, and choose the right data types. Maybe check query plans, too.
I have a table of 3M rows.
I wanted to retrieve all those rows and do a visualization using dc.js.
Problem I have is, for just a single column it takes about 70 secs.
And If i write my query it takes about 240 secs to retrieve those rows.
I'm using using select query on columns like this.
SELECT COL1, COL2 FROM TABLE
That's it. No grouping, nothing.
But it takes hell lot of time.
Heard of indexing and I created a Index for the columns I use. But even though no fruitful results.
We should not retrieve 3M rows in any query. And sending 3M records will always take a lot of time (nothing to do with the database, it is transfer speed). It will kill your IO. The bulk of the time taken is on the transfer (IO) from request-originator and the postgres database.
Consider to break that requests into batches of async-requests that gets streamed down to clients. That means restructuring your front-end code (javascript) for improved user-experience.
You didn't specify the environment in which you are using PostgreSQL.
As an example, in Node.js you can solve this problem by streaming the data with the help of pg-query-stream and rendering it on the client side at the same time, so the client doesn't have to wait for the query to finish and can see intermediate results.
This may not be the best solution though. A better solution would be to implement data aggregation within a database function to provide a smaller data subset.
i am new at db2 i want to select around 2 million data with single query like that
which will select and display first 5000 data and in back process it will select other 5000 data and keep on same till end of the all data help me out with this how to write query or using function
Sounds like you want what's known as blocking. However, this isn't actually handled (not the way you're thinking of) at the database level - it's handled at the application level. You'd need to specify your platform and programming language for us to help there. Although if you're expecting somebody to actually read 2 million rows, it's going to take a while... At one row a second, that's 23 straight days.
The reason that SQL doesn't really perform this 'natively' is that it's (sort of) less efficient. Also, SQL is (by design) set up to operate over the entire set of data, both conceptually and syntactically.
You can use one of the new features, that incorporates paging from Oracle or MySQL: https://www.ibm.com/developerworks/mydeveloperworks/blogs/SQLTips4DB2LUW/entry/limit_offset?lang=en
At the same time, you can influence the optimizer by indicating OPTIMIZED FOR n ROWS, and FETCH FIRST n ROWS ONLY. If you are going to read only, it is better to specify this clause in the query "FOR READ ONLY", this will increase the concurrency, and the cursor will not be update-able. Also, assign a good isolation level, for this case you could eventually use "uncommitted read" (with UR). A Previous Lock table will be good.
Do not forget the common practices like: index or cluster index, retrieve only the necessary columns, etc. and always analyze the access plan via the Explain facility.