I have 4 timeseries panels in my dashboard and the query does not have any timefilter in it. So, all the data is fetched.
Now, when ever I zoom in or out, these queries are re-executed. Since, they are going to fetch same results, there is no use in executing them again. How can I disable this re-execution?
Related
So, I have grafana, where my underlying database is AWS Timestream.
I have around 20 widgets (panes) in my dashboard, which makes 20 calls to the datasource.
AWS prices 10mb minimum per query, which causes me to pay 200mb worth of queries instead of few MB which is the amount of data scanned.
What I want to do, is to run the a unified query (take the 20 queries and run them once with outer joins), then perform the queries of the panes against this result.
Is this even possible?
You can try reuse query result https://grafana.com/blog/2020/10/14/learn-grafana-share-query-results-between-panels-to-reduce-load-time/ and apply transformations.
yes, in Grafana there is a special built-in datasource called dashboard. Select this option and you can re-use the query from another panel. Very useful for reducing the number of queries you make to a backend for more or less the same data. As Jan says, you can then use transformations to fiddle with the data:
https://grafana.com/docs/grafana/latest/datasources/
I'm developing a pipeline that communicates a lot to my postgres database.
I'm trying to figure out how I can create a "dry run" mode for my pipeline that essentially uses the fewest resources possible just to see if the pipeline can and would run properly.
I understand that I could just not commit my changes to my database at the end of my script, but it takes about an hour for my pipeline to run all the way through. So it'd be cool if I could just simulate a lot of my postgres queries without actually taking the time to run all of them.
I'll give you an extremely simple example so you can see what I mean...
Let's say the only things in my pipeline are refreshing a bunch of materialized views. But they take a long time to refresh, and there's a lot of them. So I'd like to simply ask the postgres db... "hey, if I ran these refreshes right now, would it go through correctly?"
I feel like this is kinda what happens during the "query planning" phase, but this is just an intuition.
I also acknowledge that the only way to know for absolute sure is to actually run it, but let's just say for now that I know and accept the risks that come along with that.
For INSERT, UPDATE, DELETE and SELECT statements, you could prepend EXPLAIN to the statement. That won't return the proper result set, but when it succeeds, the statement should run (unless there is a runtime error).
However, that does not work with REFRESH MATERIALIZED VIEW...
I guess the best option you have is to use a database that has all the tables, views, functions etc., but is empty. Most statements will complete very quickly on an empty database.
I want to implement stable pagination using Postgres database as a backend. By stable, I mean if I re-read a page using some pagination token, the results should be identical.
Using insertion timestamps will not work, because clock synchronization errors can make pagination unstable.
I was considering using pg_export_snapshot() as a pagination token. That way, I can reuse it on every read, and the database would guarantee me the same results since I am always using the same snapshot. But the documentation says that
"The snapshot is available for import only until the end of the transaction that exported it."
(https://www.postgresql.org/docs/9.4/functions-admin.html)
Is there any workaround for this? Is there an alternate way to export the snapshot even after the transaction is closed?
You wouldn't need to export snapshots; all you need is a REPEATABLE READ READ ONLY transaction so that the same snapshot is used for the whole transaction. But, as you say, that is a bad idea, because long transactions are quite problematic.
Using insert timestamps I see no real problem for insert-only tables, but rows that get deleted or updated will certainly vanish or move unless you use “soft delete and update” and leave the old values in the table (which gives you the problem of how to get rid of the values eventually). That would be re-implementing PostgreSQL's multiversioning on the application level and doesn't look very appealing.
Perhaps you could use a scrollable WITH HOLD cursor. Then the database server will materialize the result set when the selecting transaction is committed, and you can fetch forward and backward at your leisure. Sure, that will hog server resources, but you will have to pay somewhere. Just don't forget to close the cursor when you are done.
If you prefer to conserve server resources, the obvious alternative would be to fetch the whole result set to the client and implement pagination on the client side alone.
Is there a way to limit a query execution by time?
I'm familiar with the TIMEOUT clause of a SELECT statement, but I can't seem to find an equivalent for the graph queries, such as MATCH or TRAVERSE.
An alternative for a timeout will be to identify somehow the problematic query and avoid it, but I can't find any EXPLAIN statement to perform without actually perform the query...
The situation is quite problematic as performing 2-3 MATCH queries may explore the whole graph, take forever to complete and only hard-reset releases them.
I've also searched for an option to cancel/interrupt/kill a query but couldn't find it (only via HTTP API).
We're using the JAVA API with version 2.2.30.
Thanks.
so i am constructing a table within grafana with prometheus as a data source. right now, my queries are set to instant, and thus it's showing scrape data from the instant that the query is made (in my case, shows data from the past 2 days)
however, i want to see data from the past 14 days. i know that you can adjust time shift in grafana as well as use the offset <timerange> command to shift the moment when the query is run, however these only adjust query execution points.
using a range vector such as go_info[10h] does indeed go back that range, however the scrapes are done in 15s intervals and as such produce duplicate data in addition to producing query results for a query done in that instant
(and not an offset timepoint), which I don't want
I am wondering if there's a way to gather data from two weeks ago until today, essentially aggregating data from multiple offset time points.
i've tried writing multiple queries on my table to perform this,
e.g:
go_info offset 2d
go_info offset 3d
and so on..
however this doesn't seem very efficient and the values from each query end up in different columns (a problem i could probably alleviate with an alteration to the query, however that doesn't solve the issue of complexity in queries)
is there a more efficient, simpler way to do this? i understand that the latest version of Prometheus offers subquerys as a feature, but i am currently not able to upgrade Prometheus (at least in a simple manner with the way it's currently set up) and am also not sure it would solve my problem. if it is indeed the answer to my question, it'll be worth the upgrade. i just haven't had the environment to test it out
thanks for any help anyone can provide :)
figured it out;
it's not pretty but i had to use offset <#>d for each query in a single metric.
e.g.:
something_metric offset 1d
something_metric offset 2d