Measure the time it takes to execute a PostgreSQL query - postgresql

Based on Measure the time it takes to execute a t-sql query, how would one time several trials of a query in PostgreSQL?
A general outline would be
-- set up number of trials (say 1000)
SELECT CURRENT_DATE ; -- save start time
BEGIN
LOOP
-- execute query to be tested
END LOOP;
END;
SELECT CURRENT_DATE ; -- save end time
I.E. I want a PostgreSQL equivalent of the following TSQL code, taken from an answer by HumbleWebDev from the linked TSQL question: see [reference for code]
declare #tTOTAL int = 0
declare #i integer = 0
declare #itrs integer = 100
while #i < #itrs
begin
declare #t0 datetime = GETDATE()
--your query here
declare #t1 datetime = GETDATE()
set #tTotal = #tTotal + DATEDIFF(MICROSECOND,#t0,#t1)
set #i = #i + 1
end
select #tTotal/#itrs
-- your query here: Standard SQL queries such as Select * from table1 inner -- join table2, or executing stored procedure, etc.

Coming from an MSSQL background myself and now more often working in Postgres I feel your pain =)
The "trouble" with Postgres is that it supports only 'basic' SQL commands (SELECT, INSERT, UPDATE, CREATE, ALTER, etc...) but the moment you want to add logic (IF THEN, WHILE, variables, etc.) you need to switch to pl/pgsql which you can only use inside functions (AFAIK). From a TSQL POV there are quite some limitations and in fact, some things suddenly don't work anymore (or need to be done differently.. e.g. SELECT * INTO TEMPORARY TABLE tempTable FROM someTable will not work but CREATE TABLE tempTable AS SELECT * FROM someTable will)
Something I learned the hard way too is that CURRENT_TIMESTAMP (or Now()) will return the same value within a transaction. And since everything inside a function runs inside a transaction this means you have to use clock_timstamp()
Anyway, to answer your question, I think this should get you going:
CREATE OR REPLACE FUNCTION fn_test ( nbrOfIterations int)
RETURNS TABLE (iterations int, totalTime interval, secondsPerIteration int)
AS $$
DECLARE
i int;
startTime TIMESTAMP;
endTime TIMESTAMP;
dummy text;
BEGIN
i := 1;
startTime := clock_timestamp();
WHILE ( i <= nbrOfIterations) LOOP
-- your query here
-- (note: make sure to not return anything or you'll get an error)
-- example:
SELECT pg_sleep INTO dummy FROM pg_sleep(1);
i := i + 1;
END LOOP;
endTime := clock_timestamp();
iterations := nbrOfIterations;
totalTime := (endTime - startTime);
secondsPerIteration := (EXTRACT(EPOCH FROM endTime) - EXTRACT(EPOCH FROM startTime)) / iterations;
RETURN NEXT;
END;
$$ language plpgsql;
SELECT * FROM fn_test(5);

While the accepted answer is correct, this tweaking of it worked better for me. Again, I want to emphasize this extra answer below is based on the above answer, and it would not be possible without it. It just works better in my own situation to use the tweak I made below.
The answer below is indeed almost entirely based on the accepted answer. However, I changed how the return is used and also seconds to milliseconds:
----------------------------------------------------------------------------------------------------
-- fn__myFunction_Q.sql
----------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
-- DROP FUNCTION mySchema.fn__myFunction
--------------------------------------------------------------------------------------------
CREATE OR REPLACE FUNCTION mySchema.fn__myFunction ( nbrOfIterations int)
RETURNS TABLE (iterations int, totalTime interval, millisecondsPerIteration int) -- interval --
AS $$
declare
i int;
startTime TIMESTAMP;
endTime TIMESTAMP;
-- dummy text;
iterations int;
millisecondsPerIteration int;
totalTime interval;
BEGIN
i := 1;
startTime := clock_timestamp();
WHILE ( i <= nbrOfIterations) LOOP
PERFORM /* Put your query here, replacing SELECT with PERFORM */
--------------------------------------------------------------------------------------------
--SELECT
-- YOUR QUERY HERE
-- ...
--------------------------------------------------------------------------------------------
i := i + 1; -- very important to increment loop counter, else one gets an infinite loop!!!
END LOOP;
endTime := clock_timestamp();
iterations := nbrOfIterations;
totalTime := (endTime - startTime);
millisecondsPerIteration := 1000 * (EXTRACT(EPOCH FROM endTime) - EXTRACT(EPOCH FROM startTime)) / iterations;
RETURN QUERY select iterations, totalTime, millisecondsPerIteration;
-- RETURNS TABLE (iterations int, totalTime interval, secondsPerIteration int) -- interval --
-- RETURN NEXT;
END;
$$ language plpgsql;
--------------------------------------------------------------------------------------------
To call this function, just use:
SELECT * from mySchema.fn__myFunction(1000) as ourTableResult;

Related

Commit changes after each iteration inside a function postgres

I am trying to commit the changes after each iteration inside this function to be able to save my progress in case of stopping the function in the middle of work. what should I do?
CREATE OR REPLACE FUNCTION delete_in_loop()
RETURNS INTEGER AS $$
DECLARE
counter INTEGER = 0 ;
i INTEGER = 0 ;
BEGIN
i = (select COUNT("ID") from "AwsSesNotification"
where "UTADateCreatedOn" < (now() - interval '3 month'))/1000 + 1 ;
LOOP
EXIT WHEN counter > i ;
counter = counter+1;
delete from "AwsSesNotification"
where "ID" in(
select "ID" from "AwsSesNotification"
where "UTADateCreatedOn" < (now() - interval '3 month')
limit 1000
);
RAISE NOTICE 'Counter: %', counter;
RAISE NOTICE 'From: %', i;
PERFORM pg_sleep(2);
END LOOP;
return counter;
END;
$$ LANGUAGE plpgsql;
This is known as AUTONOMOUS TRANSACTIONS and is unfortunately not supported in PostgreSQL.
The only way I know to achieve similar effects is by using dblink(), essentially writing to another database which has a separate transaction context. Much slower than normal writing.
Best regards,
Bjarni

Postgres: timing the runtime of a function

I want to update an audit table that stores the duration of a function/stored proc,
so far I have
drop table if exists tmp_interval_test;
create table tmp_interval_test (
id serial primary key,
duration interval
);
drop function if exists tmp_interval;
create or replace function tmp_interval()
returns void as
$body$
declare
sleep int;
start_time timestamp;
end_time timestamp;
diff interval;
begin
start_time := now();
sleep := floor(random() * 10 + 1)::int;
-- actual code goes here
perform pg_sleep(sleep);
end_time := now();
diff := age(end_time, start_time);
insert into tmp_interval_test (duration) values (diff);
end;
$body$
language 'plpgsql' volatile;
However, when I test this function, the duration shows
id|duration|
--|--------|
1|00:00:00|
How do I correctly insert the duration into my table?
The now() functions returns transaction time - it is same inside one transaction. So 0 is correct result. You should to use different functions, that returns real time - Use clock_timestamp() function instead.
On second hand, if you want to collect times of functions, you can use a buildin functionality in Postgres (if has superuser rights). Activate tracking functions. Then you can see what you need in system table pg_stat_user_function.
See SO regarding now()
Updated function and used clock_timestamp() instead of now(), e.g.,
start_time := clock_timestamp();

Using parameters passed to stored procedure in query

I have a sql query where I want to extract records older than 'X' number of days, here for eg its 7 days:
SELECT * FROM BOOKMARK.MONITORING_TABLE WHERE inserteddatetime < (now() - '7 day'::interval);
I have to execute this query through a stored procedure passing in the configurable 'X' no of days as arguments.
The procedure is as below:
CREATE OR REPLACE FUNCTION DELETE_REDUNDANT_RECORDS_STORED_PROCEDURE(days int)
RETURNS void AS
$func$
DECLARE
rec_old RECORD;
cursor_data CURSOR FOR
SELECT * FROM BOOKMARK.MONITORING_TABLE WHERE inserteddatetime < now() - '$1 day'::interval;
BEGIN
OPEN cursor_data;
// business logic for the procedure
CLOSE cursor_data;
END;
$func$
LANGUAGE plpgsql;
This doesn't work as I am not able to use the placeholder for days in my query. How do we use the arguments passed to my query in this case.
To create an interval based on an integer, make_interval() is much easier to use than casting to an interval type.
Additional I wouldn't use a cursor, but a FOR loop based on a SELECT statement (maybe using make_interval(days => $1) works in the cursor declaration as well)
CREATE OR REPLACE FUNCTION DELETE_REDUNDANT_RECORDS_STORED_PROCEDURE(days int)
RETURNS void AS
$func$
DECLARE
rec_old record;
BEGIN
for rec_old in SELECT *
FROM BOOKMARK.MONITORING_TABLE
WHERE inserteddatetime < now() - make_interval(days => $1)
loop
raise notice 'records %', rec_old;
end loop;
END;
$func$
LANGUAGE plpgsql;

ERROR: query has no destination for result data: Postgresql

I am using Postgresql11 and a function that works well in a single run fails when I add a LOOP statement with
"ERROR: query has no destination for result data HINT: If you want to
discard the results of a SELECT, use PERFORM instead."
The function has VOID as return value, selects data from a source table into a temp table, calculates some data and inserts the result into a target table. The temp table is then dropped and the function ends. I would like to repeat this procedure in defined intervals and have included a LOOP statement. With LOOP it does not insert into the target table and does not actually loop at all.
create function transfer_cs_regular_loop(trading_pair character varying) returns void
language plpgsql
as
$$
DECLARE
first_open decimal;
first_price decimal;
last_close decimal;
last_price decimal;
highest_price decimal;
lowest_price decimal;
trade_volume decimal;
n_trades int;
start_time bigint;
last_entry bigint;
counter int := 0;
time_frame int := 10;
BEGIN
WHILE counter < 100 LOOP
SELECT max(token_trades.trade_time) INTO last_entry FROM token_trades WHERE token_trades.trade_symbol = trading_pair;
RAISE NOTICE 'Latest Entry: %', last_entry;
start_time = last_entry - (60 * 1000);
RAISE NOTICE 'Start Time: %', start_time;
CREATE TEMP TABLE temp_table AS
SELECT * FROM token_trades where trade_symbol = trading_pair and trade_time > start_time;
SELECT temp_table.trade_time,temp_table.trade_price INTO first_open, first_price FROM temp_table ORDER BY temp_table.trade_time ASC FETCH FIRST 1 ROW ONLY;
SELECT temp_table.trade_time,temp_table.trade_price INTO last_close, last_price FROM temp_table ORDER BY temp_table.trade_time DESC FETCH FIRST 1 ROW ONLY;
SELECT max(temp_table.trade_price) INTO highest_price FROM temp_table;
SELECT min(temp_table.trade_price) INTO lowest_price FROM temp_table;
SELECT INTO trade_volume sum(temp_table.trade_quantity) FROM temp_table;
SELECT INTO n_trades count(*) FROM temp_table;
INSERT INTO candlestick_data_5min_test(open, high, low, close, open_time, close_time, volume, number_trades, trading_pair) VALUES (first_price, highest_price, lowest_price, last_price, first_open, last_close, trade_volume, n_trades, trading_pair);
DROP TABLE temp_table;
counter := counter + 1;
SELECT pg_sleep(time_frame);
RAISE NOTICE '**************************Counter: %', counter;
END LOOP;
END;
$$;
The error refers to the last SELECT statement in the function. If there is a SELECT without INTO it will always return results. When there's no LOOP this result will be used as the return value of the function (even if it is void).
When you add a LOOP there can't be any SELECT without INTO inside the loop because a single return value would be needed and now there will be many. In this case you need to use PERFORM which does exactly the same thing as a SELECT but discards the results.
Therefore change the last SELECT into a PERFORM and the error will go away:
PERFORM pg_sleep(time_frame);

Convert time to seconds in PostgreSQL

I have a time value 04:30:25 that I want to convert to seconds.
Is there any dedicated function to do this?
I know that we can extract hours, minutes and seconds, then calculate the seconds.
SELECT EXTRACT(hour FROM t)*60*60
+ EXTRACT(minutes FROM t)*60
+ EXTRACT(seconds FROM t)
FROM test;
But I want some other way...
Have you tried using:
SELECT EXTRACT(EPOCH FROM INTERVAL '04:30:25');
If that doesn't work you could try to prefix your time value with '1970-01-01' and try:
SELECT EXTRACT(EPOCH FROM TIMESTAMP '1970-01-01 04:30:25');
Not tested but it seems these are your only options. Probably.
You may skip epoch or interval, ie:
SELECT EXTRACT(EPOCH FROM column ) from table
Perhaps you can make it a function (just a quick setup, please review and change as needed)?
CREATE OR REPLACE FUNCTION to_seconds(t text)
RETURNS integer AS
$BODY$
DECLARE
hs INTEGER;
ms INTEGER;
s INTEGER;
BEGIN
SELECT (EXTRACT( HOUR FROM t::time) * 60*60) INTO hs;
SELECT (EXTRACT (MINUTES FROM t::time) * 60) INTO ms;
SELECT (EXTRACT (SECONDS from t::time)) INTO s;
SELECT (hs + ms + s) INTO s;
RETURN s;
END;
$BODY$
LANGUAGE 'plpgsql';
Then just use it in your queries:
SELECT to_seconds('04:30:25');
Returns:
16225
If you want to emulate MySQL's time_to_sec function, you could use a function like this:
CREATE OR REPLACE FUNCTION time_to_sec(t text)
RETURNS integer AS
$BODY$
DECLARE
s INTEGER;
BEGIN
SELECT (EXTRACT (EPOCH FROM t::interval)) INTO s;
RETURN s;
END;
$BODY$
LANGUAGE 'plpgsql';
It has the advantage that it will work with PostgreSQL intervals (ie: more than 24-hour periods), which would break the to_seconds function in the accepted answer.
As a simplified approach to #hdiogenes solution, just use this in the query:
SELECT EXTRACT (EPOCH FROM '04:30:25'::time)
from_seconds also to convert back
CREATE OR REPLACE FUNCTION from_seconds(t integer)
RETURNS time AS
$BODY$
DECLARE
h INTEGER;
m INTEGER;
s INTEGER;
rv TIME;
BEGIN
SELECT t / 3600 INTO h;
SELECT t % 3600 / 60 INTO m;
SELECT t % 60 INTO s;
SELECT (h::text || ':' || m::text || ':' || s::text)::time INTO rv;
RETURN rv;
END;
$BODY$
LANGUAGE 'plpgsql';