I have a Windows-based production system which has started spawning 4 sessions every minute but leaving them in an IDLE state. While I'm trying to track down the cause of this issue, I want to create a scheduled task which shall call my function every hour or so to drop all IDLE sessions.
So far, I've written the following:
CREATE OR REPLACE FUNCTION drop_idle_session_connections()
RETURNS VOID AS $$
DECLARE
i_total INTEGER;
i_active INTEGER;
i_idle INTEGER;
retval BOOLEAN;
BEGIN
SELECT
(SELECT count(*) FROM pg_catalog.pg_stat_activity),
(SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE state = 'active'),
(SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE state IN ('idle', 'idle in transaction', 'idle in transaction (aborted)', 'disabled'))
INTO i_total, i_active, i_idle;
RAISE NOTICE 'DROPPING IDLE CONNECTIONS ...';
RAISE NOTICE ' BEFORE : Total: % Active: % Idle: %',
to_char(i_total, '9999'),
to_char(i_active, '9999'),
to_char(i_idle, '9999');
WITH inactive_connections AS (
SELECT
pid,
rank() over (partition by client_addr order by backend_start ASC) as rank
FROM
pg_stat_activity
WHERE
pid <> pg_backend_pid( )
AND
application_name !~ '(?:psql)|(?:pgAdmin.+)|(?:DataGrip.+)'
AND
datname = current_database()
AND
state IN ('idle', 'idle in transaction', 'idle in transaction (aborted)', 'disabled')
AND
current_timestamp - state_change > interval '5 minutes'
)
SELECT
pg_terminate_backend(pid)
INTO
retval
FROM
inactive_connections
WHERE
rank > 1;
SELECT
(SELECT count(*) FROM pg_catalog.pg_stat_activity),
(SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE state = 'active'),
(SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE state IN ('idle', 'idle in transaction', 'idle in transaction (aborted)', 'disabled'))
INTO i_total, i_active, i_idle;
RAISE NOTICE ' AFTER : Total: % Active: % Idle: %',
to_char(i_total, '9999'),
to_char(i_active, '9999'),
to_char(i_idle, '9999');
END;
$$ LANGUAGE plpgsql;
However, even if I SET ROLE postgres the main CTE doesn't close any connections at all.
What am I doing wrong here? If I run the CTE from psql, it works fine
Related
I have table1 which contains around 900k line of records, and table2 around 500k rows and 26 columns. sample table
I want to update table1 with total of unique combinations from table2
Tried with different type of counts, but still the performance is extremally slow. Is there any alternative options to improve the performance please?
query #1, the completion time for just 10 lines is 1 min 47 secs.
do $$
declare
rec record;
current_joins text;
current_result int;
begin for rec in (select joins from table1 where line<=10) loop
select rec.joins into current_joins;
execute format('select count(*) from (select 1 from table2 group by %1$s) as some_alias;', current_joins) into current_result;
update table1 set result = current_result where joins=current_joins;
end loop;
end $$;
query #2, the completion time for just 10 lines is 1 min 48 secs
do $$
declare
rec record;
current_joins text;
current_result int;
begin for rec in (select joins from table1 where line<=10) loop
select rec.joins into current_joins;
execute format('select count(*) from (select 1 from table2 group by %1$s having count(*)>=1) as some_alias;', current_joins) into current_result;
update table1 set result = current_result where joins=current_joins;
end loop;
end $$;
query #3, the completion time for just 10 lines is 1 min 52 secs
do $$
declare
rec record;
current_joins text;
current_result int;
begin for rec in (select joins from table1 where line<=10) loop
select rec.joins into current_joins;
execute format('select count(distinct (%1$s)) from table2', current_joins) into current_result;
update table1 set result = current_result where joins=current_joins;
end loop;
end $$;
query #4, the completion time for just 10 lines is 1 min 55 secs
do $$
declare
rec record;
current_joins text;
current_result int;
begin for rec in (select joins from table1 where line<=10) loop
select rec.joins into current_joins;
execute format('select count(*) from (select distinct (%1$s) from table2) as temp', current_joins) into current_result;
update table1 set result = current_result where joins=current_joins;
end loop;
end $$;
other option, tried with Microsoft Excel, the completion time for just 10 lines is <10 secs which is much faster than all the above sql codes, but still slow with 500k data
=LET(A,Data!$C$2:$AB$12,B,ROWS(A),C,FILTERXML("<A><B>"&SUBSTITUTE(C2,",","</B><B>")&"</B></A>","//B"),ROWS(UNIQUE(INDEX(A,SEQUENCE(B),TRANSPOSE(C)))))
I was running some tests to better understanding read commits for postgresql.
I have two transactions running in parallel:
-- transaction 1
begin;
select id from item order by id asc FETCH FIRST 500 ROWS ONLY;
select pg_sleep(10);
commit;
--transaction 2
begin;
select id from item order by id asc FETCH FIRST 500 ROWS ONLY;
commit;
The first transaction will select first 500 ids and then hold the id by sleeping 10s
The second transaction will in the mean while querying for first 500 rows in the table.
Based my understanding of read commits, first transaction will select 1 to 500 records and second transaction will select 501 to 1000 records.
But the actual result is that both two transactions select 1 to 500 records.
I will be really appreciated if someone can point out which part is wrong. Thanks
You are misinterpreting the meaning of read committed. It means that a transaction cannot see (select) updates that are not committed. Try the following:
create table read_create_test( id integer generated always as identity
, cola text
) ;
insert into read_create_test(cola)
select 'item-' || to_char(n,'fm000')
from generate_series(1,50) gs(n);
-- transaction 1
do $$
max_val integer;
begin
insert into read_create_test(cola)
select 'item2-' || to_char(n+100,'fm000')
from generate_series(1,50) gs(n);
select max(id)
into max_val
from read_create_test;
raise notice 'Transaction 1 Max id: %',max_val;
select pg_sleep(30); -- make sure 2nd transaction has time to start
commit;
end;
$$;
-- transaction 2 (run after transaction 1 begins but before it ends)
do $$
max_val integer;
begin
select max(id)
into max_val
from read_create_test;
raise notice 'Transaction 2 Max id: %',max_val;
end;
$$;
-- transaction 3 (run after transaction 1 ends)
do $$
max_val integer;
begin
select max(id)
into max_val
from read_create_test;
raise notice 'Transaction 3 Max id: %',max_val;
end;
$$;
Analyze the results keeping in mind that A transaction cannot see uncommitted DML.
I am trying to convert MSSQL queries to POSTGRES queries.
I am not able to execute the below query in Postgres
DO $$
BEGIN
IF EXISTS (SELECT ID FROM PROCESS WHERE ID = (SELECT MAX(ID) FROM PROCESS WHERE NAME = 'TRANSACTION')) THEN
SELECT * FROM MSG
WHERE msg_timestamp >= ( SELECT start_time FROM PROCESS WHERE NAME = 'TRANSACTION' AND STATUS = 'STARTED');
ELSE
SELECT * FROM MSG;
END IF;
END $$;
ERROR:
query has no destination for result data
HINT: If you want to discard the results of a SELECT, use PERFORM instead.
CONTEXT: PL/pgSQL function inline_code_block line 7 at SQL statement
FCMDBPOSTGRES=#
My corresponding MSSQL query is as below which works fine
IF EXISTS (SELECT ID FROM PROCESS WHERE ID = (SELECT MAX(ID) FROM PROCESS WHERE NAME = 'TRANSACTION'))
BEGIN
SELECT * FROM MSG
WHERE msg_timestamp >= ( SELECT start_time FROM PROCESS WHERE NAME = 'TRANSACTION' AND STATUS = 'STARTED')
END
ELSE
BEGIN
SELECT * FROM MSG
END
You do not need the do statement for this, just plain SQL:
SELECT * FROM MSG
WHERE
NOT EXISTS (SELECT ID FROM PROCESS WHERE ID = (SELECT MAX(ID) FROM PROCESS WHERE NAME = 'TRANSACTION'))
OR msg_timestamp >= (SELECT start_time FROM PROCESS WHERE NAME = 'TRANSACTION' AND STATUS = 'STARTED');
Or to be more verbose:
SELECT * FROM MSG
WHERE
CASE
WHEN EXISTS (SELECT ID FROM PROCESS WHERE ID = (SELECT MAX(ID) FROM PROCESS WHERE NAME = 'TRANSACTION'))
THEN msg_timestamp >= (SELECT start_time FROM PROCESS WHERE NAME = 'TRANSACTION' AND STATUS = 'STARTED')
ELSE TRUE
END;
Or to be more simple:
SELECT * FROM MSG
WHERE
msg_timestamp >= coalesce(
(SELECT start_time FROM PROCESS WHERE NAME = 'TRANSACTION' AND STATUS = 'STARTED'),
'-infinity');
BTW If I understand correctly
EXISTS (SELECT ID FROM PROCESS WHERE ID = (SELECT MAX(ID) FROM PROCESS WHERE NAME = 'TRANSACTION'))
could be simplified to
EXISTS (SELECT 1 FROM PROCESS WHERE NAME = 'TRANSACTION')
It cannot to work. Postgres doesn't support free queries in procedures or scripts. Every result of SQL statement should be saved to variable (or returned as result of table function). More DO statement has not any input output mechanism. You can write table function:
CREATE OR REPLACE FUNCTION fx()
RETURNS SETOF MSG AS $$
BEGIN
IF EXISTS (SELECT ID FROM PROCESS
WHERE ID = (SELECT MAX(ID) FROM PROCESS
WHERE NAME = 'TRANSACTION'))
THEN
RETURN QUERY SELECT * FROM MSG
WHERE msg_timestamp >= ( SELECT start_time FROM PROCESS
WHERE NAME = 'TRANSACTION' AND STATUS = 'STARTED');
ELSE
RETURN QUERY SELECT * FROM MSG;
END IF;
END
$$ LANGUAGE plpgsql;
SELECT * FROM fx();
Note: if you have experience of MSSQL procedures, then the best start for work with Postgres is an reading of documentation - https://www.postgresql.org/docs/current/plpgsql.html . Lot of things are very very different. The stored procedures in Postgres are similar to Oracle, but very far to MSSQL.
Postgres 9.3 x64 bit ,I am using a history table with user log.huge data insertion occurring on every second till now the data is is around 90 millions and still counting
I partitioned the table using date column for every month and I used a trigger where for each insert it will check weather for the particular month table is there or not, if there it will directly insert else it will try to create and insert the data. but the trigger taking soo much time to execute is there any way to improvise the performance
CREATE OR REPLACE FUNCTION history_master_part_trigger()
RETURNS TRIGGER AS $$
DECLARE
yr int;
mnth int;
tbname character varying;
sdate character varying;
edate character varying;
cnt int;
Begin
raise notice '%','something';
EXECUTE 'select EXTRACT(YEAR FROM DATE '''||NEW.hm_date||''')' INTO yr;
EXECUTE 'select EXTRACT(month FROM DATE '''||NEW.hm_date||''')' INTO mnth;
tbname := 'history_master_part_y'||yr||'m'||mnth;
sdate :=yr||'-'||mnth||'-01';
IF mnth = 12 THEN
edate :=yr+1||'-'||'01-01';
ELSE
edate :=yr||'-'||mnth+1||'-01';
END IF;
--raise notice 'table-----', tbname;
raise notice '%', sdate;
raise notice '%', edate;
raise notice '%',yr;
If(SELECT EXISTS (
SELECT 1
FROM pg_catalog.pg_class c
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public'
AND c.relname = tbname
AND c.relkind = 'r' -- only tables(?)
)) THEN
--INSERT INTO ||tbname|| VALUES ||NEW'.*)';
raise notice '%','inserting into'||tbname;
EXECUTE format('INSERT INTO ' || tbname || ' SELECT ($1).*')
USING NEW;
ELSE
EXECUTE 'CREATE TABLE '|| tbname||'(
CHECK ( hm_date >= DATE '''||sdate||''' AND hm_date < DATE '''||edate||''' )
) INHERITS (history_master_part)';
EXECUTE format('INSERT INTO ' || tbname || ' SELECT ($1).*')
USING NEW;
--RAISE EXCEPTION 'Caution Caution Caution Amit is getting angry';
END IF;
Return NULL;
End;
$$
LANGUAGE 'plpgsql'
CREATE OR REPLACE FUNCTION add_() RETURNS void AS
$BODY$
declare
foo int;
BEGIN
FOR i IN 1..50 LOOP
foo = i;
RAISE NOTICE 'combination_array(%)', foo ;
UPDATE table_1 set r_id = foo WHERE id = (select id from table_1 where r_id is null order by id limit 1);
END LOOP;
END; $BODY$ LANGUAGE 'plpgsql' ;
SELECT add_();
after this execution when ever i execute
UPDATE table_1
set r_id = foo
WHERE id = (select id from table_1 where r_id is null order by id limit 1);
END LOOP;
its going to be busy any one tell me how to clear the pull in pgsql
it's return all connection with ids
SELECT * from pg_stat_activity;
You can kill the specific connection by procpid though following query.
select pg_terminate_backend(procpid)
from pg_stat_activity
where datname = 'database-name'
You can send signal to this busy connected backend process.
To cancel a running query, send the SIGINT signal to the process running that command. To terminate a backend process cleanly, send SIGTERM to that process.
for exp :
pg93#db-172-16-3-150-> psql
psql (9.3.3)
Type "help" for help.
digoal=# select pg_sleep(1000000);
-- find the pid
-- ps -ewf|grep postgres
pg93 24872 23190 0 00:11 ? 00:00:00 postgres: postgres digoal [local] SELECT
-- send signal
pg93#db-172-16-3-150-> kill -s SIGINT 24872
-- then we see that feedback from psql.
ERROR: canceling statement due to user request