Please can someone help on how to do this.
I have a log table as below where i am deleting all tables beyond 3 months.
Now i want add another step to clear the log table as well when the above step has been actioned. Example: drop the table if > 90 days then delete the associated rows from the log table too.
thanks
ddl_date ddl_tag ID object_name
2022-07-07 16:40:06 CREATE FOREIGN 1 raj.auth
2022-02-07 17:14:33 CREATE TABLE 6 john.plots_source
2022-03-07 17:14:33 CREATE TABLE 7 john.plots1
2022-04-07 17:14:33 CREATE TABLE 8 johnb.plots_pkey2
2022-05-07 17:14:33 CREATE TABLE 9 johna.plots_address3
DO $$
DECLARE
r RECORD;
BEGIN
FOR r IN
(
SELECT id, object_name from user_monitor.ddl_history WHERE ddl_date < NOW() - INTERVAL '90 days'
)
LOOP
EXECUTE 'DROP TABLE IF EXISTS ' || r.object_name || ' CASCADE';
END LOOP;
delete from user_monitor.ddl_history where not exists (select tablename from pg_catalog.pg_tables b where r.object_name = b.tablename);
commit;
END $$ ;
SQL Error [42601]: ERROR: query has no destination for result data
Hint: If you want to discard the results of a SELECT, use PERFORM
instead. Where: PL/pgSQL function inline_code_block line 12 at SQL
statement
You're referencing r.object_name outside of the LOOP.
Try pulling the DELETE statement inside the LOOP so that "r" is a valid reference that can cycle among all cursor values within the execution context.
Related
I was attempting an INSERT INTO.... ( SELECT... ) (inserting a batch of rows from SELECT... subquery), onto the same table in my database. For the most part it was working, however, I did see a "Deadlock" exception logged every now and then. Does it make sense to do this or is there a way to avoid a deadlock scenario? On a high-level, my queries both resemble this structure:
CREATE OR REPLACE PROCEDURE myConcurrentProc() LANGUAGE plpgsql
AS $procedure$
DECLARE
BEGIN
LOOP
EXIT WHEN row_count = 0
WITH cte AS (SELECT *
FROM TableA tbla
WHERE EXISTS (SELECT 1 FROM TableB tblb WHERE tblb.id = tbla.id)
INSERT INTO concurrent_table (SELECT id FROM cte);
COMMIT;
UPDATE log_tbl
SET status = 'FINISHED',
WHERE job_name = 'tblA_and_B_job';
END LOOP;
END
$procedure$;
And the other script that runs in parallel and INSERTS... also to the same table is also basically:
CREATE OR REPLACE PROCEDURE myConcurrentProc() LANGUAGE plpgsql
AS $procedure$
DECLARE
BEGIN
LOOP
EXIT WHEN row_count = 0
WITH cte AS (SELECT *
FROM TableC c
WHERE EXISTS (SELECT 1 FROM TableD d WHERE d.id = tblc.id)
INSERT INTO concurrent_table (SELECT id FROM cte);
COMMIT;
UPDATE log_tbl
SET status = 'FINISHED',
WHERE job_name = 'tbl_C_and_D_job';
END LOOP;
END
$procedure$;
So you can see I'm querying two different tables in each script, however inserting into the same some_table. I also have the UPDATE... statement that writes to a log table so I suppose that could also cause issues. Is there any way to use BEGIN... END here and COMMIT to avoid any deadlock/concurrency issues or should I just create a 2nd table to hold the "tbl_C_and_D_job" data?
I want to add a new column to all tables with table name pattern table_<>_details.
I use this query :
select 'alter table ' || table_name || ' ADD COLUMN CREATED TIMESTAMP;'
from information_schema.tables
where table_name like 'table_%_details';
to generate the DDL queries which looks like :
alter table table_1_details ADD COLUMN CREATED TIMESTAMP;
alter table table_2_details ADD COLUMN CREATED TIMESTAMP;
alter table table_3_details ADD COLUMN CREATED TIMESTAMP;
alter table table_4_details ADD COLUMN CREATED TIMESTAMP;
alter table table_5_details ADD COLUMN CREATED TIMESTAMP;
alter table table_6_details ADD COLUMN CREATED TIMESTAMP;
alter table table_7_details ADD COLUMN CREATED TIMESTAMP;
I tried to loop through these records using the following script :
do $$ declare c_query cursor for
select
'alter table ' || table_name || ' ADD COLUMN CREATED TIMESTAMP;'
from
information_schema.tables
where
table_name like 'table_%_details';
begin
for rec in c_query loop
execute rec;
end loop;
close c_query;
end $$
I have tried to fine tune this script but with no success, I'm getting the following error:
SQL Error [42601]: ERROR: syntax error at or near ""alter table table_1_details ADD COLUMN CREATED TIMESTAMP;""
Where: PL/pgSQL function inline_code_block line 13 at EXECUTE statement
my question is how to modify this scrip to loop through all these results and apply the DDL to database , note (I do not want to create functions).
please any Ideas will be appreciated.
Just loop over the resultset of infomation_schema.tables and then use EXECUTE with your concatenated ALTER TABLE statements
DO $$
DECLARE
row record;
BEGIN
FOR row IN SELECT table_name FROM information_schema.tables
WHERE table_name LIKE 'table_%_details' LOOP
EXECUTE 'ALTER TABLE ' || row.table_name || ' ADD COLUMN CREATED TIMESTAMP;';
END LOOP;
END;
$$;
EDIT: Alternatively you can use FORMAT to concatenate your strings instead of using ||, as pointed out by #a_horse_with_no_name
EXECUTE FORMAT('ALTER TABLE %I ADD COLUMN CREATED TIMESTAMP;',row.table_name);
Check this db<>fiddle
I have two stored functions. delete_item which deletes one item, logging it's success or failure in an actionlog table, it returns a 1 on errors and 0 when running successfully.
Secondly I have another function remove_expired that finds what to delete and loops through it calling delete_item.
All this is intended to be called using a simple bash script (hard requirement from operations, so calling like this is not up for discussion), and it have to give an error code when things doesn't work for their reporting tools.
We want all the deletions possible to succeed (we don't expect errors, but humans are still humans, and errors does happen), so if we want to delete 10 items and 1 fails, we still want the other 9 to be deleted.
Secondly we would really like the logs to be in the table actionlog both in the success and error case. I.e., we want that log to be complete.
Since plpgsql functions doesn't allow manual transaction management that seems to not be an option (unless I missed a way to circumvent this?).
The only way I've found so far to achieve this is to wrap scripts around it outside plpgsql, but we would very much like this to be possible in pure plpgsql so we can just give operations a pssql -C ... command and then they shouldn't be concerned with anything else.
SQL to reproduce the problem:
DROP FUNCTION IF EXISTS remove_expired(timestamp with time zone);
DROP FUNCTION IF EXISTS delete_item(integer);
DROP TABLE IF EXISTS actionlog;
DROP TABLE IF EXISTS evil;
DROP TABLE IF EXISTS test;
CREATE TABLE test (
id serial primary key not null,
t timestamp with time zone not null
);
CREATE TABLE evil (
test_id integer not null references test(id)
);
CREATE TABLE actionlog (
eventTime timestamp with time zone not null default now(),
message text not null
);
INSERT INTO test (actualTime, t)
VALUES ('2020-04-01T10:00:00+0200'),
('2020-04-01T10:15:00+0200'), -- Will not be deleable due to foreign key
('2020-04-01T10:30:00+0200')
;
INSERT INTO evil (test_id) SELECT id FROM test WHERE id = 2;
CREATE OR REPLACE FUNCTION remove_expired(timestamp with time zone)
RETURNS void
AS
$$
DECLARE
test_id int;
failure_count int = 0;
BEGIN
FOR test_id IN
SELECT id FROM test WHERE t < $1
LOOP
failure_count := delete_item(test_id) + failure_count;
END LOOP;
IF failure_count > 0 THEN
-- I want this to cause 'psql ... -c "SELECT * FROM remove_expred...' to exit with exit code != 0
RAISE 'There was one or more errors deleting. See the log for details';
END IF;
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION delete_item(integer)
RETURNS integer
AS
$$
BEGIN
DELETE FROM test WHERE id = $1;
INSERT INTO actionlog (message)
VALUES ('Deleted with ID: ' || $1);
RETURN 0;
EXCEPTION WHEN OTHERS THEN
INSERT INTO actionlog (message)
VALUES ('Error deleting ID: ' || $1 || '. The error was: ' || SQLERRM);
RETURN 1;
END
$$ LANGUAGE plpgsql;
Thanks in advance for any useful input
You can have something close to what your expect in PostgreSQL 11 or PostgreSQL 12 but only with procedures because as already said functions will always roll everything back in case of errors.
With:
DROP PROCEDURE IF EXISTS remove_expired(timestamp with time zone);
DROP PROCEDURE IF EXISTS delete_item(integer);
DROP FUNCTION IF EXISTS f_removed_expired;
DROP SEQUENCE IF EXISTS failure_count_seq;
DROP TABLE IF EXISTS actionlog;
DROP TABLE IF EXISTS evil;
DROP TABLE IF EXISTS test;
CREATE TABLE test (
id serial primary key not null,
t timestamp with time zone not null
);
CREATE TABLE evil (
test_id integer not null references test(id)
);
CREATE TABLE actionlog (
eventTime timestamp with time zone not null default now(),
message text not null
);
INSERT INTO test (t)
VALUES ('2020-04-01T10:00:00+0200'),
('2020-04-01T10:15:00+0200'), -- Will not be removed due to foreign key
('2020-04-01T10:30:00+0200')
;
select * from test where t < current_timestamp;
INSERT INTO evil (test_id) SELECT id FROM test WHERE id = 2;
CREATE SEQUENCE failure_count_seq MINVALUE 0;
SELECT SETVAL('failure_count_seq', 0, true);
CREATE OR REPLACE PROCEDURE remove_expired(timestamp with time zone)
AS
$$
DECLARE
test_id int;
failure_count int = 0;
return_code int;
BEGIN
FOR test_id IN
SELECT id FROM test WHERE t < $1
LOOP
call delete_item(test_id);
COMMIT;
END LOOP;
SELECT currval('failure_count_seq') INTO failure_count;
IF failure_count > 0 THEN
-- I want this to cause 'psql ... -c "SELECT * FROM remove_expred...' to exit with exit code != 0
RAISE 'There was one or more errors deleting. See the log for details';
END IF;
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE PROCEDURE delete_item(in integer)
AS
$$
DECLARE
forget_value int;
BEGIN
DELETE FROM test WHERE id = $1;
INSERT INTO actionlog (message)
VALUES ('Deleted with ID: ' || $1);
EXCEPTION WHEN OTHERS THEN
INSERT INTO actionlog (message)
VALUES ('Error deleting ID: ' || $1 || '. The error was: ' || SQLERRM);
COMMIT;
SELECT NEXTVAL('failure_count_seq') INTO forget_value;
END
$$ LANGUAGE plpgsql;
--
I get:
select * from test;
id | t
----+------------------------
1 | 2020-04-01 10:00:00+02
2 | 2020-04-01 10:15:00+02
3 | 2020-04-01 10:30:00+02
(3 rows)
select current_timestamp;
current_timestamp
-------------------------------
2020-04-01 16:52:26.171975+02
(1 row)
call remove_expired(current_timestamp);
psql:test.sql:80: ERROR: There was one or more errors deleting. See the log for details
CONTEXT: PL/pgSQL function remove_expired(timestamp with time zone) line 17 at RAISE
select currval('failure_count_seq');
currval
---------
1
(1 row)
select * from test;
id | t
----+------------------------
2 | 2020-04-01 10:15:00+02
(1 row)
select * from actionlog;
eventtime | message
-------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------
2020-04-01 16:52:26.172173+02 | Deleted with ID: 1
2020-04-01 16:52:26.179794+02 | Error deleting ID: 2. The error was: update or delete on table "test" violates foreign key constraint "evil_test_id_fkey" on table "evil"
2020-04-01 16:52:26.196503+02 | Deleted with ID: 3
(3 rows)
I use a sequence to record the number of failures: you can use this sequence to test for failures and return the right return code.
I need to remove selected tables from the Postgres SQL. It would better to use like or where clause.
Like I have
TABLE_A
TABLE_B
TABLE_C
-
-
-
TABLE_N
I need to delete
TABLE_A to TABLE_X
Can be done with a single command, which is faster - in case this is a recurring task.
Add IF EXISTS if the existence of any tables is uncertain. This way we save an extra trip to the system catalogs (information_schema.tables or pg_catalog.pg_tables).
And you may want to add CASCADE:
DO
$do$
BEGIN
-- child safety device: quote RAISE instead of EXECUTE to prime the bomb
-- EXECUTE (
RAISE NOTICE '%', (
SELECT 'DROP TABLE IF EXISTS'
|| string_agg('table_' || chr(ascii('a') + g) , ', ')
|| ' CASCADE;'
FROM generate_series(0,13) g
);
END
$do$;
Generates a command of the form:
DROP TABLE IF EXISTS table_a, table_b, ... , table_n CASCADE;
Using generate_series() to generate requested table names. More here:
How to drop many (but not all) tables in one fell swoop?
How to check if a table exists in a given schema
DO
$$
DECLARE
r RECORD;
BEGIN
FOR r IN SELECT oid::REGCLASS table_name
FROM pg_class
WHERE relname <= 'table_x'
AND relkind = 'r'
LOOP
EXECUTE 'DROP TABLE' || r.table_name;
END LOOP;
END;
$$ LANGUAGE plpgsql;
I am currently using postgres 8.3. I have created a table that acts as a dirty flag table for members that exist in another table. I have applied triggers after insert or update on the members table that will insert/update a record on the modifications table with a value of true. The trigger seems to work, however I am noticing that something is flipping the boolean is_modified value. I have no idea how to go about trying to isolate what could be flipping it.
Trigger function:
BEGIN;
CREATE OR REPLACE FUNCTION set_member_as_modified() RETURNS TRIGGER AS $set_member_as_modified$
BEGIN
LOOP
-- first try to update the key
UPDATE member_modification SET is_modified = TRUE, updated = current_timestamp WHERE "memberID" = NEW."memberID";
IF FOUND THEN
RETURN NEW;
END IF;
--member doesn't exist in modification table, so insert them
-- if someone else inserts the same key conncurrently, raise a unique-key failure
BEGIN
INSERT INTO member_modification("memberID",is_modified,updated) VALUES(NEW."memberID", TRUE,current_timestamp);
RETURN NEW;
EXCEPTION WHEN unique_violation THEN
-- do nothing, and loop to try the update again
END;
END LOOP;
END;
$set_member_as_modified$ LANGUAGE plpgsql;
COMMIT;
CREATE TRIGGER set_member_as_modified AFTER INSERT OR UPDATE ON members FOR EACH ROW EXECUTE PROCEDURE set_member_as_modified();
Here is the sql I run and the results:
$CREATE TRIGGER set_member_as_modified AFTER INSERT OR UPDATE ON members FOR EACH ROW EXECUTE PROCEDURE set_member_as_modified();
Results:
UPDATE 1
bluesky=# select * from member_modification;
-[ RECORD 1 ]---+---------------------------
modification_id | 14
is_modified | t
updated | 2011-05-26 09:49:47.992241
memberID | 182346
bluesky=# select * from member_modification;
-[ RECORD 1 ]---+---------------------------
modification_id | 14
is_modified | f
updated | 2011-05-26 09:49:47.992241
memberID | 182346
As you can see something flipped the is_modified value. Is there anything in postgres I can use to determine what queries/processes are acting on this table?
Are you sure you've posted everything needed? The two queries on member_modification suggest that a separate query is being run in between, which sets is_modified back to false.
You could add an text[] field to member_modification, e.g. query_trace text[] not null default '{}', then and a before insert/update trigger on each row on that table which goes something like:
NEW.query_trace := NEW.query_trace || current_query();
If current_query() is not available in 8.3, see this:
http://www.postgresql.org/docs/8.3/static/monitoring-stats.html
SELECT pg_stat_get_backend_pid(s.backendid) AS procpid,
pg_stat_get_backend_activity(s.backendid) AS current_query
FROM (SELECT pg_stat_get_backend_idset() AS backendid) AS s;
You could then get the list of subsequent queries that affected it:
select query_trace[i] from generate_series(1, array_length(query_trace, 1)) as i