tuple concurrently updated when creating functions in postgresql / PL/pgSQL - postgresql

When initializing my process, it runs the PL/pgSQL statement below creating two functions. However, every time I create multiple processes simultaneously as part of an end-to-end test, parallel execution of this statement leads to a tuple concurrently updated error that I can't seem to get around. Any help would be much appreciated.
CREATE OR REPLACE FUNCTION
count_rows(schema text, tablename text) returns integer
AS
$body$
DECLARE
result integer;
query varchar;
BEGIN
query := 'SELECT count(1) FROM "' || schema || '"."' || tablename || '"';
execute query into result;
return result;
END;
$body$
LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION
delete_if_empty(schema text, tablename text) RETURNS INTEGER
AS
$$
DECLARE
result integer;
query varchar;
BEGIN
query := 'SELECT COUNT(*) FROM "' || schema || '"."' || tablename || '"';
execute query into result;
IF result = 0 THEN
EXECUTE 'DROP TABLE "' || schema || '"."' || tablename || '" CASCADE;';
EXECUTE 'NOTIFY "' || schema || '", ''DESTROY_TABLE:' || tablename || ''';';
RETURN 1;
END IF;
RETURN 0;
END;
$$
LANGUAGE plpgsql;
SELECT version()

As described here, postgres doesn't currently allow you to use CREATE FUNCTION concurrently:
It'd be necessary to add
some kind of locking scheme if you want to avoid "tuple concurrently
updated" errors. This is not really any different from the situation
where two transactions both want to update the same row in a user table:
unless the application takes extra steps to serialize the updates, you're
going to get "tuple concurrently updated" errors.
We do have such locking for DDL on tables/indexes, but the theory in the
past has been that it's not worth the trouble for objects represented by
single catalog rows, such as functions or roles.
A solution to this is to ensure that no two transaction try to do the CREATE FUNCTION at the same time.
You can use posgres advisory locks for that.
A good introduction to advisory locks can be found here: https://vladmihalcea.com/how-do-postgresql-advisory-locks-work/
For example, you can use:
BEGIN; -- start of transaction
SELECT pg_advisory_xact_lock(2142616474639426746); -- random 64-bit signed ('bigint') lock number
CREATE OR REPLACE FUNCTION myfunction ...
COMMIT;
This takes a transaction-level exclusive advisory lock, so that no two concurrent transaction can run create the function at the same time. At the end of the transaction, the lock is automatically released.

Related

EXECUTE fail does not proceed next

I have created a function remove_tables which is as shown below
CREATE OR REPLACE FUNCTION remove_tables (integer)
RETURNS void AS
$BODY$DECLARE
_id ALIAS FOR $1;
BEGIN
EXECUTE 'DROP TABLE something_' || _id || ' CASCADE';
DELETE FROM lot WHERE id = _id ;
END;$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION delete_tables(integer)
OWNER TO postgres;
My problem is, EXECUTE may failed but i still want to proceed with DELETE statement (DELETE FROM lot WHERE id = _id ;)
I try use PERFORM 'DROP TABLE something_' || _id || ' CASCADE'; but does not give what i want
Is there anything i can do to EXECUTE one statement regardless of success or failure, and still proceed with other statement
I have tried the exception trap pointed out by #a_horse_with_no_name and #Onots, using exception trap working good
I also tried the other method mentioned by #Onots to check table exists or not, also working good
due to i am not sure about trapping exception will have any impact on performance, I have used the code which check for table exists before dropping it
CREATE OR REPLACE FUNCTION remove_tables (integer)
RETURNS void AS
$BODY$DECLARE
_id ALIAS FOR $1;
BEGIN
IF EXISTS(SELECT * FROM information_schema.tables WHERE table_name = something_' || _id) THEN
EXECUTE 'DROP TABLE something_' || _id || ' CASCADE';
ENDIF;
DELETE FROM lot WHERE id = _id ;
END;$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION delete_tables(integer)
OWNER TO postgres;
You need to put exception handling around the block you expect to fail.
CREATE OR REPLACE FUNCTION remove_tables (integer)
RETURNS void AS
$BODY$DECLARE
_id ALIAS FOR $1;
BEGIN
BEGIN
EXECUTE 'DROP TABLE something_' || _id || ' CASCADE';
EXCEPTION WHEN read_only_sql_transaction THEN
-- Do nothing
END;
DELETE FROM lot WHERE id = _id ;
END;$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION delete_tables(integer)
OWNER TO postgres;
See the documentation at http://www.postgresql.org/docs/current/static/plpgsql-control-structures.html (40.6.6. Trapping Errors).
See http://www.postgresql.org/docs/9.1/interactive/errcodes-appendix.html for exception codes.

SELECTing commands into a temp table to EXECUTE later in PostgreSQL

For some fancy database maintenance for my developer database I'd like to be able to use queries to generate commands to alter the database. The thing is: I'm a complete greenhorn to PostgreSQL. I've made my attempt but have failed colorfully.
So in the end, I would like to have a table with a single column and each row would be a command (or group of commands, depending on the case) that I would think would look something like this...
DO $$
DECLARE
command_entry RECORD;
BEGIN
FOR command_entry IN SELECT * FROM list_of_commands
LOOP
EXECUTE command_entry;
END LOOP;
END;
$$;
Where the table list_of_commands could be populated with something like the following (which in this example would remove all tables from the public schema)...
CREATE TEMP TABLE list_of_commands AS
SELECT 'drop table if exists "' || tablename || '" cascade;'
FROM pg_tables
WHERE schemaname = 'public';
However, with this I get the following error...
ERROR: syntax error at or near ""drop table if exists ""dummy_table"" cascade;""
LINE 1: ("drop table if exists ""dummy_table"" cascade;")
I assume this is a matter of escaping characters, but I'm not entirely sure how to fit that into either A) the population of the table or B) the execution of each row. Does anyone know what I could do to achieve the desired result?
The command_entry variable is of type record while the EXECUTE command expects a string. What is apparently happening is that PostgreSQL turns the record into a double-quoted string, but that messes up your command. Also, your temp table does not use a column name, making things a bit awkward to work with (the column name becomes ?column?), so change both as follows:
CREATE TEMP TABLE list_of_commands AS
SELECT 'drop table if exists public.' || quote_ident(tablename) || ' cascade' AS cmd
FROM pg_tables
WHERE schemaname = 'public';
DO $$
DECLARE
command_entry varchar;
BEGIN
FOR command_entry IN SELECT cmd FROM list_of_commands
LOOP
EXECUTE command_entry;
END LOOP;
END;
$$;
But seeing that you do all of this at session level (temp table, anonymous code block), why not write a stored procedure that performs all of this housekeeping when you are ready to do spring cleaning?
CREATE FUNCTION cleanup() RETURNS void AS $$
BEGIN
FOR tbl IN SELECT tablename FROM pg_tables WHERE schemaname = 'public'
LOOP
EXECUTE 'DROP TABLE IF EXISTS ' || quote_ident(tbl) || ' CASCADE';
END LOOP;
-- More housekeeping jobs
END;
$$ LANGUAGE plpgsql;
This saves a lot of typing: SELECT cleanup();. Any other housekeeping jobs you have you simply add to the stored procedure.
I had trouble with Patrick's answers, so here is an updated version for postgreSQL 10.
CREATE FUNCTION droptables(sn varchar) RETURNS void AS $$
DECLARE
tbl varchar;
BEGIN
FOR tbl IN SELECT tablename FROM pg_tables WHERE schemaname = sn
LOOP
EXECUTE 'DROP TABLE IF EXISTS ' || quote_ident(tbl) || ' CASCADE';
END LOOP;
END;
$$ LANGUAGE plpgsql;
And then "SELECT droptables('public');".

PL/pgSQL Looping through multiple schema, tables and rows

I have a database with multiple identical schemas. There is a number of tables all named 'tran_...' in each schema. I want to loop through all 'tran_' tables in all schemas and pull out records that fall within a specific date range. This is the code I have so far:
CREATE OR REPLACE FUNCTION public."configChanges"(starttime timestamp, endtime timestamp)
RETURNS SETOF character varying AS
$BODY$DECLARE
tbl_row RECORD;
tbl_name VARCHAR(50);
tran_row RECORD;
out_record VARCHAR(200);
BEGIN
FOR tbl_row IN
SELECT * FROM pg_tables WHERE schemaname LIKE 'ivr%' AND tablename LIKE 'tran_%'
LOOP
tbl_name := tbl_row.schemaname || '.' || tbl_row.tablename;
FOR tran_row IN
SELECT * FROM tbl_name
WHERE ch_edit_date >= starttime AND ch_edit_date <= endtime
LOOP
out_record := tbl_name || ' ' || tran_row.ch_field_name;
RETURN NEXT out_record;
END LOOP;
END LOOP;
RETURN;
END;
$BODY$
LANGUAGE plpgsql;
When I attempt to run this, I get:
ERROR: relation "tbl_name" does not exist
LINE 1: SELECT * FROM tbl_name WHERE ch_edit_date >= starttime AND c...
#Pavel already provided a fix for your basic error.
However, since your tbl_name is actually schema-qualified (two separate identifiers in : schema.table), it cannot be escaped as a whole with %I in format(). You have to escape each identifier individually.
Aside from that, I suggest a different approach. The outer loop is necessary, but the inner loop can be replaced with a simpler and more efficient set-based approach:
CREATE OR REPLACE FUNCTION public.config_changes(_start timestamp, _end timestamp)
RETURNS SETOF text AS
$func$
DECLARE
_tbl text;
BEGIN
FOR _tbl IN
SELECT quote_ident(schemaname) || '.' || quote_ident(tablename)
FROM pg_tables
WHERE schemaname LIKE 'ivr%'
AND tablename LIKE 'tran_%'
LOOP
RETURN QUERY EXECUTE format (
$$
SELECT %1$L || ' ' || ch_field_name
FROM %1$s
WHERE ch_edit_date BETWEEN $1 AND $2
$$, _tbl
)
USING _start, _end;
END LOOP;
RETURN;
END
$func$ LANGUAGE plpgsql;
You have to use dynamic SQL to parametrize identifiers (or code), like #Pavel already told you. With RETURN QUERY EXECUTE you can return the result of a dynamic query directly. Examples:
Return SETOF rows from PostgreSQL function
Refactor a PL/pgSQL function to return the output of various SELECT queries
Remember that identifiers have to be treated as unsafe user input in dynamic SQL and must always be sanitized to avoid syntax errors and SQL injection:
Table name as a PostgreSQL function parameter
Note how I escape table and schema separately:
quote_ident(schemaname) || '.' || quote_ident(tablename)
Consequently I just use %s to insert the already escaped table name in the later query. And %L to escape it a string literal for output.
I like to prepend parameter and variable names with _ to avoid naming conflicts with column names. No other special meaning.
There is a slight difference compared to your original function. This one returns an escaped identifier (double-quoted only where necessary) as table name, e.g.:
"WeIRD name"
instead of
WeIRD name
Much simpler yet
If possible, use inheritance to obviate the need for above function altogether. Complete example:
Select (retrieve) all records from multiple schemas using Postgres
You cannot use a plpgsql variable as SQL table name or SQL column name. In this case you have to use dynamic SQL:
FOR tran_row IN
EXECUTE format('SELECT * FROM %I
WHERE ch_edit_date >= starttime AND ch_edit_date <= endtime', tbl_name)
LOOP
out_record := tbl_name || ' ' || tran_row.ch_field_name;
RETURN NEXT out_record;
END LOOP;

Stored procedure which returns table and being returned table gathers 1 rows from 9000 table

I need a stored function which returns table and being returned table gathers 1 rows from all 9000 tables.
Prerequisites:
There are same structure 9000 tables.
Table name are subsquent, like X1000 X1001 .... X9999.
Issue query to all table and get 1 rows from all table.
Gather all rows and return as 1 table.
Envirnment are PostgreSQL 9.3 and PL/Pgsql
I hope you will give me some easy example.
I think I can not understand how to deal variable value.
I made myself, however it didn't work.
create or replace function ListOfInc
(ddate int) returns table(col1 int, col2 dec) as $$
declare
numofrow integer;
begin
for i in 1000..9999 loop
numofrow := execute 'select count(*) from x' || i || ';';
if numofrow > 0 then
return query execute 'select ' || i || ' Trunc(100 * (CAST(Lag(adjust) over (order by'
|| %1 || ') as dec) / CAST(adjust as dec) - 1) ,2) from x' || i || ' order by ' || %1 || ' desc limit 1;';
end if;
end loop;
end;
$$ language plpgsql;
This is an ideal job for PostgreSQL's inheritance features.
If you want to avoid inheritance, you'll need to build a union all of all the tables using dynamic SQL, or loop over each table using RETURN QUERY EXECUTE ....
Personally I recommend consolodating all the data into one table. Create views to emulate the old structure, so your apps don't notice the change.

postgresql: USING CURSOR for extracting data from one database and inserting them to another

here is another algorithm using cursor but i'm having a hard time fixing its error ...
CREATE OR REPLACE FUNCTION extractstudent()
RETURNS VOID AS
$BODY$
DECLARE
studcur SCROLL cursor FOR SELECT fname, lname, mname, address FROM student;
BEGIN
open studcur;
Loop
--fetching 1 row at a time
FETCH First FROM studcur;
--every row fetched is being inserted to another database on the local site
--myconT is the name of the connection to the other database in the local site
execute 'SELECT * from dblink_exec(''myconT'', ''insert into temp_student values(studcur)'')';
--move to the next row and execute again
move next from studcur;
--exit when the row content is already empty
exit when studcur is null;
end loop;
close studcur;
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION extractstudent() OWNER TO postgres;
You rarely need to explicitly use cursors in postgresql or pl/pgsql. What you've written looks suspiciously like a SQL Server cursor loop construct, and you don't need to do that. Also, you can use "PERFORM" instead of "EXECUTE" to run a query and discard the results: this will avoid re-parsing the query each time (although it can't avoid dblink parsing the query each time).
You can do something more like this:
DECLARE
rec student%rowtype;
BEGIN
FOR rec IN SELECT * FROM student
LOOP
PERFORM dblink_exec('myconT',
'insert into temp_student values ('
|| quote_nullable(rec.fname) || ','
|| quote_nullable(rec.lname) || ','
|| quote_nullable(rec.mname) || ','
|| quote_nullable(rec.address) || ')');
END LOOP;
END;
Why not try it by yourself , according the error, you can try to solve them step by step !