I am using Postgres 9.6
I wish to CREATE temp table that will drop itself at the end of the transaction. I have been looking through the documentation and am having a hard time locating the answer to this.
Quote from the manual
ON COMMIT
The behavior of temporary tables at the end of a transaction block can be controlled using ON COMMIT. The three options are:
...
DROP
The temporary table will be dropped at the end of the current transaction block.
(emphasis mine)
The "end of transaction block" is defined by a commit or a rollback.
Like this:
BEGIN;
CREATE TEMP TABLE my_temp(id) ON COMMIT DROP AS VALUES
('SOME_ID_1'),
('SOME_ID_2'),
('SOME_ID_3');
-- put your query that uses the temp-table here
SELECT * FROM nodes
JOIN my_temp ON nodes.id = my_temp.id;
END;
Related
I have Postgresql Function which has to INSERT about 1.5 million data into a table. What I want is I want to see the table getting populated with every one records insertion. Currently what is happening when I am trying with say about 1000 records, the get gets populated only after the complete function gets executed. If I stop the function half way through, no data gets populated. How can I make the record committed even if I stop after certain number of records have been inserted?
This can be done using dblink. I showed an example with one insert being committed you will need to add your while loop logic and commit every loop. You can http://www.postgresql.org/docs/9.3/static/contrib-dblink-connect.html
CREATE OR REPLACE FUNCTION log_the_dancing(ip_dance_entry text)
RETURNS INT AS
$BODY$
DECLARE
BEGIN
PERFORM dblink_connect('dblink_trans','dbname=sandbox port=5433 user=postgres');
PERFORM dblink('dblink_trans','INSERT INTO dance_log(dance_entry) SELECT ' || '''' || ip_dance_entry || '''');
PERFORM dblink('dblink_trans','COMMIT;');
PERFORM dblink_disconnect('dblink_trans');
RETURN 0;
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION log_the_dancing(ip_dance_entry text)
OWNER TO postgres;
BEGIN TRANSACTION;
select log_the_dancing('The Flamingo');
select log_the_dancing('Break Dance');
select log_the_dancing('Cha Cha');
ROLLBACK TRANSACTION;
--Show records committed even though we rolled back outer transaction
select *
from dance_log;
What you're asking for is generally called an autonomous transaction.
PostgreSQL does not support autonomous transactions at this time (9.4).
To properly support them it really needs stored procedures, not just the user-defined functions it currently supports. It's also very complicated to implement autonomous tx's in PostgreSQL for a variety of internal reasons related to its session and process model.
For now, use dblink as suggested by Bob.
If you have the flexibility to change from function to procedure, from PostgreSQL 12 onwards you can do internal commits if you use procedures instead of functions, invoked by CALL command. Therefore your function will be changed to a procedure and invoked with CALL command: e.g:
CREATE PROCEDURE transaction_test2()
LANGUAGE plpgsql
AS $$
DECLARE
r RECORD;
BEGIN
FOR r IN SELECT * FROM test2 ORDER BY x LOOP
INSERT INTO test1 (a) VALUES (r.x);
COMMIT;
END LOOP;
END;
$$;
CALL transaction_test2();
More details about transaction management regarding Postgres are available here: https://www.postgresql.org/docs/12/plpgsql-transactions.html
For Postgresql 9.5 or newer you can use dynamic background workers provided by pg_background extension. It creates autonomous transaction. Please, refer the github page of the extension. The sollution is better then db_link. There is a complete guide on Autonomous transaction support in PostgreSQL. There is a third way to start autonomous transaction in Postgres, but some patching neede. Please see Peter's Eisentraut patch proposal for OracleDB-style transactions.
I have 2 tables (1 staging table and 1 main operational table).
Both tables have the same structure.
For my solution,
I am using DB2Copy in program to insert 10000 records into staging table (4 seconds)
From the staging table, will move the data into main table using stored procedure (10 seconds)
However, it will lock the main table when running stored procedure.
I am suspecting because of the BEGIN and END which cause the stored procedure to act like a transaction.
I do not want the table to be locked when running stored procedure. (any suggestion?) Prefer: Stored procedure insert record by record into main table without transaction behavior.
Below is my code:
CREATE PROCEDURE SP_NAME ( )
LANGUAGE SQL
NOT DETERMINISTIC
CALLED ON NULL INPUT
EXTERNAL ACTION
OLD SAVEPOINT LEVEL
MODIFIES SQL DATA
INHERIT SPECIAL REGISTERS
BEGIN
--DECLARE TEMP VARIABLES
BEGIN
DECLARE MYCURSOR CURSOR WITH RETURN TO CALLER FOR
--SELECT STAGING TABLE
DECLARE CONTINUE HANDLER FOR NOT FOUND SET AT_END = 1;
OPEN MYCURSOR;
-- FETCH MYCURSOR INTO TEMP VARIABLES
WHILE AT_END = 0 DO
-- INSERT MAIN TABLE
-- FETCH MYCURSOR INTO TEMP VARIABLES
END WHILE;
CLOSE MYCURSOR;
END;
END;
My Environment
I have a program "A" which is trying to insert 10k records into main table (A lot of indexes and high volume of data) which takes 10 minutes ++.
About main operational table
High number of read but minimum updates and inserts at front end.
At back end, another program will frequently insert record into this table.
Only 1 instance of the back end program is allowed to run at one time
When you create the procedure, make sure your commitment-control setting is *NONE (a.k.a. autocommit). This should not lock your whole table
Adding the example
CREATE PROCEDURE userS.SP_TEST (
IN col_DATA Varchar(10) )
LANGUAGE SQL
SPECIFIC userS.SP_TEST
NOT DETERMINISTIC
MODIFIES SQL DATA
SET OPTION COMMIT = *NONE
BEGIN INSERT INTO userS.TABLE1 VALUES(col_DATA);
END
I have googled quite a bit, and I have fairly decent reading comprehension, but I don't understand if this script will work in multiple threads on my postgres/postgis box. Here is the code:
Do
$do$
DECLARE
x RECORD;
b int;
begin
create temp table geoms (id serial, geom geometry) on commit drop;
for x in select id,geom from asdf loop
truncate table geoms;
insert into geoms (geom) select someGeomfield from sometable where st_intersects(somegeomfield,x.geom);
----do something with the records in geoms here...and insert that data somewhere else
end loop;
end;
$do$
So, if I run this in more than one client, called from Java, will the scope of the geoms temp table cause problems? If so, any ideas for a solution to this in PostGres would be helpful.
Thanks
One subtle trap you will run into though, which is why I am not quite ready to declare it "safe" is that the scope is per session, but people often forget to drop the tables (so they drop on disconnect).
I think you are much better off if you don't need the temp table after your function to drop it explicitly after you are done with it. This will prevent issues that arise from trying to run the function twice in the same transaction. (On commit you are dropping)
Temp tables in PostgreSQL (or Postgres) (PostGres doesn't exists) are local only and related to session where they are created. So no other sessions (clients) can see temp tables from other session. Both (schema and data) are invisible for others. Your code is safe.
In my script I have to do a lot of selects to a joined table, so instead I decided to put this join into a temporal table.
First I thought:
1. Create table
2. Put the data from the join into a table
3. Drop the table
But then I thought, what if the script fails before I dropped the table?
So I decided to go with:
1. Drop the table
2. Create the table
3. Put the data from the join into a table
I don't really mind if the table is left there until the next time I run the script, so the second option works too.
But what if somebody had already dropped the table?
I saw some systems have a "drop if exists" but unfortunately not DB2. I would like to do something that won't make the script die when the drop table fails.
Ideas? On any of this? Thanks!
EDIT: I forgot to say this is in a PERL script!
The best way to do this is by using an annonymous block like in this code
You need to call the drop table in a dynamic sql, and catch the exception in the block.
--#SET TERMINATOR #
begin
declare statement varchar(128);
declare continue handle for sqlstate '42710' BEGIN END;
SET STATEMENT = 'DROP TABLE MYTABLE';
EXECUTE IMMEDIATE STATEMENT;
end #
This code will run normally in DB2. It does not need to be part of a procedure nor function.
Why not look for the table first? If you find it, it needs to be dropped; if you don't, it doesn't.
db2perf_quiet_drop that might works the way you want.. Its a free add-on :)
You can look into this post too..
http://www.dbforums.com/showthread.php?1609047-DB2-equivalent-for-mysql-s-DROP-TABLE-IF-EXISTS
If this doesn't work for you please let me know what error you are getting so I can try to help :)
Or this might work
if( NOT exists( create table detailval
(
id int,
detaildeptNo int,
info varchar(255)
);
insert into detailval(1,1, 'detail values A');
insert into detailval(2,1, 'detail values B');
insert into detailval(3,1, 'detail values C');
insert into detailval(4,2, 'detail values D');
)
)
then customStoredproc('droptable');
end if;
End
I think you should look into working with temporary tables (DECLARE GLOBAL TEMPORARY TABLE). They are stored in the temporary table space and are dropped automatically after commit.
You can easily also query syscat.tables like this:
select COUNT(*) from SYSCAT.TABLES where TRIM(TABNAME) = '<some_table_name>'
if this query returns 0 then the table does not exists.
I'm trying to get the hang of using temp tables:
CREATE OR REPLACE FUNCTION test1(user_id BIGINT) RETURNS BIGINT AS
$BODY$
BEGIN
create temp table temp_table1
ON COMMIT DELETE ROWS
as SELECT table1.column1, table1.column2
FROM table1
INNER JOIN -- ............
if exists (select * from temp_table1) then
-- work with the result
return 777;
else
return 0;
end if;
END;
$BODY$
LANGUAGE plpgsql;
I want the row temp_table1 to be deleted immediately or as soon as possible, that's why I added ON COMMIT DELETE ROWS. Obviously, I got the error:
ERROR: relation "temp_table1" already exists
I tried to add IF NOT EXISTS but I couldn't, I simply couldn't find working example of it that would be the I'm looking for.
Your suggestions?
DROP Table each time before creating TEMP table as below:
BEGIN
DROP TABLE IF EXISTS temp_table1;
create temp table temp_table1
-- Your rest Code comes here
The problem of temp tables is that dropping and recreating temp table bloats pg_attribute heavily and therefore one sunny morning you will find db performance dead, and pg_attribute 200+ gb while your db would be like 10gb.
So we're very heavy on temp tables having >500 rps and async i\o via nodejs and thus experienced a very heavy bloating of pg_attribute because of that. All you are left with is a very aggressive vacuuming which halts performance.
All answers given here do not solve this, because they all bloat pg_attribute heavily.
So the solution is elegantly this
create temp table if not exists my_temp_table (description) on commit delete rows;
So you go on playing with temp tables and save your pg_attribute.
You want to DROP term table after commit (not DELETE ROWS), so:
begin
create temp table temp_table1
on commit drop
...
Documentation