I have a table in Redshift (let's call it a status table) where I set the status of tables which I want to truncate. I created a Redshift Stored Procedure in order to achieve that. Here is my code for the SP:
CREATE OR REPLACE PROCEDURE <schema>.truncate_table()
AS $$
DECLARE
v_tpsl RECORD;
exec_statement VARCHAR(256);
BEGIN
FOR v_tpsl in SELECT * from <schama>.tablename_process_status_log WHERE status = 'TRUE' LOOP
exec_statement = 'TRUNCATE TABLE <schema>.' + quote_ident(v_tpsl.staging_table_name) + '_test;';
RAISE INFO 'statement = %', exec_statement;
EXECUTE exec_statement;
END LOOP;
END;
$$
LANGUAGE plpgsql;
Now when I am CALLING the Stored Procedure, I am getting this error:
SQL Error [500310] [34000]: [Amazon](500310) Invalid operation: cursor does not exist;
I looked at the documentation of the SP to check if Truncate is possible or not. By looking at the examples, it looks like it's possible.
I am not sure what is going wrong in this. I am using RedshiftJDBC42-no-awssdk-1.2.34.1058.jar and connecting via DBeaver.
It looks like I have found the answer. According to this, Any cursor that is open (explicitly or implicitly) is closed automatically when a COMMIT, ROLLBACK, or TRUNCATE statement is processed. In my next iteration of the loop, it's trying to accessing the cursor which is already closed.
Related
I'm new to Postgres, but with experience from Oracle. Trying to create a stored procedure which is going to:
Insert a row
Handle exceptions and in case of an exception insert a row into a log table by calling dedicated procedure
Emit an audit log record into a log table in case the whole procedure ran successfully
By pseudo code:
CREATE OR REPLACE PROCEDURE test.p_insert(IN p_test_param character varying)
LANGUAGE 'plpgsql'
SECURITY DEFINER
AS $BODY$
DECLARE
-- some declarations
BEGIN
BEGIN
INSERT INTO test.a(a) VALUES (p_test_param);
EXCEPTION
WHEN OTHERS THEN
-- GET STACKED DIAGNOSTICS
CALL test.p_insert_log(...); -- Inserts a row into a log table, another COMMIT may be required?
RAISE;
END;
COMMIT; -- CAN'T DO
BEGIN
IF (SELECT test.f_debug()) THEN
CALL test.p_insert_log(...); -- Audit the execution
END IF;
END;
COMMIT; -- CAN'T DO EITHER
END;
$$BODY$$;
However when I try to test the procedure out from an anonymous block in PgAdmin such as:
BEGIN;
DO
LANGUAGE plpgsql
$$
BEGIN
CALL test.p_insert(
p_test_param => 'test'
);
END;
$$
I'm getting an error ERROR: invalid transaction termination. How can I get rid of it? My objective is to let the procedure carry out the transaction control, I don't want the caller to COMMIT or ROLLBACK anything. If I remove both COMMIT commands from the code of the procedure, it executes well, however the invoker must explicitly COMMIT or REVOKE the transaction afterwards, which is not desired. In Oracle the pseudo code with COMMIT statements would work, in Postgres it doesn't seem to work as I would like to. Could you please help me out? Thanks
Your code will work as intended. Perhaps you made some mistake in calling the code:
you cannot call the procedure from a function
you cannot call the procedure in an explicitly started transaction:
BEGIN;
CALL p_insert('something); -- will fail
COMMIT;
I have Postgresql Function which has to INSERT about 1.5 million data into a table. What I want is I want to see the table getting populated with every one records insertion. Currently what is happening when I am trying with say about 1000 records, the get gets populated only after the complete function gets executed. If I stop the function half way through, no data gets populated. How can I make the record committed even if I stop after certain number of records have been inserted?
This can be done using dblink. I showed an example with one insert being committed you will need to add your while loop logic and commit every loop. You can http://www.postgresql.org/docs/9.3/static/contrib-dblink-connect.html
CREATE OR REPLACE FUNCTION log_the_dancing(ip_dance_entry text)
RETURNS INT AS
$BODY$
DECLARE
BEGIN
PERFORM dblink_connect('dblink_trans','dbname=sandbox port=5433 user=postgres');
PERFORM dblink('dblink_trans','INSERT INTO dance_log(dance_entry) SELECT ' || '''' || ip_dance_entry || '''');
PERFORM dblink('dblink_trans','COMMIT;');
PERFORM dblink_disconnect('dblink_trans');
RETURN 0;
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION log_the_dancing(ip_dance_entry text)
OWNER TO postgres;
BEGIN TRANSACTION;
select log_the_dancing('The Flamingo');
select log_the_dancing('Break Dance');
select log_the_dancing('Cha Cha');
ROLLBACK TRANSACTION;
--Show records committed even though we rolled back outer transaction
select *
from dance_log;
What you're asking for is generally called an autonomous transaction.
PostgreSQL does not support autonomous transactions at this time (9.4).
To properly support them it really needs stored procedures, not just the user-defined functions it currently supports. It's also very complicated to implement autonomous tx's in PostgreSQL for a variety of internal reasons related to its session and process model.
For now, use dblink as suggested by Bob.
If you have the flexibility to change from function to procedure, from PostgreSQL 12 onwards you can do internal commits if you use procedures instead of functions, invoked by CALL command. Therefore your function will be changed to a procedure and invoked with CALL command: e.g:
CREATE PROCEDURE transaction_test2()
LANGUAGE plpgsql
AS $$
DECLARE
r RECORD;
BEGIN
FOR r IN SELECT * FROM test2 ORDER BY x LOOP
INSERT INTO test1 (a) VALUES (r.x);
COMMIT;
END LOOP;
END;
$$;
CALL transaction_test2();
More details about transaction management regarding Postgres are available here: https://www.postgresql.org/docs/12/plpgsql-transactions.html
For Postgresql 9.5 or newer you can use dynamic background workers provided by pg_background extension. It creates autonomous transaction. Please, refer the github page of the extension. The sollution is better then db_link. There is a complete guide on Autonomous transaction support in PostgreSQL. There is a third way to start autonomous transaction in Postgres, but some patching neede. Please see Peter's Eisentraut patch proposal for OracleDB-style transactions.
Within a plpgsql stored procedure, I am attempting to write to a list of hosts (using PL/Proxy - unrelated to this problem) based on an active column in table A, where A contains a column host and column active which indicates whether or not the host is alive and should be written to (for each row in A where active = TRUE, write_to_host(A.host)). The function write_hosts reads from the A table and only chooses the hosts that are marked active=TRUE .
Upon catching an exception while attempting to write to one of the hosts, I want to update table A, setting A.active = FALSE WHERE A.host = <badhost>, and continue attempting to write to the other hosts. It looked something like the following, where I wanted to attempt the insert, fail, update the A table marking the row as inactive, then recursively call the write_doc function, which should see the remaining active entries and only attempt to write to them:
CREATE OR REPLACE FUNCTION write_doc(args...)
RETURNS SETOF JSON AS $$
BEGIN
SELECT * FROM write_to_host(args...);
RETURN NEXT ARRAY_TO_JSON(ARRAY[TRUE::text, 'inserted']);
RETURN;
EXCEPTION WHEN OTHERS THEN
IF SQLERRM ~ '.+Connection refused.+' THEN
EXECUTE 'UPDATE hosts SET active = FALSE WHERE host = ''' || _bad_host || '''';
END;
RETURN QUERY SELECT * FROM write_doc(args...);
$$ LANGUAGE plpgsql;
This results in an infinite loop, which I have to CNTRL-C out of.
When I comment out the RETURN QUERY bit, the deactivation works for the host in question. When I leave it in, table A is not updated (at least not before I stop the execution of the procedure).
Using an alternative method with a LOOP, it still ends up looping infinitely for me, and table A is not updated... something like the following, where the RETURN statement should exit the loop:
CREATE OR REPLACE FUNCTION write_doc(args...)
RETURNS SETOF JSON AS $$
BEGIN
LOOP
BEGIN
SELECT * FROM write_to_host(args...);
RETURN NEXT ARRAY_TO_JSON(ARRAY[TRUE::text, 'inserted']);
RETURN;
EXCEPTION WHEN OTHERS THEN
IF SQLERRM ~ '.+Connection refused.+' THEN
EXECUTE 'UPDATE hosts SET active = FALSE WHERE host = ''' || _bad_host || '''';
END;
END;
END LOOP;
END;
$$ LANGUAGE plpgsql;
I understand that all stored procedures are transactional, but thought that the table would be updated within the transaction and would be viewable by the code running in the stored procedure, guess I am wrong?
Any suggestions/information someone can give to lead me in the right direction? Any questions don't hesitate to ask
I am seeking clarification of how to ensure an atomic transaction in a plpgsql function, and where the isolation level is set for this particular change to the database.
In the plpgsql function shown below, I want to make sure that BOTH the deletion AND the insertion succeed. I am getting an error when I try to wrap them in a single transaction:
ERROR: cannot begin/end transactions in PL/pgSQL
What happens during execution of the function below if another user has added a default behavior for circumstances ('RAIN', 'NIGHT', '45MPH') after this function has deleted the custom row but before it has had a chance to insert the custom row? Is there an implicit transaction wrapping the insert and delete so that both are rolled back if another user has changed either of the rows referenced by this function? Can I set the isolation level for this function?
create function foo(v_weather varchar(10), v_timeofday varchar(10), v_speed varchar(10),
v_behavior varchar(10))
returns setof CUSTOMBEHAVIOR
as $body$
begin
-- run-time error if either of these lines is un-commented
-- start transaction ISOLATION LEVEL READ COMMITTED;
-- or, alternatively, set transaction ISOLATION LEVEL READ COMMITTED;
delete from CUSTOMBEHAVIOR
where weather = 'RAIN' and timeofday = 'NIGHT' and speed= '45MPH' ;
-- if there is no default behavior insert a custom behavior
if not exists
(select id from DEFAULTBEHAVIOR where a = 'RAIN' and b = 'NIGHT' and c= '45MPH') then
insert into CUSTOMBEHAVIOR
(weather, timeofday, speed, behavior)
values
(v_weather, v_timeofday, v_speed, v_behavior);
end if;
return QUERY
select * from CUSTOMBEHAVIOR where ... ;
-- commit;
end
$body$ LANGUAGE plpgsql;
A plpgsql function automatically runs inside a transaction. It all succeeds or it all fails. The manual:
Functions and trigger procedures are always executed within a
transaction established by an outer query — they cannot start or
commit that transaction, since there would be no context for them to
execute in. However, a block containing an EXCEPTION clause
effectively forms a subtransaction that can be rolled back without
affecting the outer transaction. For more about that see Section 42.6.6.
So, if you need to, you can catch an exception that theoretically might occur (but is very unlikely).
Details on trapping errors in the manual.
Your function reviewed and simplified:
CREATE FUNCTION foo(v_weather text
, v_timeofday text
, v_speed text
, v_behavior text)
RETURNS SETOF custombehavior
LANGUAGE plpgsql AS
$func$
BEGIN
DELETE FROM custombehavior
WHERE weather = 'RAIN'
AND timeofday = 'NIGHT'
AND speed = '45MPH';
INSERT INTO custombehavior (weather, timeofday, speed, behavior)
SELECT v_weather, v_timeofday, v_speed, v_behavior
WHERE NOT EXISTS (
SELECT FROM defaultbehavior
WHERE a = 'RAIN'
AND b = 'NIGHT'
AND c = '45MPH'
);
RETURN QUERY
SELECT * FROM custombehavior WHERE ... ;
END
$func$;
If you actually need to begin/end transactions like indicated in the title look to SQL procedures in Postgres 11 or later (CREATE PROCEDURE). See:
In PostgreSQL, what is the difference between a “Stored Procedure” and other types of functions?
Update: after PostgreSQL version 11. you can control transaction inside Store Procedure.
=====
Before Version 10:
START TRANSACTION;
select foo() ;
COMMIT;
"Unfortunately Postgres has no stored procedures, so you always need to manage the transaction in the calling code" – a_horse_with_no_name
Transaction in an exception block - how?
We have a problem with our software and in order to correct the issue, I have to write a stored procedure that will be run as part of the upgrade process for upgrade installs. This stored procedure needs to find every row in a particular table that matches certain conditions and update that row. For internal reasons, the update has to be done through a stored procedure we wrote specifically for inserting and updating data.
Here is the stored procedure I have written to fix this issue:
CREATE OR REPLACE FUNCTION FixDataProblem() RETURNS VOID AS $$
DECLARE
FixCursor NO SCROLL CURSOR FOR
SELECT * FROM MyTable WHERE ProblemColumn IN ( '?', 'PR' );
RowToUpdate MyTable%ROWTYPE;
BEGIN
-- Open the cursor
OPEN FixCursor;
-- Start a loop
LOOP
-- Fetch the next row from thr cursor
FETCH FixCursor INTO RowToUpdate;
-- Did we get anything back?
IF RowToUpdate IS NULL THEN
-- We didn't. Exit the loop
EXIT;
END IF;
-- Call the UpsertMyTable stored procedure to set the ProblemColumn column to NULL
SELECT CarSystem.UpsertMyTable( RowToUpdate.RowId,
RowToUpdate.ForeignId,
RowToUpdate.CountryId,
NULL,
RowToUpdate.Plate,
RowToUpdate.HashedData,
RowToUpdate.PlateClassId,
RowToUpdate.AlarmClassId,
RowToUpdate.BeginDate,
RowToUpdate.EndDate,
RowToUpdate.ListPriorityId,
RowToUpdate.VehicleTypeId,
RowToUpdate.MakeId,
RowToUpdate.ModelId,
RowToUpdate.Year,
RowToUpdate.ColorId,
RowToUpdate.Notes,
RowToUpdate.OfficerNotes,
NULL,
UUID_GENERATE_V4() );
END LOOP;
-- Close the cursor
CLOSE ListDetailsCursor;
END;
$$ LANGUAGE plpgsql;
This stored procedure fine, but when I run it, I get:
ERROR: query has no destination for result data
HINT: If you want to discard the results of a SELECT, use PERFORM instead.
CONTEXT: PL/pgSQL function "fixdataproblem" line 22 at SQL statement
********** Error **********
ERROR: query has no destination for result data
SQL state: 42601
Hint: If you want to discard the results of a SELECT, use PERFORM instead.
Context: PL/pgSQL function "fixdataproblem" line 22 at SQL statement
How do I fix this issue? I believe I am calling the stored procedure correctly. I really don't know what the issue with this stored procedure is.
Thanks
Tony
It says right there:
ERROR: query has no destination for result data
HINT: If you want to discard the results of a SELECT, use PERFORM instead.
CONTEXT: PL/pgSQL function "fixdataproblem" line 22 at SQL statement
And on line 22:
-- Call the UpsertMyTable stored procedure to set the ProblemColumn column to NULL
SELECT CarSystem.UpsertMyTable( RowToUpdate.RowId,
...
Change it from SELECT to PERFORM. See PERFORM for why.