I have 2 tables (1 staging table and 1 main operational table).
Both tables have the same structure.
For my solution,
I am using DB2Copy in program to insert 10000 records into staging table (4 seconds)
From the staging table, will move the data into main table using stored procedure (10 seconds)
However, it will lock the main table when running stored procedure.
I am suspecting because of the BEGIN and END which cause the stored procedure to act like a transaction.
I do not want the table to be locked when running stored procedure. (any suggestion?) Prefer: Stored procedure insert record by record into main table without transaction behavior.
Below is my code:
CREATE PROCEDURE SP_NAME ( )
LANGUAGE SQL
NOT DETERMINISTIC
CALLED ON NULL INPUT
EXTERNAL ACTION
OLD SAVEPOINT LEVEL
MODIFIES SQL DATA
INHERIT SPECIAL REGISTERS
BEGIN
--DECLARE TEMP VARIABLES
BEGIN
DECLARE MYCURSOR CURSOR WITH RETURN TO CALLER FOR
--SELECT STAGING TABLE
DECLARE CONTINUE HANDLER FOR NOT FOUND SET AT_END = 1;
OPEN MYCURSOR;
-- FETCH MYCURSOR INTO TEMP VARIABLES
WHILE AT_END = 0 DO
-- INSERT MAIN TABLE
-- FETCH MYCURSOR INTO TEMP VARIABLES
END WHILE;
CLOSE MYCURSOR;
END;
END;
My Environment
I have a program "A" which is trying to insert 10k records into main table (A lot of indexes and high volume of data) which takes 10 minutes ++.
About main operational table
High number of read but minimum updates and inserts at front end.
At back end, another program will frequently insert record into this table.
Only 1 instance of the back end program is allowed to run at one time
When you create the procedure, make sure your commitment-control setting is *NONE (a.k.a. autocommit). This should not lock your whole table
Adding the example
CREATE PROCEDURE userS.SP_TEST (
IN col_DATA Varchar(10) )
LANGUAGE SQL
SPECIFIC userS.SP_TEST
NOT DETERMINISTIC
MODIFIES SQL DATA
SET OPTION COMMIT = *NONE
BEGIN INSERT INTO userS.TABLE1 VALUES(col_DATA);
END
Related
Here is the test case and results:
drop table if exists test1;
drop table if exists test2;
drop trigger if exists test1_tr on test1;
drop function if exists tf_test1;
create table test1 (name varchar(8) not null);
create table test2 (name varchar(8) not null);
\echo create trigger function tf_test1
CREATE OR REPLACE FUNCTION tf_test1() RETURNS trigger AS $BODY$
BEGIN
IF TG_OP = 'INSERT' THEN
INSERT INTO test2(name) VALUES (NEW.name);
END IF;
return new;
END
$BODY$
LANGUAGE 'plpgsql';
\echo create trigger test1_tr
CREATE TRIGGER test1_tr
AFTER INSERT OR UPDATE OR DELETE ON test1 FOR EACH ROW
EXECUTE PROCEDURE tf_test1();
\echo Insert
insert into test1 (name) values ('NAME_001');
insert into test1 (name) values ('NAME_002');
insert into test1 (name) values ('NAME_003');
insert into test1 (name) values ('NAME_004');
\echo Select test1
select * from test1;
\echo Select test2
select * from test2;
---------------------------- output -------------------------------
DROP TABLE
DROP TABLE
DROP TABLE
DROP TABLE
DROP TRIGGER
DROP FUNCTION
CREATE TABLE
CREATE TABLE
create trigger function tf_test1
CREATE FUNCTION
create trigger test1_tr
CREATE TRIGGER
Insert
INSERT 0 1
psql:test3.sql:28: ERROR: cache lookup failed for type 113
CONTEXT: SQL statement "INSERT INTO test2(name) VALUES (NEW.name)"
PL/pgSQL function tf_test1() line 4 at SQL statement
INSERT 0 1
INSERT 0 1
Select test1
name
----------
NAME_001
NAME_003
NAME_004
(3 rows)
Select test2
name
----------
NAME_001
NAME_003
NAME_004
(3 rows)
We have several servers running various flavors of RHEL 7.x. All Postgres instances are v11. This is happening on about 1/2 of them. There doesn't seem to be any consistent RH version that is the culprit.
I have queried both pg_class and pg_type for the OID referenced as the missing type. In all cases, the result set is empty.
Any help is appreciated.
I would also appreciate an insight into what's happening with Postgres. I'm a long-time Oracle DBA, but fairly new to Postgres. It seems like an internal Postgres error and not really a code problem, but a web search doesn't turn up much.
Follow-up on this to provide some closure. We had increased our buffer and effective cache size in the Postgresql.conf file and also turned Auditing on (pgaudit extension) full blast...For the machines where the PG memory conf parameters exceeded the physical memory of the machine and auditing was turned on, we would get cache lookup errors. A clue about this was the errors would hop around in the job flow, were not consistent from machine to machine and were effectively unsquashable bugs (dropping the offending trigger would just cause the cache error somewhere else in the job stream).
For now, we have increased the physical memory of the servers and turned auditing off. The cache lookup errors are gone. Further tuning is needed so we can eventually turn auditing back on.
I am trying to write a trigger which gets data from the table attribute in which multiple rows are inserted corresponding to one actionId at one time and group all that data into the one object:
Table Schema
actionId
key
value
I am firing trigger on rows insertion,SO how can I handle this multiple row insertion and how can I collect all the data.
CREATE TRIGGER attribute_changes
AFTER INSERT
ON attributes
FOR EACH ROW
EXECUTE PROCEDURE log_attribute_changes();
and the function,
CREATE OR REPLACE FUNCTION wflowr222.log_task_extendedattribute_changes()
RETURNS trigger AS
$BODY$
DECLARE
_message json;
_extendedAttributes jsonb;
BEGIN
SELECT json_agg(tmp)
INTO _extendedAttributes
FROM (
-- your subquery goes here, for example:
SELECT attributes.key, attributes.value
FROM attributes
WHERE attributes.actionId=NEW.actionId
) tmp;
_message :=json_build_object('actionId',NEW.actionId,'extendedAttributes',_extendedAttributes);
INSERT INTO wflowr222.irisevents(message)
VALUES(_message );
RETURN NULL;
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
and data format is,
actionId key value
2 flag true
2 image http:test.com/image
2 status New
I tried to do it via Insert trigger, but it is firing on each row inserted.
If anyone has any idea about this?
I expect that the problem is that you're using a FOR EACH ROW trigger; what you likely want is a FOR EACH STATEMENT trigger - ie. which only fires once for your multi-line INSERT statement. See the description at https://www.postgresql.org/docs/current/sql-createtrigger.html for a more through explanation.
AFAICT, you will also need to add REFERENCING NEW TABLE AS NEW in this mode to make the NEW reference available to the trigger function. So your CREATE TRIGGER syntax would need to be:
CREATE TRIGGER attribute_changes
AFTER INSERT
ON attributes
REFERENCING NEW TABLE AS NEW
FOR EACH STATEMENT
EXECUTE PROCEDURE log_attribute_changes();
I've read elsewhere that the required REFERENCING NEW TABLE ... syntax is only supported in PostgreSQL 10 and later.
Considering the version of postgres you have, and therefore keeping in mind that you can't use a trigger defined FOR EACH STATEMENT for your purpose, the only alternative I see is
using a trigger after insert in order to collect some information about changes in a utility table
using a unix cron that execute a pl/sql that do the job on data set
For example:
Your utility table
CREATE TABLE utility (
actionid integer,
createtime timestamp
);
You can define a trigger FOR EACH ROW with a body that do something like this
INSERT INTO utilty values(NEW.actionid, curent_timestamp);
And, finally, have a crontab UNIX that execute a file or a procedure that to something like this:
SELECT a.* FROM utility u JOIN yourtable a ON a.actionid = u.actionid WHERE u.createtime < current_timestamp;
// do something here with records selected above
TRUNCATE table utility;
If you had postgres 9.5 you could have used pg_cron instead of unix cron...
Within a plpgsql function, I need to hold the result set of a select statement and perform many subsequent queries and manipulations over that set.
I read about temp tables in PostgreSQL, but these tables are visible in session/transaction scope, while i need my table (or any data structure holding result set) to be locally visible and only exist within the function, so that each function call can have its own copy of that (table/data structure)
I simply need a table alike structure to hold select result set within a function call, instead of temp tables.
Is there such thing?
Concurrent sessions may have their own (local) temp tables with the same names. Here is an example of a function which does not do anything wise but creates a temp table, returns its data and drops the table on exit:
create or replace function my_function()
returns table (id int, str text)
language plpgsql as $$
begin
create temp table my_temp_table as
select i as id, i::text as str
from generate_series(1, 3) i;
return query
select *
from my_temp_table;
drop table my_temp_table;
end $$;
The function may be safely run in concurrent sessions.
drop table if exists... at the beginning of the function is a reasonable alternative. Do not forget to use temp or temporary in create temp table...
In a ms sql database I have a table named combo where multiple inserts, updates and deletes can happen (as well as single, of course). In another table named migrimi_temp I keep track of these changes in the form of queries (query that would have to be executed in mysql to achieve the same result).
For example, if a delete query is performed for all rows where id > 50, the trigger should activate to store the following query into the log table:
DELETE FROM combo where id > 50;
Therefore this one delete query in the combo table would result in one row in the log table.
But if instead I have an insert query inserting 2 rows, a trigger should activate to store each insert into the log table. So this one insert query in the combo table would result into 2 new rows in the log table.
I intend to handle insert, update and delete actions into separated triggers. I had managed to write triggers for single row insert / update/ delete. Then it occurred to me that multiple actions might be performed too.
This is my attempt to handle the case of multiple inserts in one single query. I resorted to using cursors after not being able to adapt the initial trigger without a cursor. The trigger is executed successfully, but when I perform an insert (single or multiple rows) the execution hangs up indefinitely, or at least longer than reasonable .
USE [migrimi_test]
GO
/****** Object: Trigger [dbo].[c_combo] Script Date: 12/11/2017 5:33:46 PM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
create TRIGGER [dbo].[u_combo]
ON [migrimi_test].[dbo].[combo]
AFTER INSERT
AS
BEGIN
SET NOCOUNT ON
DECLARE #c_id INT;
DECLARE #c_name nvarchar(100);
DECLARE #c_duration int;
DECLARE #c_isavailable INT;
DECLARE c CURSOR FOR
SELECT id, name, duration, isvisible FROM inserted
OPEN c
FETCH NEXT FROM c INTO #c_id, #c_name, #c_duration, #c_isavailable
WHILE ##FETCH_STATUS = 0
INSERT INTO [migrimi_temp].[dbo].[sql_query] (query)
VALUES ('INSERT INTO combo (id, name, duration, value, isavailable, createdAt, updatedAt) values ('+CAST(#c_id as nvarchar(50))+', '+'"'+#c_name+'"'+',
'+CAST(#c_duration as nvarchar(50))+', 1, '+CAST(#c_isavailable as nvarchar(50))+', Now(), Now());' )
FETCH NEXT FROM c INTO #c_id, #c_name, #c_duration, #c_isavailable
CLOSE c
END
DEALLOCATE c
GO
SQL server version is 2012. OS is windows server 2008 (though I doubt that is relevant). I was based mainly on these two resources: https://social.msdn.microsoft.com/Forums/sqlserver/en-US/40f5635c-9034-4e9b-8fd5-c02cec44ce86/how-to-let-trigger-act-for-each-row?forum=sqlgetstarted
and How can I get a trigger to fire on each inserted row during an INSERT INTO Table (etc) SELECT * FROM Table2?
This is part of a larger idea I am trying to accomplish, and until 2 days ago I was totally unfamiliar with triggers. I am trying to balance learning with accomplishing in reasonable amounts of time, but not doing so great
Cursors are notoriously slow in SQL Server.
Instead of using a cursor to loop over the inserted table, you can use insert...select which is a set based approach. It is much faster and is the recommended way to work in SQL:
CREATE TRIGGER [dbo].[u_combo]
ON [migrimi_test].[dbo].[combo]
AFTER INSERT
AS
BEGIN
INSERT INTO [migrimi_temp].[dbo].[sql_query] (query)
SELECT 'INSERT INTO combo (id, name, duration, value, isavailable, createdAt, updatedAt) values ('+CAST(id as nvarchar(50))+', "'+ name +'",
'+ CAST(duration as nvarchar(50)) +', 1, '+ CAST(isvisible as nvarchar(50))+ ', Now(), Now());'
FROM inserted
END
GO
I want to create a trigger after a inserted event, but I need the data that I inserted in order to register into a new table for my trigger in PostgreSQL
In SQL Server I capture these values from the Inserted or deleted pseudo tables but do these tables also exists in PostgreSQL? Or what can I do?
This is my trigger code
CREATE TRIGGER tri_compago
AFTER INSERT
ON matricula
FOR EACH ROW
EXECUTE PROCEDURE fn_insCompPago();
CREATE OR REPLACE FUNCTION fn_insCompPago()
RETURNS trigger AS
$BODY$
DECLARE
BEGIN
insert into compromisopago(codigotasa,descripcion,precio,fechavencimiento,codigomatricula)
select codigotasa,descripcion,precio,fechavencimiento,i.codigo
from programacionpago pp join inserted i on isnull(i.codigoconvenio,0) = isnull (pp.codigoconvenio,0)
and pp.codigopresentacion = i.codigopresentacion
where pp.vigencia = 1 and i.vigencia = 1;
RETURN NULL;
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION fn_insCompPago()
OWNER TO postgres;
I have no idea how triggers work in SQL Server but in PostgreSQL, you use the OLD and NEW special variables:
NEW
Data type RECORD; variable holding the new database row for INSERT/UPDATE operations in row-level triggers. This variable is NULL in statement-level triggers and for DELETE operations.
OLD
Data type RECORD; variable holding the old database row for UPDATE/DELETE operations in row-level triggers. This variable is NULL in statement-level triggers and for INSERT operations.
So you probably want to look at NEW.codigo, NEW.codigoconvenio, NEW.codigopresentacion, and NEW.vigencia in your case. You'd probably replace the i.vigencia = 1 part of the WHERE clause with a simple IF i.vigencia = 1 conditional as well.
A trigger defined as for each row is fired - well - for each row in Postgres. SQL Server does not support row level triggers, only statement level triggers.
Inside a row level trigger you always deal with exactly one row with the old and new values accessible (very roughly comparable to "inserted" and "deleted" virtual tables in SQL Server)
You can specify under which name you want to reference those records, the default is new and old (as mu is too short has already explained).
So as the values you are interested in are available as "scalar" values, you don't need any join to do your insert:
insert into compromisopago
(codigotasa,descripcion,precio,fechavencimiento,codigomatricula)
select codigotasa,
descripcion,
precio,
fechavencimiento,
new.codigo
from programacionpago pp
where pp.vigencia = 1
and i.vigencia = 1;
and pp.codigoconvenio = new.codigoconvenio
and pp.codigopresentacion = new.codigopresentacion;