How to implement dml error logging in Postgresql? - postgresql

Question

In the context of datawarehousing, ETL process must have a strategy for error handling. About that, Oracle has a great dml error logging feature that lets you insert/merge/update a million records without failing or rolling back when constraint violation occurs with one or more rows, which can be logged in a dedicated error table. After that you can investigate what is wrong with each row and correct the errors before repeating the insert/merge/update.
Is there any way to implement this feature in Postgresql ?

Answer 1

Since there is nothing built in, nor any useful extension exists, I searched for a solution based on a pgsql procedure and eventually found it. It works well in my use case, where some csv files must be loaded once a month into a staging db using foreign tables.
In the following test some records are inserted into the destination tables while other records that break an integrity constraint are inserted in an error table along with the error info.
test=# create table t1(c1 int primary key);
create table t2( f1 int ,f2 int, f3 numeric);
insert into t1 values(2),(11),(5),(12);
insert into t2 values(100,2,234),(57,11,25),(5,5,1231),(2,2,173),(2,12,240),(11,22,101),(3,12,99);
create table t3 as select * from t2 where 1+1=11;
alter table t3 add constraint t3_pk primary key(f1),add foreign key (f2) references t1(c1),add constraint f3_ck check(f3>100);
create table t3$err(f1 int,f2 int,f3 numeric, error_code varchar, error_message varchar, constraint_name varchar);
test=# do
$$
declare
rec Record;
v_err_code text;
v_err_message text;
v_constraint text;
begin
for rec in
select f1,
f2,
f3
from t2 --in my use case this is the foreign table reading a csv file
loop
begin
insert
into t3
values (rec.f1,
rec.f2,
rec.f3);
exception
when others then
get stacked diagnostics
v_err_code= returned_sqlstate,
v_err_message= MESSAGE_TEXT,
v_constraint= CONSTRAINT_NAME;
if left(v_err_code, 2) = '23' then --exception Class 23 — Integrity Constraint Violation
insert
into t3$err
values (rec.f1,
rec.f2,
rec.f3,
v_err_code,
v_err_message,
v_constraint);
raise notice 'record % inserted in error table',rec;
end if;
end;
end loop;
exception
when others then --outer exceptions different from constraint violations
get stacked diagnostics
v_err_code= returned_sqlstate;
raise notice 'sqlstate: %', v_err_code;
end;
$$;
NOTICE: record (57,11,25) inserted in error table
NOTICE: record (2,12,240) inserted in error table
NOTICE: record (11,22,101) inserted in error table
NOTICE: record (3,12,99) inserted in error table
test=# select * from t3;
f1 | f2 | f3
-----+----+------
100 | 2 | 234
5 | 5 | 1231
2 | 2 | 173
(3 rows)
test=# select * from t3$err;
f1 | f2 | f3 | error_code | error_message | constraint_name
----+----+-----+------------+-----------------------------------------------------------------------------+-----------------
57 | 11 | 25 | 23514 | new row for relation "t3" violates check constraint "f3_ck" | f3_ck
2 | 12 | 240 | 23505 | duplicate key value violates unique constraint "t3_pk" | t3_pk
11 | 22 | 101 | 23503 | insert or update on table "t3" violates foreign key constraint "t3_f2_fkey" | t3_f2_fkey
3 | 12 | 99 | 23514 | new row for relation "t3" violates check constraint "f3_ck" | f3_ck
(4 rows)
All the magics is done within the nested BEGIN..END, where each row passing the constraints is inserted in the target table or else inserted in the error table.
The above solution has many limitations, such as:
the oracle feature mentioned in the question is fully integrated with SQL (except for the plsql preliminaries in order to create the error table), while here a pgsql procedure is needed,
iterating over all the records is not exactly the most efficient way for data loading in comparison with a bulk loading executed through a pgsql procedure,
moreover, the loop is accompanied by a overhead due to the context switch between the procedural environment and the sql environment,
the error handling is not generic but must be addressing specific errors
when a record has more than one error, only the last one is inserted in the error table (there could be a solution for this point).