Related
I would like to create a basic audit log trigger in postgres that I can use across various tables.
My only requirement is that the audit log show each updated value as a separate entry.
i.e. if an INSERT is performed into a table with 5 rows, I receive 5 entries in the audit log table (one for each value added) or if that row is deleted from the table, I receive 5 entries (one for each value removed)
I have looked at various examples and am still having trouble getting the output correct, especially when the operation is UPDATE.
Here is the basic trigger flow.
-- 1. Making the Audit Table
CREATE TABLE audit_table
-- The audit table which will show:
-- Table, the ID of whats being changed, new columns, old columns,
-- user, operation(insert update delete), and timestamp
(
"src_table" TEXT NOT NULL,
"src_id" INT NOT NULL,
"col_new" TEXT,
"col_old" TEXT,
"user" TEXT DEFAULT current_user,
"action" TEXT NOT NULL,
"when_changed" TIMESTAMP
);
-- 2. Creating the base audit trigger function, the engine for processing changes --
CREATE OR REPLACE FUNCTION audit_trigger()
RETURNS trigger AS
$$
BEGIN
IF TG_OP = 'INSERT' --Only shows new values
THEN
INSERT INTO audit_table ( "src_table", "src_id", "col_new", "user", "action", "when_changed")
VALUES(TG_TABLE_NAME, TG_RELID, row_to_json(NEW), current_user, TG_OP, current_timestamp);
RETURN NEW;
ELSIF TG_OP = 'UPDATE' --Shows new and old values
THEN
INSERT INTO audit_table ("src_table", "src_id", "col_new", "col_old", "user", "action", "when_changed")
VALUES(TG_TABLE_NAME, TG_RELID, row_to_json(NEW), row_to_json(OLD), current_user, TG_OP, current_timestamp);
RETURN NEW;
ELSIF TG_OP = 'DELETE' --Only shows old values
THEN
INSERT INTO audit_table ("src_table", "src_id", "col_old", "user", "action", "when_changed")
VALUES(TG_TABLE_NAME, TG_RELID, row_to_json(OLD), current_user, TG_OP, current_timestamp);
RETURN OLD;
END IF;
END
$$
LANGUAGE 'plpgsql';
-- 3. Basic logic for calling audit trigger on tables, works for any insert, update, delete
CREATE TRIGGER test_audit_trigger
BEFORE INSERT OR UPDATE OR DELETE
ON test_table
FOR EACH ROW EXECUTE PROCEDURE audit_trigger();
The issue:
row_to_json returns the entire old or new payload as one column. I would like to return each change as a separate entry row in the audit_table, using for example column_name old_value new_value as a schema. What is the simplest way of achieving this? Having it be json is not a requirement.
Also not required:
Having it be BEFORE keyword, if AFTER or a combination of the two is better.
Having one trigger do all 3 functions, it can be a separate trigger or function per action
Returning values that are not changed, for example if an UPDATE only changes 1 value, I do not need the unchanged values in the log.
Here is a possible solution based on yours. One AFTER row trigger for insert, update and delete. The structure of audit_table is a bit different. Old and new values are saved as their text representation. You may wish to change the behaviour under insert and delete as they seem to be too verbose.
Unrelated but under your requirements the audit table has to implement the EAV (entity, attribute, value) antipattern.
drop table if exists audit_table;
create table audit_table
(
src_table text not null,
col_name text not null,
v_old text,
v_new text,
"user" text not null default current_user,
action text not null,
when_changed timestamptz not null default current_timestamp
);
create or replace function audit_tf() returns trigger language plpgsql as
$$
declare
k text;
v text;
j_new jsonb := to_jsonb(new);
j_old jsonb := to_jsonb(old);
begin
if TG_OP = 'INSERT' then -- only shows new values
for k, v in select * from jsonb_each_text(j_new) loop
insert into audit_table (src_table, col_name, v_new, action)
values (TG_TABLE_NAME, k, v, TG_OP);
end loop;
elsif TG_OP = 'UPDATE' then -- shows new and old values
for k, v in select * from jsonb_each_text(j_new) loop
if (v <> j_old ->> k) then
insert into audit_table (src_table, col_name, v_new, v_old, action)
values (TG_TABLE_NAME, k, v, j_old ->> k, TG_OP);
end if;
end loop;
elsif TG_OP = 'DELETE' then -- only shows old values
for k, v in select * from jsonb_each_text(j_old) loop
insert into audit_table (src_table, col_name, v_old, action)
values (TG_TABLE_NAME, k, v, TG_OP);
end loop;
end if;
return null;
end;
$$;
LANGUAGE 'plpgsql';
-- Demo
drop table if exists delme;
create table delme (x integer, y integer, z text, t timestamptz default now());
create trigger test_audit_trigger
after insert or update or delete on delme
for each row execute procedure audit_tf();
insert into delme (x, y, z) values (2, 2, 'two');
update delme set x = 1, y = 10 where x = 2;
delete from delme where x = 1;
select src_table, col_name, v_old, v_new, action from audit_table;
src_table
col_name
v_old
v_new
action
delme
t
2021-12-28T00:21:03.966621
INSERT
delme
x
2
INSERT
delme
y
2
INSERT
delme
z
two
INSERT
delme
x
2
1
UPDATE
delme
y
2
10
UPDATE
delme
t
2021-12-28T00:21:03.966621
DELETE
delme
x
1
DELETE
delme
y
10
DELETE
delme
z
two
DELETE
I wanted to have an explanation on triggers of Postgres views.
To make clear what I want to ask, I'll give you a very simplified example of my case.
In this example we have two tables (table_a, table_b) that joined together make the view in the example (vw_table_ab).
In this example I will use trivial names and simple DDLs/DMLs.
-- TABLE table_a
CREATE TABLE table_a
(
id serial PRIMARY KEY,
timestamp_field timestamp DEFAULT now() NOT NULL,
boolean_field boolean DEFAULT FALSE NOT NULL
);
-- TABLE table_b
CREATE TABLE table_b
(
id serial PRIMARY KEY,
timestamp_field timestamp DEFAULT now() NOT NULL,
boolean_field boolean DEFAULT FALSE NOT NULL,
id_table_a integer NOT NULL,
CONSTRAINT "fk_table_a" FOREIGN KEY (id_table_a) REFERENCES table_a (id) ON DELETE CASCADE NOT DEFERRABLE,
CONSTRAINT "u_table_a" UNIQUE (id_table_a)
);
-- VIEW vw_table_ab
CREATE VIEW vw_table_ab AS (
SELECT a.timestamp_field AS timestamp_a,
a.boolean_field AS boolean_a,
b.timestamp_field AS timestamp_b,
b.boolean_field AS boolean_b
FROM table_a a
JOIN table_b b ON a.id = b.id_table_a
);
A trigger function on standard actions (INSERT, UPDATE and DELETE) is linked to this view through an INSTEAD OF trigger.
-- TRIGGER FUNCTION fn_trigger
CREATE FUNCTION fn_trigger() RETURNS trigger LANGUAGE plpgsql AS
$_$
DECLARE
sql TEXT;
BEGIN
sql = 'SELECT ' || TG_TABLE_NAME || '_' || lower(TG_OP) || '($1);';
IF TG_OP = 'INSERT' OR TG_OP = 'UPDATE' THEN
EXECUTE (sql) USING NEW;
RAISE NOTICE '%', sql;
RETURN NEW;
ELSE
EXECUTE (sql) USING OLD;
RAISE NOTICE '%', sql;
RETURN OLD;
END IF;
END;
$_$;
-- TRIGGER tr_table_ab
CREATE TRIGGER tr_table_ab
INSTEAD OF INSERT OR UPDATE OR DELETE ON vw_table_ab
FOR EACH ROW EXECUTE PROCEDURE fn_trigger();
The example I bring has a trigger called only on the insert action, and the function that is executed is this:
-- INSERT FUNCTION vw_table_ab_insert
CREATE FUNCTION vw_table_ab_insert(new vw_table_ab) RETURNS void LANGUAGE plpgsql AS
$_$
DECLARE
id_table_a integer;
BEGIN
INSERT INTO table_a (timestamp_field, boolean_field) VALUES (new.timestamp_a, new.boolean_a)
RETURNING id
INTO id_table_a;
INSERT INTO table_b (timestamp_field, boolean_field, id_table_a) VALUES (new.timestamp_a, new.boolean_b, id_table_a);
END;
$_$;
Now we can get to my problem. I make an insert on the view, and when the action is triggered, I get a "Not null violation" error becouse I have some NOT NULL constraints on table_a and table_b like in this case:
INSERT INTO vw_table_ab (timestamp_a, boolean_a, timestamp_b, boolean_b) VALUES (now(), NULL, now(), NULL);
Suppose that the previous statement is generated through a programming language framework and I don't want to handle this case in backend code, but I want handle this case in PostgreSQL in the insert function vw_table_ab_insert. So at this point my problem is bound to the new parameter of the function because I have fields of the view that are NULL. But these fields have a DEFAULT value in the definition of the base table, and I want to use that.
...
timestamp_field timestamp DEFAULT now() NOT NULL,
boolean_field boolean DEFAULT FALSE NOT NULL
...
My question:
How can I manage the NULL values in trigger of the views using the DEFAULT defined in the tables?
Initially I thought of putting IF ... THEN ... inside the function and override null values with DEFAULT expression but I do not really like that.
For example, the function would become like this:
CREATE FUNCTION vw_table_ab_insert(new vw_table_ab) RETURNS void LANGUAGE plpgsql AS
$_$
DECLARE
id_table_a integer;
BEGIN
IF new.timestamp_a IS NULL THEN
new.timestamp_a = DEFAULT;
END IF;
IF new.boolean_a IS NULL THEN
new.boolean_a = DEFAULT;
END IF;
IF new.timestamp_b IS NULL THEN
new.timestamp_b = DEFAULT;
END IF;
IF new.boolean_b IS NULL THEN
new.boolean_b = DEFAULT;
END IF;
INSERT INTO table_a (timestamp_field, boolean_field)
VALUES (new.timestamp_a, new.boolean_a)
RETURNING id
INTO id_table_a;
INSERT INTO table_b (timestamp_field, boolean_field, id_table_a)
VALUES (new.timestamp_a, new.boolean_b, id_table_a);
END;
$_$;
Someone can help me? Is there another method for handling this case?
The easiest way would be to use ALTER VIEW ... ALTER col SET DEFAULT to define default values on the view that are the same as the default values on the base table.
Then instead of inserting explicit NULLs, omit the columns from the INSERT statement or insert DEFAULT explicitly. Your resulting view will behave just like a real table.
I have the following small MVWE for a basic micro-auditing system which works fine but lacks a functionality:
DROP TABLE IF EXISTS audit CASCADE;
CREATE TABLE audit(
Id BIGSERIAL NOT NULL
,TimeValue TIMESTAMP NOT NULL
,RoleName NAME NOT NULL
,Operation NAME NOT NULL
,SchemaName NAME NOT NULL
,TableName NAME NOT NULL
,Identifiers BIGINT[]
---
,PRIMARY KEY(Id)
);
-- Audit Trigger:
DROP FUNCTION IF EXISTS audit_trigger() CASCADE;
CREATE OR REPLACE FUNCTION audit_trigger()
RETURNS TRIGGER AS
$BODY$
BEGIN
INSERT INTO audit(TimeValue, RoleName, Operation, SchemaName, TableName) VALUES
(now()::TIMESTAMP, current_user, TG_OP, TG_TABLE_SCHEMA, TG_RELNAME);
RETURN NULL;
END;
$BODY$
LANGUAGE plpgsql SECURITY DEFINER;
-- Channels:
DROP TABLE IF EXISTS channels CASCADE;
CREATE TABLE channels(
Id INTEGER NOT NULL
,UserKey TEXT NOT NULL
,Active BOOLEAN NOT NULL DEFAULT(TRUE)
---
,PRIMARY KEY(Id)
,UNIQUE(UserKey)
);
CREATE TRIGGER channel_audit_trigger BEFORE INSERT OR UPDATE OR DELETE ON channels
FOR EACH STATEMENT EXECUTE PROCEDURE audit_trigger();
-- Perform some operations:
INSERT INTO channels(
SELECT C.Id, 'Channel-' || C.Id
FROM generate_series(1, 300, 10) AS C(Id)
);
DELETE FROM channels WHERE id < 10;
UPDATE channels
SET UserKey = 'wild channel'
WHERE id = 21;
I would like to add into the last column of audit table, identifiers of rows that have been inserted/updated/deleted in channels.
I have used STATEMENT level because I just need to collects identifiers in a array. But I do not find how to access DML statistics. Conversely at the ROW level I must handle OLD and NEW cases and I cannot succeed in aggregate all touched identifier.
How can I proceed in order to fill the last column of audit table with touched identifiers?
Update
Finally I reached my goal, but this solution might not be scalable and may have some unwanted drawbacks (I am open to any constructive feedback and advice).
Basically, how I have solved my problem:
Log at ROW level BEFORE DML is performed into a table audit_rowlevel;
Aggregate the freshly added content of audit_rowlevel into audit_statementlevel at STATEMENT level AFTER DML is performed;
Minimal Working Example is now:
DROP TABLE IF EXISTS audit_rowlevel CASCADE;
CREATE TABLE audit_rowlevel(
Id BIGSERIAL NOT NULL
,Aggregated BOOLEAN NOT NULL DEFAULT(FALSE)
,TimeValue TIMESTAMP NOT NULL
-- https://www.postgresql.org/docs/current/static/functions-info.html
,RoleName NAME NOT NULL
,ClientIP INET NOT NULL
,ClientPid INTEGER NOT NULL
-- https://www.postgresql.org/docs/current/static/plpgsql-trigger.html
,Operation TEXT NOT NULL
,SchemaName NAME NOT NULL
,TableName NAME NOT NULL
,RowId BIGINT NOT NULL
-- https://www.postgresql.org/docs/current/static/functions-json.html
,OldValue JSONB
,NewValue JSONB
---
,PRIMARY KEY(Id)
);
-- Row Level Trigger:
DROP FUNCTION IF EXISTS audit_rowlevel_trigger() CASCADE;
CREATE OR REPLACE FUNCTION audit_rowlevel_trigger()
RETURNS TRIGGER AS
$BODY$
DECLARE
history BOOLEAN := (TG_NARGS > 0) AND (TG_ARGV[0]::BOOLEAN);
rowid BIGINT;
oldvalue JSONB;
newvalue JSONB;
BEGIN
-- Handle NEW:
IF TG_OP = ANY('{INSERT,UPDATE}') THEN
IF history THEN
newvalue := to_jsonb(NEW);
END IF;
rowid := NEW.Id::BIGINT;
END IF;
-- Handle OLD:
IF TG_OP = ANY('{UPDATE,DELETE}') THEN
IF history THEN
oldvalue := to_jsonb(OLD);
END IF;
rowid := OLD.Id::BIGINT;
END IF;
-- INSERT:
INSERT INTO audit_rowlevel(TimeValue, RoleName, ClientIP, ClientPID, Operation, SchemaName, TableName, RowId, NewValue, OldValue) VALUES
(now()::TIMESTAMP, current_user, inet_client_addr(), pg_backend_pid(), TG_OP, TG_TABLE_SCHEMA, TG_RELNAME, RowId, NewValue, OldValue);
-- RETURN:
IF TG_OP = ANY('{INSERT,UPDATE}') THEN
RETURN NEW;
ELSIF TG_OP = 'DELETE' THEN
RETURN OLD;
ELSE
RETURN NULL;
END IF;
END;
$BODY$
LANGUAGE plpgsql SECURITY DEFINER;
-- Statement Level Trigger:
DROP TABLE IF EXISTS audit_statementlevel CASCADE;
CREATE TABLE audit_statementlevel(
Id BIGSERIAL NOT NULL
,TimeValue TIMESTAMP NOT NULL
,RoleName NAME NOT NULL
,ClientIP INET NOT NULL
,ClientPid INTEGER NOT NULL
,Operation TEXT NOT NULL
,SchemaName NAME NOT NULL
,TableName NAME NOT NULL
,RowCount BIGINT NOT NULL
,RowIds BIGINT[] NOT NULL
,AuditIds BIGINT[] NOT NULL
---
,PRIMARY KEY(Id)
);
-- Row Level Trigger:
DROP FUNCTION IF EXISTS audit_statementlevel_trigger() CASCADE;
CREATE OR REPLACE FUNCTION audit_statementlevel_trigger()
RETURNS TRIGGER AS
$BODY$
DECLARE
rowcount BIGINT;
BEGIN
WITH
A AS (
SELECT
TimeValue, RoleName, ClientIP, ClientPid, Operation, SchemaName, TableName
,COUNT(*)
,array_agg(RowId)
,array_agg(Id)
FROM
audit_rowlevel
WHERE
NOT Aggregated
GROUP BY
TimeValue, RoleName, ClientIP, ClientPid, Operation, SchemaName, TableName
ORDER BY
TimeValue
),
B AS (
INSERT INTO audit_statementlevel(TimeValue, RoleName, ClientIP, ClientPid, Operation, SchemaName, TableName, RowCount, RowIds, AuditIds)
(SELECT * FROM A)
RETURNING AuditIds
),
C AS (
SELECT array_agg(DISTINCT T.id) AS Ids FROM B, unnest(B.AuditIds) AS T(id)
)
UPDATE
audit_rowlevel
SET
Aggregated = TRUE
FROM
C
WHERE
Id = ANY(C.Ids);
RETURN NULL;
END;
$BODY$
LANGUAGE plpgsql SECURITY DEFINER;
-- Channels:
DROP TABLE IF EXISTS channels CASCADE;
CREATE TABLE channels(
Id INTEGER NOT NULL
,UserKey TEXT NOT NULL
,Active BOOLEAN NOT NULL DEFAULT(TRUE)
---
,PRIMARY KEY(Id)
,UNIQUE(UserKey)
);
CREATE TRIGGER channel_audit_rowlevel_trigger BEFORE INSERT OR UPDATE OR DELETE ON channels
FOR EACH ROW EXECUTE PROCEDURE audit_rowlevel_trigger(TRUE);
CREATE TRIGGER channel_audit_statementlevel_trigger AFTER INSERT OR UPDATE OR DELETE ON channels
FOR EACH STATEMENT EXECUTE PROCEDURE audit_statementlevel_trigger();
-- Perform some operations:
INSERT INTO channels(
SELECT C.Id, 'Channel-' || C.Id
FROM generate_series(1, 300, 10) AS C(Id)
);
DELETE FROM channels WHERE id < 10;
UPDATE channels
SET UserKey = 'wild channel'
WHERE id = 21;
I am interested knowing if this solution looks right to professional developers.
Am I going into the good direction or is this solution evil?
I have a few db tables.
I want write universtal postgres function on copy rows to history tables
I have tables:
table1
table1_h
table2
table2_h
I wrote function (with help stackoverflow)
CREATE OR REPLACE FUNCTION copy_history_f() RETURNS TRIGGER AS
$BODY$
DECLARE
tablename_h text:= TG_TABLE_NAME || '_h';
BEGIN
EXECUTE 'INSERT INTO ' || quote_ident(TG_TABLE_SCHEMA) || '.' || quote_ident(tablename_h) || ' VALUES (' || OLD.* ||')';
RETURN NULL;
END;
$BODY$
LANGUAGE plpgsql VOLATILE;
And functions was create, but after update is error.
ERROR: syntax error at or near ","
ROW 1: ...RT INTO table1_h VALUES ((12,,,0,,"Anto...
I know where is error in this insert but I don't know how I repair that.
Structure tables table1 and table1_h are identical but table1_h has one more column (id_h)
Can you help me, how I have create psql function?
Thnak you.
drop table if exists t;
drop table if exists t_h;
drop function if exists ftg();
create table t(i serial, x numeric);
insert into t(x) values(1.1),(2.2);
create table t_h(i int, x numeric);
create function ftg() returns trigger language plpgsql as $ftg$
declare
tablename_h text:= TG_TABLE_NAME || '_h';
begin
execute format($q$ insert into %I.%I select $1.*; $q$, TG_TABLE_SCHEMA, tablename_h) using old;
return null;
end $ftg$;
create trigger tg_t after delete on t for each row execute procedure ftg();
delete from t where i = 1;
select * from t_h;
dbfiddle
Update It solves your problem, but I think that you want to have a bit more info in your history tables. It will be more complex a bit:
drop table if exists t;
drop table if exists t_h;
drop function if exists ftg();
create table t(i serial, x numeric);
insert into t(x) values(1.1),(2.2);
create table t_h(
hi serial, -- just ID
hd timestamp, -- timestamp
hu text, -- user who made changes
ha text, -- action
i int, x numeric
);
create function ftg() returns trigger language plpgsql as $ftg$
declare
tablename_h text:= TG_TABLE_NAME || '_h';
begin
execute format(
$q$
insert into %I.%I
select
nextval(%L || '_hi_seq'),
clock_timestamp(),
current_user,
%L,
$1.*
$q$, TG_TABLE_SCHEMA, tablename_h, tablename_h, TG_OP) using old;
return null;
end $ftg$;
create trigger tg_t after delete or update on t for each row execute procedure ftg();
update t set x = x * 2;
update t set x = x * 2 where i = 2;
delete from t where i = 1;
select * from t_h;
dbfiddle
I assume you are inserting the 'old' values from table1 into table1_h.
The additional column is your problem. When you using an insert without naming columns you must use a matching number and type for the insert.
You must use column referencing.
eg.
Insert into table1_h(column1, column2, column3)
values (a,b,c)
Consider a default value for the additional column in table table1_h.
I have a trigger function that copy row of unique values to another table on update or insert that ALMOST work.
The trigger should only insert a new row to the sample table if the number don't exist in it before. Atm. it insert a new row to the sample table with the value NULL if the number already exist in the table. I dont want it to do anything if maintbl.number = sample.nb_main
EDIT: sample table and sample data
CREATE TABLE schema.main(
sid SERIAL NOT NULL,
number INTEGER,
CONSTRAINT sid_pk PRIMARY KEY (sid)
)
CREATE TABLE schema.sample(
gid SERIAL NOT NULL,
nb_main INTEGER,
CONSTRAINT gid_pk PRIMARY KEY (gid)
Example and desired result
schema.main schema.sample
number nb_main
234233 234233
234234 555555
234234
555555
555555
CREATE OR REPLACE FUNCTION schema.update_number()
RETURNS trigger AS
$BODY$
BEGIN
INSERT INTO schema.sample(
nb_main)
SELECT DISTINCT(maintbl.number)
FROM schema.maintbl
WHERE NOT EXISTS (
SELECT nb_main FROM schema.sample WHERE maintbl.number = sample.nb_main);
RETURN NEW;
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION schema.update_number()
OWNER TO postgres;
CREATE TRIGGER update_number
AFTER INSERT OR UPDATE
ON schema.maintbl
FOR EACH ROW
EXECUTE PROCEDURE schema.update_number();
I just found out that my select query is probably wrong, if I run SELECT query by itself it return one row 'NULL' but i should not?
SELECT DISTINCT(maintbl.number)
FROM schema.maintbl
WHERE NOT EXISTS (
SELECT nb_main FROM schema.sample WHERE maintbl.number = sample.nb_main);
Any good advice?
Best
If I understood correctly, you wish to append to schema.sample a number that has been inserted or updated in schema.maintbl, right?
CREATE OR REPLACE FUNCTION schema.update_number()
RETURNS trigger AS
$BODY$
BEGIN
IF (SELECT COUNT(*) FROM schema.sample WHERE number = NEW.number) = 0 THEN
INSERT INTO schema.sample(nb_main) VALUES (NEW.number);
END IF;
RETURN NEW;
END;
$BODY$
LANGUAGE plpgsql VOLATILE;