I am looking for a way to audit DML changes which logs the old and new data along with the date and editor information, with editor being the logged in user.
With Spring boot for Java and PostgreSQL database, I had a look at the extension Cyan Audit The logged in user here however is the database connection user.
I have previously used a separate table to log the audit data using trigger on INSERT, UPDATE, DELETE. But the table was overloaded with data.
I need to keep the audit data for 7 years before I can start deleting old data.
Is there any better way of handling this?
Related
How can I (using PG-Admin) access or create a log (ideally timestamped) that displays the changes that have happened to a table?
First of the way enable pg_stat_statements on PostgreSQL.
create extension pg_stat_statements;
After then you can view all SQL queries executed on your DB. At this you can view when executed SQL, when finished, duration of execute times and etc.
For more details: PostgreSQL Documentation - pg_stat_statements
If you need history of updated or deleted records on the tables, you can do it for all tables manually writing triggers or own functions, using JSON(JSONB) data types.
I have a local database which is the production database, on which all operations are being done real time. I am storing the log on each action in an audit log table in another database via trigger. It basically checks if any change is made in any of the row's column it will remove that row and add it AGAIN (which is not a good way I think as it should simply update it but due to some reasons I need to delete and add it again).
There are some tables on which operations are being done rapidly like 100s of rows are being added in database. This is slowing the process of saving the data into audit log table. Now if trigger has to like delete 100 rows and add 100 again it will affect the performance obviously and if number of rows increases it will reduce the performance more.
What should be the best practice to tackle this, I have been looking into Read Replica and Foreign Data Wrapper but as for Read Replica it's only Readable and not writable for PostgreSQL and I don't really get to know how Foreign Data Wrapper gonna help me as this was suggested by one of my colleague.
Hope someone can guide me in right direction.
A log is append-only by definition. Loggers should never be modifying or removing existing entries.
Audit logs are no different. Audit triggers should INSERT an entry for each change (however you want to define "change"). They should never UPDATE or DELETE anything*.
The change and the corresponding log entry should be written to the same database within the same transaction, to ensure atomicity/consistency; logging directly to a remote database will always leave you with a window where the log is committed but the change is not (or vice versa).
If you need to aggregate these log entries and push them to a different database, you should do it from an external process, not within the trigger itself. If you need this to happen in real time, you can inform the process of new changes via a notification channel.
* In fact, you should revoke UPDATE/DELETE privileges on the audit table from the user inserting the logs. Furthermore, the trigger should ideally be a SECURITY DEFINER function owned by a privileged user with INSERT rights on the log table. The user connecting to the database should not be given permission to write to the log table directly.
This ensures that if your client application is compromised (whether due to a malfunction, or a malicious user e.g. exploiting an SQL injection vulnerability), then your audit log retains a complete and accurate record of everything it changed.
I am working on PostgreSQL database and after some months I have added new columns in many tables according to the new requirements on my test server and also I have done changes in functions.
Now what I want that I want to do same changes on my Live server without losing data.
I have taken backup on test server schema using below answer
https://stackoverflow.com/a/7804825/7223676
Is it possible to have a MS access backend database (Microsoft JET or Access Database Engine) set up so that whenever entries are inserted/updated those changes are replicated* to a PostgreSQL database?
Two-way synchronization would be nice, but one way would be acceptable.
I know it's popular to link the two and use one as a frontend, but it's essential that both be backend.
Any suggestions?
* ie reflected, synchronized, mirrored
Can you use Microsoft SQL Server Express Edition? Or do you have to use Microsoft Access Database Engine? It's possible you'll have more options using MS SQL express, like more complete triggers and logging.
Either way, you're going to need a way to accumulate a log of changed rows from the source database engine, and a program to sync them to PostgreSQL by reading the log and converting it into suitable PostgreSQL INSERT, UPDATE and DELETE statements.
You could do this by having audit triggers in MADB/Express insert a row into an audit shadow table for every "real" table whenever it changed, including inserting special "row deleted" audit entries. Then your sync program could connect to both MADB/Express, read the audit tables, apply the changes to PostgreSQL, and empty the audit tables.
I'll be surprised if you find anything to do this out of the box. It's one area where Microsoft SQL Server has a big advantage because of all the deep Access and MADB engine integation to support the synchronisation and integration features.
There are some ETL ("Extract, Transform, Load") tools that might be helpful, like Pentaho and Talend. I don't know if you can achieve the desired degree of automation with them though.
We have DB that has massive amount of business logic stored in triggers inside DB. Is there a way to log firing of triggers along with arguments that they have been fired, and what they have changed?
I saw a lot of tutorials on how to do table audit with triggers, but I would like to audit triggers not tables :)
Take one of the examples that do table auditing with triggers. Use their approach to extract the changed data, but do not write the data into an audit table, but use it for a RAISE NOTICE.
That notice will then be written to the PostgreSQL log file if you set the logging configuration correctly (log_min_messages = notice)
See the manual for details on RAISE: http://www.postgresql.org/docs/current/static/plpgsql-errors-and-messages.html