Capturing changes occurring on tables? - postgresql

I want to see what tables and fields in PostgreSQL that a module in OpenERP is changing/updating when it runs.
Any suggestions?
application_name | pgAdmin III - Query Tool | client
bytea_output | escape | session
client_encoding | UNICODE | session
client_min_messages | notice | session
DateStyle | ISO,MDY | session
default_text_search_config | pg_catalog.english | configuration file
lc_messages | English_United States.1252 | configuration file
lc_monetary | English_United States.1252 | configuration file
lc_numeric | English_United States.1252 | configuration file
lc_time | English_United States.1252 | configuration file
listen_addresses | * | configuration file
log_destination | csvlog | configuration file
log_line_prefix | %t | configuration file
log_timezone | US/Pacific | configuration file
logging_collector | on | configuration file
max_connections | 100 | configuration file
max_stack_depth | 2MB | environment variable
port | 5432 | configuration file
shared_buffers | 32MB | configuration file
TimeZone | US/Pacific | configuration file

If OpenERP is using a specific ROLE when it connects ("openerp" is used in the example), you can log the statements a couple of different ways:
1). ALTER ROLE openerp SET log_min_duration_statement TO 0;
2). ALTER ROLE openerp SET log_statement TO 'mod';
My preference is option #1, but you might want to try both.
To revert the settings to defaults:
1). ALTER ROLE openerp SET log_min_duration_statement TO DEFAULT;
2). ALTER ROLE openerp SET log_statement TO DEFAULT; -- or 'none'
To see what the current settings are (when not set via ROLE), paste the results of the following query:
SELECT name,
current_setting(name) AS current_setting,
source
FROM pg_settings
WHERE source <> ALL (ARRAY['default'::text, 'override'::text]);

Related

What data entered into postgres

This is a web application which uses Postgres to store data pushed from various modules of the web application.
From Postgres side, how can I know what data entered/modified into Postgres? I mean if there is any Postgres data logging?
Postgres has 4-5 schemas & each schema has 2-3 tables. Each table has 10-20 records.
EDIT (8 Sept 2021):
log_destination=csvlog - I tried different destinations to see if I'm getting the required logs. But all log destination are capturing the same logs.
2: reload - Did restarted server every time I made
changes to postgresql.conf.
3: The logs are being generated at
/var/lib/postgresql/data/pg_log directory with format
postgresql-%Y-%m-%d_%H%M%S.log. Where latest logs are generated, but as mentioned in comment, I'm not getting all the query logs that are being executed on Postgres by the web application
4: I installed Postgres 9.6 as a Docker container.
5: I'm making changes in
/var/lib/postgresql/data/postgresql.conf which is reflected in
database query SELECT name, setting FROM pg_settings WHERE name LIKE '%log%'; after restart.
EDIT (7 Sept 2021):
name |setting |
---------------------------+------------------------------+
log_autovacuum_min_duration|-1 |
log_checkpoints |off |
log_connections |off |
log_destination |csvlog |
log_directory |pg_log |
log_disconnections |off |
log_duration |off |
log_error_verbosity |default |
log_executor_stats |off |
log_file_mode |0600 |
log_filename |postgresql-%Y-%m-%d_%H%M%S.log|
log_hostname |off |
log_line_prefix | |
log_lock_waits |off |
log_min_duration_statement |-1 |
log_min_error_statement |error |
log_min_messages |warning |
log_parser_stats |off |
log_planner_stats |off |
log_replication_commands |off |
log_rotation_age |1440 |
log_rotation_size |10240 |
log_statement |all |
log_statement_stats |off |
log_temp_files |-1 |
log_timezone |Etc/UTC |
log_truncate_on_rotation |off |
logging_collector |on |
syslog_facility |local0 |
syslog_ident |postgres |
syslog_sequence_numbers |on |
syslog_split_messages |on |
wal_log_hints |off |
EDIT
Check a the logging configuration in postgresql.conf. You want to set up this What to log in particular log_statement. For your use the mod setting would be appropriate. Then you can look at the Postgres log to see what was changed and when. You might also want to set log_connections and log_disconnections to see what user the connection is running as.

Upgrading Postgres from 10 to 12 -- problem with encodings

I'm trying to upgrade a postgres server from 10 12, and am having problems with the encodings. I'm following what I believe to be established recipes.
Behold:
postgres#serverbot:~$ psql -l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-------------+----------+-----------+---------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | postgres=CTc/postgres+
| | | | | =c/postgres
thingsboard | postgres | SQL_ASCII | C | C | =Tc/postgres +
| | | | | postgres=CTc/postgres+
| | | | | nagios=c/postgres
(4 rows)
Notice the encodings on the 10 database. Time to create the database for 12.
postgres#serverbot:~$ sudo service postgresql stop
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/initdb -E SQL_ASCII --locale=C -D /var/lib/postgresql/12/data
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /var/lib/postgresql/12/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/New_York
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/usr/lib/postgresql/12/bin/pg_ctl -D /var/lib/postgresql/12/data -l logfile start
Excellent! Let's fire up the new server.
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_ctl -D /var/lib/postgresql/12/data -l logfile start
waiting for server to start.... done
server started
And verify the encodings...
postgres#serverbot:~$ psql -l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+-----------+---------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
Everything matches... time to upgrade!
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_upgrade \
--old-datadir=/var/lib/postgresql/10/main \
--new-datadir=/var/lib/postgresql/12/main \
--old-bindir=/usr/lib/postgresql/10/bin \
--new-bindir=/usr/lib/postgresql/12/bin \
--old-options '-c config_file=/etc/postgresql/10/main/postgresql.conf' \
--new-options '-c config_file=/etc/postgresql/12/main/postgresql.conf' \
--link --check
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for tables WITH OIDS ok
Checking for invalid "sql_identifier" user columns ok
encodings for database "postgres" do not match: old "SQL_ASCII", new "UTF8"
Failure, exiting
postgres#serverbot:~$
Doh!
What is wrong here? I assert that the encodings do match, and I'm stuck.
Can anyone offer any advice?
What I see is:
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_ctl -D /var/lib/postgresql/12/data -l logfile start
waiting for server to start.... done
server started
and then:
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_upgrade \
--old-datadir=/var/lib/postgresql/10/main \
--new-datadir=/var/lib/postgresql/12/main \
[...]
Note the $PGDATA directories. The database cluster you did the initdb for is different from the one you are doing the pg_upgrade to.
UPDATE. Since you seem to be using a Debian based OS and it's Postgres packaging, it might work better to stick with the packaging tools:
sudo pg_createcluster --locale=C 12 ascii
Creating new PostgreSQL cluster 12/ascii ...
/usr/lib/postgresql/12/bin/initdb -D /var/lib/postgresql/12/ascii --auth-local peer --auth-host md5 --locale C
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/12/ascii ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/Los_Angeles
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctlcluster 12 ascii start
Ver Cluster Port Status Owner Data directory Log file
12 ascii 5434 down postgres /var/lib/postgresql/12/ascii /var/log/postgresql/postgresql-12-ascii.log
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+-----------+---------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
Also I would use pg_upgradecluster, see here for more information. This keeps everything in the same system.

Setting up production postgres (createdb permissions)

I'm setting up postgres on DigitalOcean. I've set PG envs. The 9s are obviously redacted data. To facilitate the ssl connection, ~/.postgresql/root.crt exists.
# env | grep PG
PGPORT=99999
PGPASSWORD=9999999999999
PGSSLMODE=require
PGUSER=doadmin
PGDATABASE=auth_service_prod
PGHOST=private-db-postgresql-sfo9-99999-do-user-9999999-9.a.db.ondigitalocean.com
When calling createdb, it fails:
# createdb
createdb: could not connect to database template1: FATAL: pg_hba.conf rejects connection for host "10.999.9.99", user "doadmin", database "template1", SSL on
I can connect to psql, though. I don't know enough about configuring postgres to allow doadmin to create a db, but hopefully here is some useful info:
# psql
psql (11.7 (Debian 11.7-0+deb10u1), server 11.8)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
auth_service_prod=> \du
List of roles
Role name | Attributes | Member of
-----------+------------------------------------------------------------+-----------
_dodb | Superuser, Replication | {}
doadmin | Create role, Create DB, Replication, Bypass RLS | {}
postgres | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
auth_service_prod=> \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-------------------+----------+----------+-------------+-------------+-----------------------
_dodb | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
auth_service_prod | doadmin | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
defaultdb | doadmin | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(5 rows)
DigitalOcean doesn't allow createdb from postgres, apparently. I removed the code creating databases, and created them "manually" from their GUI. The app is now only responsible for managing the schema, not the database(s).

Can't get new postgres config file settings to take effect

I have a somewhat large table in my database and I am inserting new records to it. As the number of records grow, I started having issues and can't insert.
My postgresql log files suggest I increase WAL size:
[700] LOG: checkpoints are occurring too frequently (6 seconds apart)
[700] HINT: Consider increasing the configuration parameter "max_wal_size".
I got the path to my config file with =# show config_file; and made some modifications with vim:
max_wal_senders = 0
wal_level = minimal
max_wal_size = 4GB
When I check the file I see the changes I made.
I then tried reloading and restarting the database:
(I get the data directory with =# show data_directory ;)
I tried reload:
pg_ctl reload -D path
server signaled
I tried restart
pg_ctl restart -D path
waiting for server to shut down.... done
server stopped
waiting for server to start....
2020-01-17 13:08:19.063 EST [16913] LOG: listening on IPv4 address
2020-01-17 13:08:19.063 EST [16913] LOG: listening on IPv6 address
2020-01-17 13:08:19.079 EST [16913] LOG: listening on Unix socket
2020-01-17 13:08:19.117 EST [16914] LOG: database system was shut down at 2020-01-17 13:08:18 EST
2020-01-17 13:08:19.126 EST [16913] LOG: database system is ready to accept connections
done
server started
But when I connect to the database and check for my settings:
name | setting | unit | category | short_desc | extra_desc | context | vartype | source | min_val | max_val | enumvals | boot_val | reset_val | sourcefile | sourceline | pending_restart
-----------------+---------+------+-------------------------------+-------------------------------------------------------------------------+------------+------------+---------+---------+---------+------------+---------------------------+----------+-----------+------------+------------+-----------------
max_wal_senders | 10 | | Replication / Sending Servers | Sets the maximum number of simultaneously running WAL sender processes. | | postmaster | integer | default | 0 | 262143 | | 10 | 10 | | | f
max_wal_size | 1024 | MB | Write-Ahead Log / Checkpoints | Sets the WAL size that triggers a checkpoint. | | sighup | integer | default | 2 | 2147483647 | | 1024 | 1024 | | | f
wal_level | replica | | Write-Ahead Log / Settings | Set the level of information written to the WAL. | | postmaster | enum | default | | | {minimal,replica,logical} | replica | replica | | | f
(3 rows)
I still see the old default settings.
What am I missing here? How can I get these settings to take effect?
Configuration settings can come from several sources:
postgresql.conf
postgresql.auto.conf (set with ALTER SYSTEM)
command line arguments at server start
set with ALTER DATABASE or ALTER USER
Moreover, if a parameter occurs twice in a configuration file, the second entry wins.
To figure out from where in this mess your setting originates, run
SELECT name, source, sourcefile, sourceline, pending_restart
FROM pg_settings
WHERE name IN ('wal_level', 'max_wal_size', 'max_wal_senders');
If the source is database or user, you can user the psql command \drds to figure out details.
The result of the queries shows that your PostgreSQL has been modified or built so that these values are the default values.
You'd have to override these defaults with any of the methods shown above.
Locations of config files. Ordered by priority.
/var/lib/postgresql/12/main/postgresql.auto.conf
/etc/postgresql/12/main/postgresql.conf

FATAL: password authentication when bucardo add tables

System specification:
Ubuntu 14.04
bucardo version 4.99.10
psql (PostgreSQL) 9.3.14
What I have tried so far?
I have created master database with openerp postgres user.
- Setup Source Database
export SOURCE_HOST=localhost
export SOURCE_PORT=5432
export SOURCE_DATABASE=master
export SOURCE_USERNAME=openerp
export SOURCE_PASSWORD=openerp
- Setup Destination Database
export DEST_HOST=localhost
export DEST_PORT=5432
export DEST_DATABASE=slave
export DEST_USERNAME=openerp
export DEST_PASSWORD=openerp
- Finally, the tables to be migrated
export TABLES="-t res_partner"
export TABLES_WITH_SPACES="res_partner"
- Set up the Destination Database
cat >> $HOME/.pgpass <<EOL
$DEST_HOST:$DEST_PORT:$DEST_DATABASE:$DEST_USERNAME:$DEST_PASSWORD
$SOURCE_HOST:$SOURCE_PORT:$SOURCE_DATABASE:$SOURCE_USERNAME:$SOURCE_PASSWORD
EOL
- Create schema.sql
- Create Slave database
- Load schema into Slave
- Setting up Bucardo Sync (Source Database and Destination Database)
- Now I am trying to add the tables we want to migrate and add them to a "copying herd"
./bucardo add tables res_partner db=master
I got following error:
openerp#odedra-Lenovo-Z50-70:/usr/local/src/bucardo$ ./bucardo add tables res_partner db=master
DBI connect('dbname=master;host=localhost;port=5432','openerp',...) failed: FATAL: password authentication failed for user "openerp"
FATAL: password authentication failed for user "openerp" at ./bucardo line 8632.
I have analytic script bucardo. I have confusion with dbuser and dbpass variable declared at line number 84, 85.
(I have tried with dbuser = 'openerp' and dbpass='openerp' but result same)
My pg_hba.conf
# Database administrative login by Unix domain socket
local all postgres ident
# TYPE DATABASE USER ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all md5
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
#local replication postgres peer
#host replication postgres 127.0.0.1/32 md5
#host replication postgres ::1/128 md5
List of database:
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-------------+----------+----------+-------------+-------------+-----------------------
bucardo | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
master | openerp | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
slave | openerp | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
slave_copy | openerp | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
slave_copy1 | openerp | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
slave_copy2 | openerp | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
May you guide me what should I change to fix above error or helpful related links?
How can I achieve master-slave functionality using Bucardo ?
I have found that have problem with pg_hba.conf file. To resolve error, I have added below line to that file.
local all openerp ident