pgbench benchmark rises using pgbouncer - postgresql

I've got Postgres 13 and PGbouncer.
When I make pgbench test, when I rise client connection and quantity of query,
tps also rises. Why?
For example:
pgbench -U postgres -h .... -p 6544 datafactory -c 700 -j 8 -t 50 -S
latency average = 1591.633 ms
tps = 439.799988 (including connections establishing)
tps = 442.163375 (excluding connections establishing)
pgbench -U postgres -h ... -p 6544 datafactory -c 700 -j 8 -t 100 -S
latency average = 1286.131 ms
tps = 544.268178 (including connections establishing)
tps = 545.953341 (excluding connections establishing)
pgbench -U postgres -h ... -p 6544 datafactory -c 700 -j 8 -t 300 -S
latency average = 1246.031 ms
tps = 561.783731 (including connections establishing)
tps = 562.399700 (excluding connections establishing)
Finally, if I lower amount of query, I get low level of tps:
pgbench -U postgres -h .. -p 6544 datafactory -c 700 -j 8 -t 10 -S
latency average = 8633.526 ms
tps = 81.079273 (including connections establishing)
tps = 81.465337 (excluding connections establishing)
So, I've got a question - why does the tps rise? I thought that if load increase, then tps should reduce.
If I connect to database directly on port 5433, not using PGbouncer, then if load increases, benchmark reduces
Additional information:
All 3 (pgbench, pgbouncer, db) are running on one machine.
When directly to port 5433, pgbouncer doesn't hold connection.
Here are examples of direct connection to database on port 5433:
pgbench -U postgres -h ... -p 5433 datafactory -c 250 -j 8 -t 500
latency average = 2155.394 ms
tps = 115.988055 (including connections establishing)
tps = 116.134037 (excluding connections establishing)
pgbench -U postgres -h ... -p 5433 datafactory -c 250 -j 8 -t 700
latency average = 835.555 ms
tps = 299.202467 (including connections establishing)
tps = 299.228977 (excluding connections establishing)
Here it fails
pgbench -U postgres -h ... -p 5433 datafactory -c 250 -j 8 -t 1000
WARNING: terminating connection because of crash of another server process
pgbench -U postgres -h ... -p 5433 datafactory -c 250 -j 8 -t 700
connection to database "datafactory" failed:
FATAL: the database system is in recovery mode
PGBOUNCER:
listen_port = 6544
listen_addr = '*'
auth_type = scram-sha-256
auth_file = /etc/pgbouncer/userlist.txt
auth_proxy = on
auth_failure_threshold = 3
auth_inactivity_period = 60
auth_last_size = 10
log_audit = 1
logfile = /pgerrorlogs/tkldd-ldisu0001/pgbouncer.log
pidfile = /var/run/pgbouncer/pgbouncer.pid
admin_users = pgbouncer
max_client_conn = 1000
pool_mode = transaction
min_pool_size = 0
default_pool_size = 30
max_db_connections = 30
max_user_connections = 30
ignore_startup_parameters = extra_float_digits

Related

PgBouncer - Server DNS lookup failed

I am trying to test PgBouncer connection with pgbench on a PostgreSQL server. But I'm getting error as closing because: server DNS lookup failed and at pgbench getting this message:
pgbench -c 10 -t 10 -C -f C:\Users\Administrator\Downloads\query.sql -U postgres -p 6432 -n tags
Password:
pgbench: error: connection to server at "localhost" (::1), port 6432 failed: FATAL: client_login_timeout (server down)
This is observed in PgBouncer log:
LOG C-01715ff0: tags/postgres#[::1]:52768 login attempt: db=tags user=postgres tls=no
WARNING DNS lookup failed: localhost: result=11001
LOG S-0174d218: tags/postgres#(bad-af):0 closing because: server DNS lookup failed (age=0s)
I'm running PostgreSQL-14.5 on Windows Server 2019
PgBouncer Config:
[databases]
postgres = host=localhost port=5435
tags = dbname=postgres host=localhost port=5435 user=postgres password=Admin#!23 auth_user=postgres
[pgbouncer]
logfile = C:\Program Files (x86)\PgBouncer\log\pgbouncer.log
pidfile = C:\Program Files (x86)\PgBouncer\log\pgbouncer.pid
listen_addr = *
listen_port = 6432
auth_type = md5
auth_file = C:\Program Files (x86)\PgBouncer\etc\userlist.txt
admin_users = postgres
stats_users = postgres
pool_mode = session
max_client_conn = 100
default_pool_size = 20
I tried solutions suggested in this post, like, setting
listen_addresses = '*' in postgresql.conf
but could not get the issue resolved. I also searched over the internet for the solution & did not find any.
Am I missing anything or please suggest a solution to this issue. Thanks in advance.
I can connect to Postgres DB after setting [databases] configuration in pgbouncer.ini like this : tags = host=127.0.0.1 port=5435 auth_user=postgres dbname=postgres.
Turns out the issue was with localhost not added in pg_hba.conf.

Cannot connect Barman to PostgreSQL 12

I have 2 ubuntu-20.04 VM on VMWARE with Postgres 12 installed on each
pgprimary on ip 192.168.1.131
pgbackup on ip 192.168.1.130
barman CLI tools are installed on pgprimary
barman is installed on pgbackup
I want to backup data from pgprimary on pgbackupsame 2 users as Postgress users
on each machine I created
2 Linux sudoist users
useradd barman
useradd streaming_barman
also created the same two user as Postgress users
createuser --superuser --replication -P barman
createuser --superuser --replication -P streaming_barman
here are relevant parts on the configuration files
On pgprimary
postgressql.conf
listen_addresses = '*' # what IP address(es) to listen on;
port = 5432
archive_mode = on
archive_command = 'cp %p /var/lib/postgresql/12/arc/%f'
wal_level = replica
restore_command = 'cp /var/lib/postgresql/12/arc/%f %p'
recovery_target_time = '2021-03-24 16:18:11.319298+05:30'
recovery_target_inclusive = false
pg_hba.conf
local all postgres peer
# TYPE DATABASE USER ADDRESS METHOD
local all all peer
host all all 127.0.0.1/32 md5
host all all ::1/128 md5
#local replication all peer
#host replication all 127.0.0.1/32 md5
#host replication all ::1/128 md5
# FOR TESTING
local replication all trust
host replication all 127.0.0.1/32 trust
host replication all ::1/128 trust
also did
firewall-cmd --permanent --add-port=5432/tcp
firewall-cmd --reload
========================
con
On pgbackup
sudo cat <<'EOF' >> /etc/barman.d/pgprimary.conf
[pgprimary]
description = "Example of PostgreSQL Database (Streaming-Only)"
conninfo = host=192.168.1.131 user=barman dbname=training
streaming_conninfo = host=192.168.1.131 user=streaming_barman dbname=training
backup_method = postgres
streaming_archiver = on
slot_name = barman
create_slot = auto
EOF
pg_hba.conf
cat <<'EOF' >>~/.pgpass
pgprimary:*:*:barman:barman
pgprimary:*:*:streaming_barman:barman
EOF
Then I did
barman cron
Output
Starting WAL archiving for server pgprimary
Starting streaming archiver for server pgprimary
barman check pgprimary
Then I get this error
[13643] barman.utils WARNING: Failed opening the requested log file. Using standard error instead.
Server pgprimary:
2021-10-30 21:39:15,982 [13643] barman.server ERROR: Check 'WAL archive' failed for server 'pgprimary'
WAL archive: FAILED (please make sure WAL shipping is setup)
2021-10-30 21:39:37,006 [13643] barman.postgres WARNING: Error retrieving PostgreSQL status: connection to server at "192.168.131" (192.168.0.131), port 5432 failed: Connection refused
2021-10-30 21:39:58,021 [13643] barman.server ERROR: Check 'check timeout' failed for server 'pgprimary'
check timeout: FAILED (barman check command timed out)
Why cannot connect barman to the server ?
UPDATE:
psql -h 192.168.1.131 -U barman -d training
Password for user barman:
psql (12.8 (Ubuntu 12.8-0ubuntu0.20.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
I also can connect to server via netstat

pg_restore: processing data for table "public.planet_osm_line" out of memory

I have made a dump of an open street map table and tried to restore it on another computer.
I already tried to tweak my Postgres settings but that didn't help.
Hope someone can help me out
The command I used to restore it:
pg_restore -Fc -h 127.0.0.1 -U postgres -W -d postgres -v /bin/planet_osm_line
the size of the "planet_osm_line" dump is 7.81GB
Postgres.conf
shared_buffers = 128MB
work_mem = 128MB
maintenance_work_mem = 128MB
max_locks_per_transaction = 64
max_pred_locks_per_transaction = 64
Hardware / OS
OS: centos 7
postgres 9.5 + postgis 2.2
(Memory) free -h
MEM: Total:15G Used:258M
Swap: Total: 7.9G Used:4.9M
(diskspace) df -h
Size: 292G Used: 57G
Thanks

pg_basebackup fails with " too many connections for role "replication""

I am trying to set up a standby server and keep getting this error. My primary server has more than enough connections to handle the load:
listen_addresses = '*'
wal_level = hot_standby
max_wal_senders = 10
max_connections=100
checkpoint_segments = 8
wal_keep_segments = 8
archive_mode = on
archive_command = 'cp %p /var/lib/postgresql/archive/%f'
This is the command that fails on the standby server:
pg_basebackup -h ${MASTER_PORT_5432_TCP_ADDR} -D ${PGDATA} -U ${REP_USER} -vPw --xlog-method=stream
I don't understand why this happens.

How to install PostgreSQL 9.1 on OSX Lion

I downloaded PostgreSQL from the official website and ran the .dmg installer. After that I downloaded pgadmin3 and I am indeed able to connect to the database.
when I run 'psql' I get the following error:
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
After hours of googling I read about some $PATH issues, so I put this into my .bashrc:
export PATH=/Library/PostgreSQL/9.1/bin:$PATH
However, this doesn't solve the error above at all. After some more hours of googling I tried to run 'psql -l localhost -U postgres'. This gives another error:
psql: could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (fe80::1) and accepting
TCP/IP connections on port 5432?
After some more googling I tried to edit /Library/PostgreSQL/9.1/data/pg_hba.conf and replaced all occurrences of 'md5' with 'trust'.
Then I changed user to postgres and executed 'pg_ctl stop' and 'pg_ctl start', switched back to my own user and tried to connect again, no luck.
Here are some more infos:
[~]$ which psql
/Library/PostgreSQL/9.1/bin/psql
ps aux | grep postgres
postgres 19022 0.0 0.0 2446096 484 ?? Ss 11:31PM 0:00.01 postgres: stats collector process
postgres 19021 0.0 0.0 2486532 1776 ?? Ss 11:31PM 0:00.01 postgres: autovacuum launcher process
postgres 19020 0.0 0.0 2486400 576 ?? Ss 11:31PM 0:00.03 postgres: wal writer process
postgres 19019 0.0 0.0 2486400 820 ?? Ss 11:31PM 0:00.05 postgres: writer process
postgres 19017 0.0 0.0 2446096 356 ?? Ss 11:31PM 0:00.01 postgres: logger process
postgres 19015 0.0 0.1 2486400 8216 s001 S 11:31PM 0:00.17 /Library/PostgreSQL/9.1/bin/postgres
sudo find / -name .s.PGSQL.5432
No file was found?!?
Update 1:
In /etc/sysctl.conf I added the values suggested by the installer's README:
kern.sysv.shmmax=1610612736
kern.sysv.shmall=393216
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8
kern.maxprocperuid=512
Before these settings, the installer quits with an error, afterwards the wizard appears and installs postgres (and again, using pgadmin3 works, so I assume that the database is running fine).
Run lsof on the master process to all of this. In your case it's 19015 (shown with my PID):
> sudo lsof -p 286 | awk '$5 == "unix" && $NF ~ /\// { print $NF }'
/tmp/.s.PGSQL.5432
You can leave off the awk, but basically it's getting the UNIX socket on which postgres is listening. From there, you can use the -h option to psql (but only include the directory).
> psql -h /tmp template1
template1=# \q
If that doesn't work, you can check the lsof output to show you what TCP port it's actually listening on, and if it's not 5432, use the -p option to postgres
If you are able to connect to the database through pgadmin3, then your database is running.
(1) I'd verify by running netstat -an | grep 5432 on the command line, as a paranoia check. You are in business i.e. you actually installed Postgresql if you get this below:
tcp4 0 0 *.5432 . LISTEN
tcp6 0 0 *.5432 . LISTEN
ffffff80133bfed8 stream 0 0 ffffff8013be8000 0 0 0 /tmp/.s.PGSQL.5432
(2) Speaking for myself, I found that just running psql gets me your psql error message. I had to run
psql -h 127.0.0.1 -p 5432 [-d database] -U postgres
to connect with the database. In other words, I had to explicitly specify IP and port. Since you were able to run psql and get a non-connect error message, the OS understands that psql is a command and you don't have a pathing issue.