My project is written with golang,I use postgresql Database and use pgbouncer as a database pool connection. I did a pressure test after I finished the basic function.
It runs well as test by the command(just 1 concurrency):
ab -n 1000 -c 1 -p '/home/mwh/postfile.txt' -T 'application/x-www-form-urlencoded' 'http://192.168.1.229:8090/hwSdk/tsk/get_t_k.json'
But when I
increase the concurrency:
ab -n 10000 -c 100 -p '/home/mwh/postfile.txt1 -T 'application/x-www-form-urlencoded' 'http://192.168.1.229:8090/hwSdk/tsk/get_t_k.json'
something happened error with database query in the app log:
[2017-06-01 12:04:00] (pq: bind message supplies 18 parameters, but prepared statement "" requires6)
[2017-06-01 12:04:00] (pq: bind message supplies 19 parameters, but prepared statement "" requires 3)
In the pgbouncer.log file
8035 2017-06-01 12:04:01.702 45619 LOG C-0xf9a4c0: cygnus/cygnus#192.168.1.229:40397 closing because: client close request (age=0)
8036 2017-06-01 12:04:01.703 45619 LOG C-0xf8b9f0: cygnus/cygnus#192.168.1.229:40406 closing because: client close request (age=0)
8037 2017-06-01 12:04:01.703 45619 LOG C-0xf8b248: cygnus/cygnus#192.168.1.229:40574 login attempt: db=cygnus user=cygnus tls=no
8038 2017-06-01 12:04:01.703 45619 LOG C-0xf8c198: cygnus/cygnus#192.168.1.229:40468 closing because: client close request (age=0
8039 2017-06-01 12:04:01.703 45619 LOG C-0xfac6f0: cygnus/cygnus#192.168.1.229:40504 closing because: client close request (age=0)
8040 2017-06-01 12:04:01.707 45619 LOG C-0xf9e698: cygnus/cygnus#192.168.1.229:40235 closing because: client unexpected eof (age=0)
my pgbouncer.ini settings:
pool_mode = Transaction
listen_port = 6543
listen_addr = *
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
logfile = /tmp/pgbouncer.log
pidfile = /tmp/pgbouncer.pid
stats_users = readonly,cygnus
max_client_conn = 2000
default_pool_size = 400
ignore_startup_parameters = extra_float_digits
what's the problem?
If you're using transaction mode in PgBouncer you have to set binary_parameters=yes to your database dsn connection
postgres://user:pass#host:post/db_name?binary_parameters=yes
Related
I am trying to test PgBouncer connection with pgbench on a PostgreSQL server. But I'm getting error as closing because: server DNS lookup failed and at pgbench getting this message:
pgbench -c 10 -t 10 -C -f C:\Users\Administrator\Downloads\query.sql -U postgres -p 6432 -n tags
Password:
pgbench: error: connection to server at "localhost" (::1), port 6432 failed: FATAL: client_login_timeout (server down)
This is observed in PgBouncer log:
LOG C-01715ff0: tags/postgres#[::1]:52768 login attempt: db=tags user=postgres tls=no
WARNING DNS lookup failed: localhost: result=11001
LOG S-0174d218: tags/postgres#(bad-af):0 closing because: server DNS lookup failed (age=0s)
I'm running PostgreSQL-14.5 on Windows Server 2019
PgBouncer Config:
[databases]
postgres = host=localhost port=5435
tags = dbname=postgres host=localhost port=5435 user=postgres password=Admin#!23 auth_user=postgres
[pgbouncer]
logfile = C:\Program Files (x86)\PgBouncer\log\pgbouncer.log
pidfile = C:\Program Files (x86)\PgBouncer\log\pgbouncer.pid
listen_addr = *
listen_port = 6432
auth_type = md5
auth_file = C:\Program Files (x86)\PgBouncer\etc\userlist.txt
admin_users = postgres
stats_users = postgres
pool_mode = session
max_client_conn = 100
default_pool_size = 20
I tried solutions suggested in this post, like, setting
listen_addresses = '*' in postgresql.conf
but could not get the issue resolved. I also searched over the internet for the solution & did not find any.
Am I missing anything or please suggest a solution to this issue. Thanks in advance.
I can connect to Postgres DB after setting [databases] configuration in pgbouncer.ini like this : tags = host=127.0.0.1 port=5435 auth_user=postgres dbname=postgres.
Turns out the issue was with localhost not added in pg_hba.conf.
I have a Greenplum deployment with several segments.
version: postgres (Greenplum Database) 8.2.15
A few days ago after an error as seen from pg_log (below)
FATAL 54000 out of on_shmem_exit slots
WARNING 1000 StartTransaction while in START state
PANIC XX000 Waiting on lock already held! (lwlock.c:557)
LOG 0 server process (PID 224596) was terminated by signal 6: Aborted
LOG 0 terminating any other active server processes
FATAL 57P01 terminating connection due to administrator command
LOG 0 sweeper process (PID 102635) exited with exit code 2
LOG 0 seqserver process (PID 102632) exited with exit code 2
FATAL 57P03 the database system is in recovery mode
LOG 0 ftsprobe process (PID 102633) exited with exit code 2
FATAL 57P03 the database system is in recovery mode
FATAL 57P03 the database system is in recovery mode
FATAL 57P03 the database system is in recovery mode
FATAL 57P03 the database system is in recovery mode
FATAL 57P03 the database system is in recovery mode
For 4 days now database remains in recovery mode, gpstart, gpstop all return with errors "the database system is in recovery mode" afterwards fails.
See ps response below:
[gpadmin#mdw1 ~]$ ps -ef | grep post
gpadmin 2979 189094 0 12:25 pts/0 00:00:00 grep post
postfix 3264 3251 0 2015 ? 00:34:40 qmgr -l -t fifo -u
gpadmin 102637 230099 0 May18 ? 00:01:47 postgres: port 5432, stats sender process
gpadmin 230099 1 0 Apr24 ? 02:47:53 /usr/local/greenplum-db-4.3.10.0/bin/postgres -D /data/master/gpseg-1 -p 5432 -b 1 -z 96 --silent-mode=true -i -M master -C -1 -x 194 -E
gpadmin 230100 230099 0 Apr24 ? 00:49:45 postgres: port 5432, master logger process
[gpadmin#mdw1 ~]$
I have searched a lot but am not able to find as a solution kindly assist with pointers on how to bring the database up.
I'm trying to do python manage.py syncdb on a Django installation, but I keep getting OperationalError: ERROR: pgbouncer cannot connect to server. pgbouncer.log contains lines such as:
2017-09-19 19:44:15.107 1128 LOG C-0x8a9930: mydb/myuser#unix:6432 closing because: pgbouncer cannot connect to server (age=0)
2017-09-19 19:44:15.107 1128 WARNING C-0x8a9930: mydb/myuser#unix:6432 Pooler Error: pgbouncer cannot connect to server
2017-09-19 19:44:15.107 1128 LOG S-0x8c72e0: mydb/myuser#35.154.149.188:5432 new connection to server
2017-09-19 19:44:15.107 1128 LOG C-0x8a9930: mydb/myuser#unix:6432 login failed: db=mydb user=myuser
2017-09-19 19:44:30.108 1128 LOG S-0x8c72e0: mydb/myuser#35.154.149.188:5432 closing because: connect failed (age=15)
In case needed, ps -aef | grep pgbouncer yields:
postgres 1128 1 0 18:38 ? 00:00:00 /usr/sbin/pgbouncer -d /etc/pgbouncer/pgbouncer.ini
myuser 1919 1533 0 19:45 pts/0 00:00:00 grep --color=auto pgbouncer
Moreover, grep port /etc/pgbouncer/pgbouncer.ini results in:
;; dbname= host= port= user= password=
mydb = host=xx.xxx.xxx.xxx port=5432 dbname=mydb
;forcedb = host=127.0.0.1 port=300 user=baz password=foo client_encoding=UNICODE datestyle=ISO connect_query='SELECT 1'
listen_port = 6432
Lastly, the relevant parts of settings.py contain:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'mydb',
'USER': 'myuser',
'PASSWORD': 'mypassword',
'HOST': '/var/run/postgresql',
'PORT': '6432',
}
I turned log_connections to on in postgresql.conf, restarted PG and tried again. Here's the relevant lines:
2017-09-20 07:50:59 UTC LOG: database system is ready to accept connections
2017-09-20 07:50:59 UTC LOG: autovacuum launcher started
2017-09-20 07:51:00 UTC LOG: connection received: host=[local]
2017-09-20 07:51:00 UTC LOG: incomplete startup packet
2017-09-20 07:51:00 UTC LOG: connection received: host=[local]
2017-09-20 07:51:00 UTC LOG: connection authorized: user=postgres database=postgres
2017-09-20 07:51:01 UTC LOG: connection received: host=[local]
2017-09-20 07:51:01 UTC LOG: connection authorized: user=postgres database=postgres
2017-09-20 07:51:01 UTC LOG: connection received: host=[local]
2017-09-20 07:51:01 UTC LOG: connection authorized: user=postgres database=postgres
It seems the connection is going through, but the user and database name is postgres. Those credentials aren't what I supplied in pgbouncer.ini.
However, explicitly adding myuser in the connection string described in pgbouncer.ini leads to:
2017-09-20 09:37:37 UTC FATAL: Peer authentication failed for user "myuser"
2017-09-20 09:37:37 UTC DETAIL: Connection matched pg_hba.conf line 90: "local all all peer"
Totally stumped.
It seems the mis-configuration emanated from this line in settings.py:
'PORT': '6432',
I commented it and pgbouncer started working.
Though I'm not sure 'why'.
Maybe there's a collision on this port; Pgbouncer and PG coexist on a single server in my case. I've set them up over different VMs in the past without a hitch (and without needing to comment 'PORT': '6432',)
I am trying to setup pgbouncer on my local machine. I have standard (didnt change anything after installation) configuration file with entries:
postgres = host=127.0.0.1 port=5432
listen_addr = *
listen_port = 6432
auth_type = trust //also tested with md5 and plain
My postgresql (ver 9.4) is running on port 5432. When I execute
psql -U postgres -p 5432 -d postgres
i can successfully connect. Now i am trying to connect to pgbouncer
psql -U postgres -p 6432 -d postgres
after providing password pgbouncer cannot connect (it hangs for 60 sec) and then timeouts with error
psql: ERROR: client_login_timeout (server down)
Pgbouncer logs:
2017-05-05 00:17:27.084 14696 LOG File descriptor limit: -1 (H:-1), max_client_conn: 100, max fds possible: 130
2017-05-05 00:17:27.104 14696 LOG listening on ::/6432
2017-05-05 00:17:27.105 14696 LOG listening on 0.0.0.0:6432
2017-05-05 00:17:27.106 14696 LOG process up: pgbouncer 1.7.2, libevent 2.0.21-stable (win32), adns: evdns2, tls: OpenSSL 1.0.2k 26 Jan 2017
2017-05-05 00:18:27.104 14696 LOG Stats: 0 req/s, in 0 b/s, out 0 b/s,query 0 us
2017-05-05 00:18:51.852 14696 LOG C-009B8FE0: postgres/postgres#[::1]:55878 login attempt: db=postgres user=postgres tls=no
2017-05-05 00:18:51.854 14696 WARNING
2017-05-05 00:18:51.854 14696 LOG S-009EF248: postgres/postgres#127.0.0.1:5432 closing because: connect failed (age=0)
2017-05-05 00:19:06.929 14696 WARNING
2017-05-05 00:19:06.929 14696 LOG S-009EF248: postgres/postgres#127.0.0.1:5432 closing because: connect failed (age=0)
2017-05-05 00:19:21.949 14696 WARNING
2017-05-05 00:19:21.950 14696 LOG S-009EF248: postgres/postgres#127.0.0.1:5432 closing because: connect failed (age=0)
2017-05-05 00:19:27.105 14696 LOG Stats: 0 req/s, in 0 b/s, out 0 b/s,query 0 us
2017-05-05 00:19:36.969 14696 WARNING
2017-05-05 00:19:36.970 14696 LOG S-009EF248: postgres/postgres#127.0.0.1:5432 closing because: connect failed (age=0)
2017-05-05 00:19:51.990 14696 LOG C-009B8FE0: postgres/postgres#[::1]:55878 closing because: client_login_timeout (server down) (age=60)
2017-05-05 00:19:51.991 14696 WARNING C-009B8FE0: postgres/postgres#[::1]:55878 Pooler Error: client_login_timeout (server down)
2017-05-05 00:20:27.105 14696 LOG Stats: 0 req/s, in 0 b/s, out 0 b/s,query 0 us
hba.conf
host all all 127.0.0.1/32 md5
host all all ::1/128 md5
postgresql.conf
listen_addresses = '*'
What I am doing wrong?
EDIT 1:
Struggling to make this work I tried:
Connecting to other db versions: 9.5 and 9.6
Since I saw in logs that various ports are used I opened whole pgbouncer app in firewall, before that I opened only 6432 port
I thought maybe pgbouncer has problems connecting to localhost so I tried to connect to remote server
Even disabled antivirus
None of above worked. Always the same log shows up (port 5434 is 9.5 version):
2017-05-05 22:26:01.899 8008 LOG C-010C8FF0: postgres/postgres#[::1]:61687 login attempt: db=postgres user=postgres tls=no
2017-05-05 22:26:01.899 8008 LOG C-010C8FF0: postgres/postgres#[::1]:61687 closing because: client unexpected eof (age=0)
2017-05-05 22:26:04.753 8008 LOG C-010C8FF0: postgres/postgres#[::1]:61690 login attempt: db=postgres user=postgres tls=no
2017-05-05 22:26:04.753 8008 WARNING
2017-05-05 22:26:04.753 8008 LOG S-010FF258: postgres/postgres#127.0.0.1:5434 closing because: connect failed (age=0)
2017-05-05 22:26:19.803 8008 WARNING
2017-05-05 22:26:19.803 8008 LOG S-010FF258: postgres/postgres#127.0.0.1:5434 closing because: connect failed (age=0)
2017-05-05 22:26:35.086 8008 WARNING
2017-05-05 22:26:35.086 8008 LOG S-010FF258: postgres/postgres#127.0.0.1:5434 closing because: connect failed (age=0)
2017-05-05 22:26:41.581 8008 LOG Stats: 0 req/s, in 0 b/s, out 0 b/s,query 0 us
2017-05-05 22:26:50.359 8008 WARNING
2017-05-05 22:26:50.359 8008 LOG S-010FF258: postgres/postgres#127.0.0.1:5434 closing because: connect failed (age=0)
2017-05-05 22:27:04.961 8008 LOG C-010C8FF0: postgres/postgres#[::1]:61690 closing because: client_login_timeout (server down) (age=60)
2017-05-05 22:27:04.961 8008 WARNING C-010C8FF0: postgres/postgres#[::1]:61690 Pooler Error: client_login_timeout (server down)
Can someone explain why it tries to connect on port 61687? On this port it gets unexpected eof.
Here is the whole pgbouncer.ini (lines that are not commented out):
[databases]
postgres = host=127.0.0.1 port=5434 dbname=postgres
[pgbouncer]
logfile = C:\Program Files\PostgreSQL\PgBouncer\log\pgbouncer.log
pidfile = C:\Program Files\PostgreSQL\PgBouncer\log\pgbouncer.pid
listen_addr = *
listen_port = 6432
auth_type = md5
auth_file = C:\Program Files\PostgreSQL\PgBouncer\etc\userlist.txt
admin_users = postgres
stats_users = postgres
pool_mode = session
max_client_conn = 100
default_pool_size = 20
For me it was an issue of Port number in the [Database] section.
The default (if not set) is 5432.
On my machine, Ubuntu 22 and PostgreSQL 15 the port is 5433.
I set it explicitly in the [Database] section, now it works perfectly.
Im trying to get pgbouncer to work on my standalone postgresql server, but when I try to connect to the database using the pgbouncer port, I get an timeout error.
Postgresql: 9.1.8
pgBouncer: 1.4.2
OS: Ubuntu 12.04.2 LTS (GNU/Linux 3.8.4-x86_64-linode31 x86_64)
Here is my conf file:
[databases]
postgres = host=127.0.0.1 port=5432 dbname=postgres
mydb = host=127.0.0.1 port=5432 dbname=mydb
[pgbouncer]
logfile = /home/username/pg_log/pgbouncer.log
pidfile = /tmp/pgbouncer.pid
listen_addr = xxx.xxx.xxx.xxx
listen_port = 6432
unix_socket_dir = /var/run/postgresql
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
admin_users = postgres
stats_users = stats, root
user = postgres
pool_mode = session
server_reset_query = DISCARD ALL
max_client_conn = 1000
default_pool_size = 20
log_connections = 1
log_disconnections = 1
log_pooler_errors = 1
Im trying to test the connection like this:
psql -p 6432 -U postgres mydb
Here is my log file:
28396 LOG listening on xxx.xxx.xxx.xxx:6432
28396 LOG listening on unix:/var/run/postgresql/.s.PGSQL.6432
28396 LOG process up: pgbouncer 1.4.2, libevent 2.0.16-stable (epoll), adns: evdns2
28864 LOG C-0x23f7890: mydb/postgres#unix:6432 login attempt: db=mydb user=postgres
28864 LOG C-0x23f7890: mydb/postgres#unix:6432 closing because: client unexpected eof (age=0)
28864 LOG C-0x23f7890: mydb/postgres#unix:6432 login attempt: db=mydb user=postgres
28864 LOG S-0x2415240: mydb/postgres#127.0.0.1:5432 new connection to server
28864 LOG S-0x2415240: mydb/postgres#127.0.0.1:5432 closing because: connect failed (age=0)
28864 LOG S-0x2415240: mydb/postgres#127.0.0.1:5432 new connection to server
28864 LOG S-0x2415240: mydb/postgres#127.0.0.1:5432 closing because: connect failed (age=0)
28864 LOG S-0x2415240: mydb/postgres#127.0.0.1:5432 new connection to server
28864 LOG S-0x2415240: mydb/postgres#127.0.0.1:5432 closing because: connect failed (age=0)
28864 LOG S-0x2415240: mydb/postgres#127.0.0.1:5432 new connection to server
28864 LOG S-0x2415240: mydb/postgres#127.0.0.1:5432 closing because: connect failed (age=0)
28864 LOG Stats: 0 req/s, in 0 b/s, out 0 b/s,query 0 us
28864 LOG C-0x23f7890: mydb/postgres#unix:6432 closing because: client_login_timeout (server down) (age=60)
28864 WARNING C-0x23f7890: mydb/postgres#unix:6432 Pooler Error: client_login_timeout (server down)
28864 LOG Stats: 0 req/s, in 0 b/s, out 0 b/s,query 0 us
28864 LOG Stats: 0 req/s, in 0 b/s, out 0 b/s,query 0 us
Is there something else I need to change to make pgbouncer work?
Change
listen_addresses = 'private_ip'
to
listen_addresses = '*'
In the postgresql.conf file.
Looks like you are trying to connect using a Unix socket. Depending on your Postgres installation, the socket may have been created in /tmp rather than /var/run/postgres. I've run into this same issue and it has been discussed here.
Try changing the pgbouncer.ini config to this:
unix_socket_dir = /tmp
You could also try just connecting to localhost by adding the -h as follows:
psql -p 6432 -h localhost -U postgres mydb
Hope this helps!
For me the issue seemed to be when I defined the database in the [databases] section of the config file; I was not able to use host=127.0.0.1 or host=localhost, but I had to use the actual IP, host=192.168.0.1. This could be due to a misconfigured postgresql pg_hba.conf file, but I couldn't find a way to make it work with local IPs.