What's the difference between ORM/query builder library connection pool size and pgbouncer connection pool size? - postgresql

I am confused about pgbouncer pool size configuration and ORM(like sequelize.js), query builder(like knex.js) library pool size configuration. The architecture like this:
Application code => pgbouncer => postgresql
pgbouncer.ini:
;; ...
;; Default pool size. 20 is good number when transaction pooling
;; is in use, in session pooling it needs to be the number of
;; max clients you want to handle at any moment
;default_pool_size = 20
;; ...
sequelize connection pool configuration:
const sequelize = new Sequelize(/* ... */, {
// ...
pool: {
max: 5,
min: 0,
acquire: 30000,
idle: 10000
}
});
knex.js connection pool configuration:
var knex = require('knex')({
client: 'mysql',
connection: {
host : '127.0.0.1',
user : 'your_database_user',
password : 'your_database_password',
database : 'myapp_test'
},
pool: { min: 0, max: 7 }
});
What happened if I use sequelize.js connection pool configuration and pgbouncer connection pool size configuration together? Which configuration does the database server use? Should I use only one of them? Thanks.

If you have for example 3 application processes running knex or sequelize, then you should setup pgbouncer poolsize to be 3 times bigger than single knex / sequelize pool uses.
Then you also need to make sure that postgres server also has enough connections configured to handle pgbouncer connections.
Though as #jjanes said. There is no reason to use pgbouncer with knex / sequelize, because they already provide pooling. I suppose pgbouncer is meant to be used with frameworks, which doesn't support pooling. For example if PHP or cgi stript reinitializes on every page load and makes calls to database.

It rarely makes sense to daisy chain connection pools together. So there is probably no point in using pgbouncer in addition to the built-in ones. The database server doesn't know about your connection poolers except to the extent the pool manager sends its own explicit commands to the database, and it has its own configuration file which it uses (postgresql.conf).

Related

pg-promise not reconnecting clients in pool after broken connection

I'm using pg-promise (and the underlying pg-pool lib).
When I pull the plug on my pg database briefly (break TCP connections), I get connect ETIMEDOUT errors (expected) however it is taking a very long time for the pool to re-establish a connection. Like in the order of 10 minutes.
Here is a snippet for how I am using pg-promise:
const pgp = pgPromise({
error(err, e) {
if (e.query) {
// Do stuff
}
},
});
...
const pgpArgs = {
host,
port,
database,
user,
password,
max,
idleTimeoutMillis: 5000,
};
const db = pgp(pgpArgs);
I saw this potential answer from the author, but I don't want to kill my entire process just to re-establish the pool.
Is there a way to force clients in pool to re-connect, other than restarting the nodejs process?
Thanks in advance.

is it correct parameters for pgbouncer.ini and postgresql.conf?

I have pgbouncer.ini file with the below configuration
[databases]
test_db = host=localhost port=5432 dbname=test_db
[pgbouncer]
logfile = /var/log/postgresql/pgbouncer.log
pidfile = /var/run/postgresql/pgbouncer.pid
listen_addr = 0.0.0.0
listen_port = 5433
unix_socket_dir = /var/run/postgresql
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
admin_users = postgres
#pool_mode = transaction
pool_mode = session
server_reset_query = RESET ALL;
ignore_startup_parameters = extra_float_digits
max_client_conn = 25000
autodb_idle_timeout = 3600
default_pool_size = 250
max_db_connections = 250
max_user_connections = 250
and I have in my postgresql.conf file
max_connections = 2000
does it effect badly on the performance ? because of max_connections in my postgresql.conf ? or it doesn't mean anything and already the connection handled by the pgbouncer ?
one more question. in pgpouncer configuration, does it right listen_addr = 0.0.0.0 ? or should to be listen_addr = * ?
Is it better to set default_pool_size on PgBouncer equal to the number of CPU cores available on this server?
Shall all of default_pool_size, max_db_connections and max_user_connections to be set with the same value ?
So the idea of using pgbouncer is to pool connections when you can't afford to have a higher number of max_connections in PG itself.
NOTE: Please DO NOT set max_connections to a number like 2000 just like that.
Let's start with an example, if you have a connection limit of 20 and then your app or organization wants to have a 1000 connections at a given time, that is where pooler comes into picture and in this specific case you want the 20 connections to pool that 1000 coming in from the application.
To understand how it actually works let's take a step back and understand what happens when you do not have a connection pooler and only rely on PG config setting for the max connections which in our case is 20.
So when a connection comes in from a client\application etc. the main process of postgresql(PID, i.e. parent ID) spawns a child for that. So each new connection spawns a child process under the main postgres process, like so:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24379 postgres 20 0 346m 148m 122m R 61.7 7.4 0:46.36 postgres: sysbench sysbench ::1(40120)
24381 postgres 20 0 346m 143m 119m R 62.7 7.1 0:46.14 postgres: sysbench sysbench ::1(40124)
24380 postgres 20 0 338m 137m 121m R 57.7 6.8 0:46.04 postgres: sysbench sysbench ::1(40122)
24382 postgres 20 0 338m 129m 115m R 57.4 6.5 0:46.09 postgres: sysbench sysbench ::1(40126)
So now once a connection request is sent, it is received by the POSTMASTER process and creates a child process at OS level under the main parent process. This connection then has a life span of "unlimited" unless close by the application or you have a time out set for idle connections in postgresql.
Now here comes the situation where it can be a very costly affair to manage the connections with a given compute, if they exceed a certain limit. Meaning n number of connections when served have a given compute cost and after some time the OS won't be able to handle a situation with HUGE connections and will in turn cause contentions at different compute level(i.e. Memory, CPU, I/O).
What if you can use the presently spawned child processes(backends) if they are not doing any work. You will save time on getting the child process(backends) and the additional cost as well(this can be different at times). This is where the pool of connections that are always open help to serve different client requests comes in and is also called pooling.
So basically now you have only n connections available but the pooler can manage n+i number of connections to serve the client requests.
This where pg-bouncer helps to reuse the connections. It can be configured with 3 types of pooling i.e Session pooling, Statement pooling and Transaction pooling. Basically bouncer returns the connection back to the pool once it has done, statement level work or transaction level work etc. Only during session pooling it keeps the connections unless it disconnects.
So basically lower down the number of connections at PG conf file level and tune all settings in the bouncer.ini.
To answer the second part:
one more question. in pgpouncer configuration, does it right listen_addr = 0.0.0.0 ? or should to be listen_addr = * ?
It depends if you have a standalone deployment, server etc.
basically if its on the server itself and you want it to allow connections from everywhere(incoming) use "*" if you want only the local network to be allowed use "127.0.0.0".
For the rest of your questions check this link: pgbouncer docs
I have tried to share a little of what I know, feel free to ask away if anything was unclear or or correct if it was incorrectly mentioned.

High number of connections on Mongoose 5.6.1

I use mongoose to connect to a replica set consisting of 1 primary and 3 secondary nodes. I recently updated my application from Mongoose 4.9.1 to 5.6.2. I've kept the connection pool size same as before (30) and used the new param minSize to set minimum as 2.
I see my application using more connections than before with the same traffic. It's keeping ~20-22 connections open to primary as compared to 8-10 earlier. This leads me to reach the connection limit on my mongodb cluster.
Is there any change in mongoose that leads to this high number of connections being kept ready?
Here are the config options I am using.
{
minSize: 2,
poolSize: 15,
autoReconnect: true,
reconnectTries: Number.MAX_VALUE,
reconnectInterval: 5000,
useNewUrlParser: true,
useCreateIndex: true,
socketTimeoutMS: 30000
}
Try setting the maxPoolSize option in the connection string (https://docs.mongodb.com/manual/reference/connection-string/#urioption.maxPoolSize), although the behavior of this option with the connection pool may be dependent on the driver.
That said, a database side connection limit of 20-22 seems really low.

Grails 3 Setup application.yml production mongodb

How to set the use connectionstring which usess mongodb in production.
development:
grails:
mongodb:
connectionString: "mongodb://localhost:27017/fanfest"
production:
dataSource:
dbCreate: update
url: jdbc:h2:./prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
properties:
jmxEnabled: true
initialSize: 5
maxActive: 50
minIdle: 5
maxIdle: 25
maxWait: 10000
maxAge: 600000
timeBetweenEvictionRunsMillis: 5000
minEvictableIdleTimeMillis: 60000
validationQuery: SELECT 1
validationQueryTimeout: 3
validationInterval: 15000
testOnBorrow: true
testWhileIdle: true
testOnReturn: false
jdbcInterceptors: ConnectionState
defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED
Working with Grails 3 version. Able to connect with mongodb in development environment. Kindly provide some suggestions to set the mongodb in production environment.
You currently have development pointing at a mongo instance. The production configuration is pointed at the in memory H2 database. If you would like to configure a mongo database for your production environment may I suggest you take a look at Getting Started guide for Mongo and GORM.
In the production section of your configuration file you can use the connection string parameter as follows:
production {
grails {
mongodb {
connectionString = "mongodb://localhost:27017/PROD_fanfest"
}
}
}
Please note I have used your development URL but changed the table name since I recommend you keep the development database and production database separate. Configuring production datasource to use Mongo is that easy. There are more configuration options described in the Getting Started documentation.
Relevant information on configuring options below:
options {
connectionsPerHost = 10 // The maximum number of connections allowed per host
threadsAllowedToBlockForConnectionMultiplier = 5
maxWaitTime = 120000 // Max wait time of a blocking thread for a connection.
connectTimeout = 0 // The connect timeout in milliseconds. 0 == infinite
socketTimeout = 0 // The socket timeout. 0 == infinite
socketKeepAlive = false // Whether or not to have socket keep alive turned on
writeConcern = new com.mongodb.WriteConcern(0, 0, false) // Specifies the number of servers to wait for on the write operation, and exception raising behavior
sslEnabled = false // Specifies if the driver should use an SSL connection to Mongo
socketFactory = … // Specifies the SocketFactory to use for creating connections
}

Sail.js multiple connections on start

I've got an odd problem - on start of my sails app (which is connecting with postgres and deployed on heroku ) there are multiple connections (around 10) to database, and since it's free account, if I then try to launch app on localhost to test some new code I get an error "too many connections for a role". So does anyone know why there are so many connections to database and can I change it, to have only one connection per app?
EDIT:
Error creating a connection to Postgresql: error: too many connections for role
"xwoellnkvjcupt"
Error creating a connection to Postgresql: error: too many connections for role
"xwoellnkvjcupt"
error: Hook failed to load: orm (error: too many connections for role "xwoellnkv
jcupt")
error: Error encountered while loading Sails core!
error: error: too many connections for role "xwoellnkvjcupt"
at Connection.parseE (C:\Studia\szachman2\node_modules\sails-postgresql\node
_modules\pg\lib\connection.js:561:11)
at Connection.parseMessage (C:\Studia\szachman2\node_modules\sails-postgresq
l\node_modules\pg\lib\connection.js:390:17)
at null. (C:\Studia\szachman2\node_modules\sails-postgresql\node_
modules\pg\lib\connection.js:98:18)
at CleartextStream.EventEmitter.emit (events.js:95:17)
at CleartextStream. (_stream_readable.js:746:14)
at CleartextStream.EventEmitter.emit (events.js:92:17)
at emitReadable_ (_stream_readable.js:408:10)
at _stream_readable.js:401:7
at process._tickDomainCallback (node.js:459:13)
this is an error I am getting often when trying to test some new code on localhost.
#jantar #sgress454 I just added a troubleshooting message in sails-postgresql to try and make this better. Here's what it says:
-> Maybe your poolSize configuration is set too high? e.g. If your Postgresql database only supports 20 concurrent connections, you should make sure you have your poolSize set as something < 20. The default poolSize is 10.
To override the default poolSize, specify a poolSize property on the relevant Postgresql "connection" config object. If you're using Sails, this is generally located in config/connections.js, or wherever your environment-specific database configuration is set.
-> Do you have multiple Sails instances sharing the same Postgresql database? Each Sails instance may use up to the configured poolSize # of connections. Assuming all of the Sails instances are just copies of one another (a reasonable best practice) we can calculate the actual # of Postgresql connections used (C) by multiplying the configured poolSize (P) by the number of Sails instances (N). If the actual number of connections (C) exceeds the total # of AVAILABLE connections to your Postgresql database (V), then you have problems. If this applies to you, try reducing your poolSize configuration. A reasonable poolSize setting would be V/N.
This is due to Sails's auto-migration feature which attempts to keep your models and database synced up. It's not intended to be used in production. You can turn auto-migration off on a single model by adding migrate: safe to the model definition:
module.exports = {
migrate: 'safe',
attributes: {...}
}
You can turn auto-migration off for all models by adding a model config, usually in your config/locals.js:
module.exports = {
model: {
migrate: 'safe'
},
environment: 'production',
...other local config...
}
A little update for the V1. Your adapter in config/datastore.js should look like this if you want to set a maximum size for the connection pool :
{
adapter: 'sails-postgresql',
url: 'yourconnectionurl',
max: 1 // This is the important part for poolSize, I set 1 because I don't want more than 1 connection ^^
}
If you want to know all infos you can set, look here : https://github.com/sailshq/machinepack-postgresql/blob/176413efeab90dc5099dc60718e8b520942ce3be/machines/create-manager.js , at line 162 :
// Basic:
'host', 'port', 'database', 'user', 'password', 'ssl',
// Advanced Client Config:
'application_name', 'fallback_application_name',
// General Pool Config:
'max', 'min', 'refreshIdle', 'idleTimeoutMillis',
// Advanced Pool Config:
// These should only be used if you know what you are doing.
// https://github.com/coopernurse/node-pool#documentation
'name', 'create', 'destroy', 'reapIntervalMillis', 'returnToHead',
'priorityRange', 'validate', 'validateAsync', 'log'