I'm using pg-promise (and the underlying pg-pool lib).
When I pull the plug on my pg database briefly (break TCP connections), I get connect ETIMEDOUT errors (expected) however it is taking a very long time for the pool to re-establish a connection. Like in the order of 10 minutes.
Here is a snippet for how I am using pg-promise:
const pgp = pgPromise({
error(err, e) {
if (e.query) {
// Do stuff
}
},
});
...
const pgpArgs = {
host,
port,
database,
user,
password,
max,
idleTimeoutMillis: 5000,
};
const db = pgp(pgpArgs);
I saw this potential answer from the author, but I don't want to kill my entire process just to re-establish the pool.
Is there a way to force clients in pool to re-connect, other than restarting the nodejs process?
Thanks in advance.
Related
I have a Go program which uses Mongo DB. I run it from my home pc and I keep getting this error very often:
panic: server selection error: server selection timeout, current
topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr:
cluster0-shard-00-00.tvm1o.mongodb.net:27017, Type: Unknown, Last
error: connection() error occurred during connection handshake: dial
tcp
3.216.112.85:27017: i/o timeout }, { Addr: cluster0-shard-00-01.tvm1o.mongodb.net:27017, Type: Unknown, Last
error: connection() error occurred during connection handshake: dial
tcp 34.197.85.254:27017: i/o timeout }, { Addr:
cluster0-shard-00-02.tvm1o.mongodb.net:27017, Type: Unknown, Last
error: connection() error occurred during connection handshake: dial
tcp 18.206.5.2:27017: i/o timeout }, ] }
And this is the exact code where it breaks:
if err := clientMongo.Ping(context.TODO(), readpref.Primary()); err != nil {
panic(err)
}
I understand this is a connection timeout, but I don't understand how can this happen at all during a simple client connection. I made a speedtest and my current upload speed is 22 Mbps, I am not uploading big json arrays or anything. It happens always when I try to connect to the client. So I would like to know if this can be caused because my internet connection or something on Mongo's end?
You might need to add your IP to the whitelist of MongoDB.
A few things --
we would need to see the complete code for creating a connection. I'm going to assume you're using exactly what is in the documentation here?
You should try to connect with mongosh and Compass also. If you have problems with another tool, then the odds are it is your Atlas cluster OR your security settings on the cluster, rather than your application code.
that being said about 95% of the time the issue is whitelist or database users. do you have a user created in database access area of the UI that has admin/read/write any database? is your IP in the whitelist?
if 3 is good, and 2 doesn't work, there is free Atlas support available in the green chat button of the UI in MongoDB.
I am confused about pgbouncer pool size configuration and ORM(like sequelize.js), query builder(like knex.js) library pool size configuration. The architecture like this:
Application code => pgbouncer => postgresql
pgbouncer.ini:
;; ...
;; Default pool size. 20 is good number when transaction pooling
;; is in use, in session pooling it needs to be the number of
;; max clients you want to handle at any moment
;default_pool_size = 20
;; ...
sequelize connection pool configuration:
const sequelize = new Sequelize(/* ... */, {
// ...
pool: {
max: 5,
min: 0,
acquire: 30000,
idle: 10000
}
});
knex.js connection pool configuration:
var knex = require('knex')({
client: 'mysql',
connection: {
host : '127.0.0.1',
user : 'your_database_user',
password : 'your_database_password',
database : 'myapp_test'
},
pool: { min: 0, max: 7 }
});
What happened if I use sequelize.js connection pool configuration and pgbouncer connection pool size configuration together? Which configuration does the database server use? Should I use only one of them? Thanks.
If you have for example 3 application processes running knex or sequelize, then you should setup pgbouncer poolsize to be 3 times bigger than single knex / sequelize pool uses.
Then you also need to make sure that postgres server also has enough connections configured to handle pgbouncer connections.
Though as #jjanes said. There is no reason to use pgbouncer with knex / sequelize, because they already provide pooling. I suppose pgbouncer is meant to be used with frameworks, which doesn't support pooling. For example if PHP or cgi stript reinitializes on every page load and makes calls to database.
It rarely makes sense to daisy chain connection pools together. So there is probably no point in using pgbouncer in addition to the built-in ones. The database server doesn't know about your connection poolers except to the extent the pool manager sends its own explicit commands to the database, and it has its own configuration file which it uses (postgresql.conf).
Is there anything in the modern PostgreSQL connection protocol that would indicate the server version?
And if not, is there a special low-level request that an endpoint can execute against an open connection to pull the server details that would contain the version?
I'm looking at a possible extension of node-postgres that would automatically provide the server version upon every fresh connection. And I want to know if this is at all possible.
Having to execute SELECT version() upon every new connection and then parsing it is too high-level for the base driver that manages the connection. It should be done on the protocol level.
After a bit of research, I found that PostgreSQL does provide server version during connection, within the start-up message.
And specifically within node-postgres driver, we can make Pool provide a custom Client that handles event parameterStatus on the connection, and exposes the server version:
const {Client, Pool} = require('pg');
class MyClient extends Client {
constructor(config) {
super(config);
this.connection.on('parameterStatus', msg => {
if (msg.parameterName === 'server_version') {
this.version = msg.parameterValue;
}
});
}
}
const cn = {
database: 'my-db',
user: 'postgres',
password: 'bla-bla',
Client: MyClient // here's our custom Client type
};
const pool = new Pool(cn);
pool.connect()
.then(client => {
console.log('Server Version:', client.version);
client.release(true);
})
.catch(console.error);
On my test PC, I use PostgreSQL v11.2, so this test outputs:
Server Version: 11.2
UPDATE - 1
Library pg-promise has been updated to support the same functionality in TypeScript. And you can find a complete example in this ticket.
UPDATE - 2
See example here:
// tests connection and returns Postgres server version,
// if successful; or else rejects with connection error:
async function testConnection() {
const c = await db.connect(); // try to connect
c.done(); // success, release connection
return c.client.serverVersion; // return server version
}
I'm trying to create an alarm system for my application, that will trigger when one of the services (e.g. MongoDB) is not working.
What I'm doing is, once the application is started, I shut down my MongoDB server and try to connect to it, but instead of receiving an error my application just gets stuck into the execution of the method. The server console looks like something is in execution.
My current code (coffeescript) is:
checkMongoService: ()->
mongo = Npm.require 'mongodb'
assert = Npm.require 'assert'
url = 'mongodb://....'
mongo.connect url, (err, db) ->
assert.equal null, err
console.log 'Connected correctly to server'
db.close()
return
I've also been trying by doing a simple
Meteor.users.find().count();
or using MongoInternals with
testConnection = new MongoInternals.RemoteCollectionDriver("mongodb://...);
but still same issue, when mongo is not running no error is thrown and the console stops to work. If then I start Mongo again, it will just return the result (in this case the log 'Connected correctly to server')
Something that I've noticed is if I try with meteor shell to execute testConnection = new MongoInternals.RemoteCollectionDriver("mongodb://...); I get an error "Error: failed to connect to [127.0.0.1:27017]"
TL;DR
Do you might have an idea on how I can check if mongo is reachable or do you know if I'm doing something wrong with the code above?
Try setting the timeouts to be a bit shorter than the default 30 seconds:
mongo.connect(url, {
connectTimeoutMS: 1000,
socketTimeoutMS: 1000,
reconnectTries: 1
}, function(err, db) {...}
(Full set of connection params are here)
Meteor.status().status
from the docs
This method returns the status of the connection between the client and the server. The return value is an object with the following fields:
connected (Boolean)
True if currently connected to the server. If false, changes and method invocations will be queued up until the connection is reestablished.
status (String)
Blockquote
Describes the current reconnection status. The possible values are connected (the connection is up and running), connecting (disconnected and trying to open a new connection), failed (permanently failed to connect; e.g., the client and server support different versions of DDP), waiting (failed to connect and waiting to try to reconnect) and offline (user has disconnected the connection).
https://docs.meteor.com/api/connections.html
I've got an odd problem - on start of my sails app (which is connecting with postgres and deployed on heroku ) there are multiple connections (around 10) to database, and since it's free account, if I then try to launch app on localhost to test some new code I get an error "too many connections for a role". So does anyone know why there are so many connections to database and can I change it, to have only one connection per app?
EDIT:
Error creating a connection to Postgresql: error: too many connections for role
"xwoellnkvjcupt"
Error creating a connection to Postgresql: error: too many connections for role
"xwoellnkvjcupt"
error: Hook failed to load: orm (error: too many connections for role "xwoellnkv
jcupt")
error: Error encountered while loading Sails core!
error: error: too many connections for role "xwoellnkvjcupt"
at Connection.parseE (C:\Studia\szachman2\node_modules\sails-postgresql\node
_modules\pg\lib\connection.js:561:11)
at Connection.parseMessage (C:\Studia\szachman2\node_modules\sails-postgresq
l\node_modules\pg\lib\connection.js:390:17)
at null. (C:\Studia\szachman2\node_modules\sails-postgresql\node_
modules\pg\lib\connection.js:98:18)
at CleartextStream.EventEmitter.emit (events.js:95:17)
at CleartextStream. (_stream_readable.js:746:14)
at CleartextStream.EventEmitter.emit (events.js:92:17)
at emitReadable_ (_stream_readable.js:408:10)
at _stream_readable.js:401:7
at process._tickDomainCallback (node.js:459:13)
this is an error I am getting often when trying to test some new code on localhost.
#jantar #sgress454 I just added a troubleshooting message in sails-postgresql to try and make this better. Here's what it says:
-> Maybe your poolSize configuration is set too high? e.g. If your Postgresql database only supports 20 concurrent connections, you should make sure you have your poolSize set as something < 20. The default poolSize is 10.
To override the default poolSize, specify a poolSize property on the relevant Postgresql "connection" config object. If you're using Sails, this is generally located in config/connections.js, or wherever your environment-specific database configuration is set.
-> Do you have multiple Sails instances sharing the same Postgresql database? Each Sails instance may use up to the configured poolSize # of connections. Assuming all of the Sails instances are just copies of one another (a reasonable best practice) we can calculate the actual # of Postgresql connections used (C) by multiplying the configured poolSize (P) by the number of Sails instances (N). If the actual number of connections (C) exceeds the total # of AVAILABLE connections to your Postgresql database (V), then you have problems. If this applies to you, try reducing your poolSize configuration. A reasonable poolSize setting would be V/N.
This is due to Sails's auto-migration feature which attempts to keep your models and database synced up. It's not intended to be used in production. You can turn auto-migration off on a single model by adding migrate: safe to the model definition:
module.exports = {
migrate: 'safe',
attributes: {...}
}
You can turn auto-migration off for all models by adding a model config, usually in your config/locals.js:
module.exports = {
model: {
migrate: 'safe'
},
environment: 'production',
...other local config...
}
A little update for the V1. Your adapter in config/datastore.js should look like this if you want to set a maximum size for the connection pool :
{
adapter: 'sails-postgresql',
url: 'yourconnectionurl',
max: 1 // This is the important part for poolSize, I set 1 because I don't want more than 1 connection ^^
}
If you want to know all infos you can set, look here : https://github.com/sailshq/machinepack-postgresql/blob/176413efeab90dc5099dc60718e8b520942ce3be/machines/create-manager.js , at line 162 :
// Basic:
'host', 'port', 'database', 'user', 'password', 'ssl',
// Advanced Client Config:
'application_name', 'fallback_application_name',
// General Pool Config:
'max', 'min', 'refreshIdle', 'idleTimeoutMillis',
// Advanced Pool Config:
// These should only be used if you know what you are doing.
// https://github.com/coopernurse/node-pool#documentation
'name', 'create', 'destroy', 'reapIntervalMillis', 'returnToHead',
'priorityRange', 'validate', 'validateAsync', 'log'