Why does mongodb logs show too many opened connections? It's showing me more than the maximum connection limit and number of current operations in db.
Also my primary refused to create more connections after reaching 819 limit. That time, the number of current operations in db were less than 819. Raising ulimit has solved my problem temporarily, but why were idle connections not utilized to serve the requests?
I was having this exact same problem. My connection number just kept growing and growing until it hit 819 and then no more connections were allowed.
I'm using mongo-java-driver version 2.11.3. What has seemed to fix the problem is explicitly setting the connectionsPerHost and threadsAllowedToBlockForConnectionMultiplier properties of the MongoClient. Before I was not setting these values myself and accepting the defaults.
MongoClientOptions mco = new MongoClientOptions.Builder()
.connectionsPerHost(100)
.threadsAllowedToBlockForConnectionMultiplier(10)
.build();
MongoClient client = new MongoClient(addresses, mco); //addresses is a pre-populated List of ServerAddress objects
In my application the MongoClient is defined as a static singleton.
I was watching the mongodb logs, and once the application hit 100 open connections I didn't see any more connections established from the client application. I am running a replica set, so you still see the internal connections being made which properly close.
From the MongoDB documentation:
"If you see a very large number connection and re-connection messages in your MongoDB log,then clients are frequently connecting and disconnecting to the MongoDB server. This is normal behavior for applications that do not use request pooling, such as CGI. Consider using FastCGI, an Apache Module, or some other kind of persistent application server to decrease the connection overhead.
If these connections do not impact your performance you can use the run-time quiet option or the command-line option --quiet to suppress these messages from the log."
http://docs.mongodb.org/manual/faq/developers/#why-does-mongodb-log-so-many-connection-accepted-events
Make sure you are using latest driver of mongodb.
Related
I’m using Postgres provisioned by Google Cloud SQL,
Recently we see the number of connections to increase by a lot.
Had to raise the limit from 200 to 500, then to 1000. In Google Cloud console Postgres reports 800 currenct connections.
However I have no idea where these connections come from. We have one app engine service, with not a lot of traffic at the moment accessing it, another application hosted on kubernetes. And a dozen or so batch jobs that connect to it. Clearly there must be some connection leakage somewhere.
Is there any way I can see from where these connections originate ?
All applications connecting to it are Java based at the moment.
They use the HikariCP connection pool. I’m considering changing the “test query”upon connection to insert a record in a log table. Hence I could perhaps find out from where the connections originate.
But are there better ways available?
Thanks,
Consider monitoring connection activity with pg_stat_activity, i.e: SELECT * from pg_stat_activity;.
As per the documentation:
Connections that show an IP address, such as 1.2.3.4, are connecting using IP. Connections with cloudsqlproxy~1.2.3.4 are using the Cloud SQL Proxy, or else they originated from App Engine. Connections from localhost are usually to a First Generation instance from App Engine, although that path is also used by some internal Cloud SQL processes.
Also, take a look at the best practices for managing database connections that contain information on opening and closing connections, connection count, or on how to set a connection duration in the Java programming language.
in Java, application servers like JBoss EAP have the option to periodically verify the connections in a database pool (https://access.redhat.com/documentation/en-us/red_hat_jboss_enterprise_application_platform/6.4/html/administration_and_configuration_guide/sect-database_connection_validation). This has been very useful for removing stale connections.
I'm now looking at a ADO.NET application, and I was wondering if there was any similar functionality that could be used with a Microsoft SQL Server?
I ended up find this post by redgate that describes some of the validation that goes on when connections are taken from the pool:
If the connection has died because a router has decided that it no
longer wants to forward your packets and no other routers like you
either then there is no way to know this unless you try to send some
data and don’t get a response.
If you create a connection and a connection pool is created and
connections are put into the pool and not used, the longer they are in
there, the bigger the chance of something bad happening to it.
When you go to use a connection there is nothing to warn you that a
router has stopped forwarding your packets until you go to use it; so
until you use it, you do not know that there is a problem.
This was an issue with connection pooling that was fixed in the first
.Net 4 reliability update (see issue 14 which vaguely describes this)
with a feature called “Connection Pool Resiliency”. The update meant
that when a connection is about to be taken from the pool, it is
checked for TCP validity and only returned if it is in a good state.
In Postgres, is there a one-to-one relationship between a client and a connection? In other word, is a client always one connection and no client can open more than one connection?
For example, when Postgres says:
org.postgresql.util.PSQLException: FATAL: sorry, too many clients already.
is that equivalent to "too many connections already"?
Also, as far as I understand, Postgres uses one process for each client. So does this mean that each process is used for one connection only?
Refer - https://www.postgresql.org/docs/9.6/static/connect-estab.html
PostgreSQL is implemented using a simple "process per user"
client/server model. In this model there is one client process
connected to exactly one server process. As we do not know ahead of
time how many connections will be made, we have to use a master
process that spawns a new server process every time a connection is
requested.
So yes, one server process serves one connection.
You can have as many connections from a single client (machine, application) as the server can manage. The server can support a given number of connections, whether or not these come from different clients (machine, application) is irrelevant to the server.
The connection is made to the postmaster process that is listening on the port that PG is configured to listen to (5432 by default). When a connection is established (after authentication), the server spawns a process which is used exclusively by a single client. That client can make multiple connections to the same server, for instance to connect to different databases, or the same database using different credentials, etc.
As far as I know when one establishes multiple Connection objects via JDBC to one database then each connection occupies a separate port on the machine where the Connection is established (they all connect to one port on the server where the DBMS is running).
I tried to extract the port that corresponds to the Connection objects. Unfortunately I did not find any way to do so.
Background: I'm doing performance analysis where I setup multiple clients which issue queries on the db. I'm logging the execution time of queries on the database server. In the resulting log I have - among others - information about the connection who initiated the query e.g. localhost.localdomain:44760 I hope it is possible to use this information to map each query to the client or more precisely the Connection object who initiated the query (which is my ultimate goal and serves analysis purposes).
Just run this select through the JDBC connection:
select inet_client_port()
More functions like that are in the manual:
http://www.postgresql.org/docs/current/static/functions-info.html
I've set up PGBouncer and configured it to connect to my postgres DB and it all connects just fine, however I'm not sure if its actually working.
I have a php script that runs as a daemon and picks up beanstalk jobs. The problem is for every distinct user/action on the system it opens a new connection to postgres and then leaves that connection idling because the daemon doesn't actually stop running so the connection is never terminated (a quick fix for this was to reset the connection at the end of the script loop, but that is going to be inefficient with many connects).
Anyway, this caused postgres to eventually run out of connections and lock up...
So PGBouncer seems like the answer.
But now when I run it, I see the same db connection multiple times when i do ps ax | grep postgres.
Isn't PGBouncer supposed to only ever have 1 connection open to the DB and route all traffic through that connection? Then open a new connection if that one is full?
At present I have 3 for one db connection (my access control system) and 2 for the other database (my client specific data).
To me, it feels like if I roll out these changes then I will just be faced with the same problem that the connections will just get eaten up again because they're not being released.
I hope that explains enough for someone to offer any advice.
It sounds like an important step here is to fix the scripts so they release the connections when they're done. Once you do that, PgBouncer will help reduce the connection setup/teardown overhead, but in connection pooling mode it won't give you the ability to maintain more connections to Pg than you otherwise could.
However, you can also use PgBouncer in transaction-pooling mode. When used for transaction pooling, PgBouncer keeps a pool of idle backend transactions and only assigns them when a client does a BEGIN. Connections are returned to the pool after the client does a COMMIT or ROLLBACK. That means that in transaction pooling mode you can have large numbers of open connections to PgBouncer; they don't each need a corresponding connection to a PostgreSQL backend, so long as most of them are idle at any point in time and don't have any transaction open.
The only real downside of transaction pooling mode is that it breaks applications that expect to SET session-level variables, keep prepared statements across transactions, etc. Applications may need modification, e.g. to use SET LOCAL.