How to get the number of connections used (and free) to the MongoDB (from a client perspective)? - mongodb

I'm posting the question here just to be sure I'm not barking on the wrong tree.
How to get the number of connections used (and free) to the MongoDB, but from a client perspective (eg. Java client), using the 4.x driver?
There are posts regarding using the serverStatus(Get the number of open connections in mongoDB using java), but it presumes having 'admin' access to the MongoDB. Using a 'regular user'(an db user with lower privileges (e.g access to only one database)) cannot run the serverStatus(). But this provides only a view from the server-side (there are N connections from IP x).
Other posts mentioned how to setup the connection pool size (eg. using the MongoClients.create​(MongoClientSettings settings) (see the 4.x API reference (https://mongodb.github.io/mongo-java-driver/4.0/apidocs/mongodb-driver-sync/com/mongodb/client/MongoClients.html)):
MongoCredential credential = MongoCredential.createCredential(
username,
"admin",
password.toCharArray());
MongoClient mongoClient = MongoClients.create(MongoClientSettings.builder()
.applyToClusterSettings(
builder -> builder.hosts(Arrays.asList(new ServerAddress(hostname, portNumber))))
.credential(credential)
.applyToConnectionPoolSettings(builder -> builder
.minSize(connectionPoolMinimumSize)
.maxSize(connectionPoolMaximumSize))
.readConcern(readConcern)
.readPreference(readPreference)
.writeConcern(writeConcern)
.build());
But none provided means to get the used and available connections the connection pool.
As mentioned by Oleg, using the ConnectionPoolListener would be a way, but that is available only in the 3.x drivers. The ConnectionPoolListener methods are marked as deprecated on 4.x (although it is still mentioned in the JMX Monitoring section (http://mongodb.github.io/mongo-java-driver/4.0/driver-reactive/reference/monitoring/).

You can use connection pool monitoring which is described here to keep track of connection states, and deduce the counts you are looking for.
I don't know if Java driver exposes the counters you are looking for as public APIs; many drivers don't.

Finally got this working:
created a custom connection pool listener, implementing the com.mongodb.event.ConnectionPoolListener...
public class CustomConnectionPoolListener implements ConnectionPoolListener {
...
}
... and having the stats counters updated on a store (accessible later)
#Override
public void connectionCreated(ConnectionCreatedEvent event) {
ConnectionPoolStatsPOJO cps = mongoConnectionPoolList.get(connectionPoolAlias);
cps.incrementConnectionsCreated();
mongoConnectionPoolList.put(connectionPoolAlias, cps);
}
attached this custom connection pool listener to the MongoClient connection:
ConnectionPoolListener customConnPoolListener = new CustomConnectionPoolListener(...); /* added some references in the */
...
MongoClientSettings mongoClientSettings = MongoClientSettings.builder()
.applicationName(applicationName)
.applyConnectionString(connURI)
.credential(credential)
.readConcern(readConcern)
.readPreference(readPreference)
.writeConcern(writeConcern)
.applyToConnectionPoolSettings(builder -> builder
.minSize(connectionPoolMinimumSize)
.maxSize(connectionPoolMaximumSize)
.addConnectionPoolListener(customConnPoolListener)
)
.retryWrites(true)
.retryReads(true)
.build();
...
MongoClient mongoClient = MongoClients.create(mongoClientSettings);
....
finally, to access the connection pool stats, just have to query out the store:
ConnectionPoolStatsPOJO connectionPoolStats = MongoDB_ConnectionPool_Repository.getInstance().getMongoConnectionPoolList().get(connectionPoolAlias);
Therefore, thanks to "#D. SM" for pointing to the right direction.

Related

How do I set MaxConnPerRoute, ConnectionRequestTimeout, keepAliveStrategy in Spring WebFlux WebClient

we have the following custom connection pooling implemented for RestTemplate.
PoolingHttpClientConnectionManager poolingConnManager =
new PoolingHttpClientConnectionManager();
poolingConnManager.setDefaultMaxPerRoute(restClientprops.getRestClientMaxPerRoutePool());
poolingConnManager.setMaxTotal(restClientprops.getRestClientMaxTotalPool());
HttpClientBuilder httpClientBuilder = HttpClients.custom()
.setConnectionManager(poolingConnManager)
.setKeepAliveStrategy(DefaultConnectionKeepAliveStrategy.INSTANCE)
.setMaxConnPerRoute(restClientprops.getRestClientMaxPerRoutePool())
.setMaxConnTotal(restClientprops.getRestClientMaxTotalPool());
HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
requestFactory.setConnectTimeout(restClientprops.getConnectTimeout());
requestFactory.setReadTimeout(restClientprops.getReadTimeout());
requestFactory.setConnectionRequestTimeout(restClientprops.getConnectionRequestTimeout());
requestFactory.setHttpClient(httpClientBuilder.build());
this.restTemplate = new RestTemplate(requestFactory);
I am changing it to WebClient implementation, and this is what I could come up with.
HttpClient httpClient = HttpClient
.create(ConnectionProvider.create("webclient-pool", restClientprops.getRestClientMaxTotalPool()))
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, restClientprops.getConnectTimeout())
.responseTimeout(Duration.ofMillis(restClientprops.getConnectionRequestTimeout()))
.doOnConnected(conn -> conn.addHandler(new ReadTimeoutHandler(restClientprops.getReadTimeout(), TimeUnit.MILLISECONDS)))
.keepAlive(true);
Per this URL https://github.com/reactor/reactor-netty/issues/1159
from what I understood connection request timeout is renamed to responseTimeOut in webclient httpclient. Is that accurate?
How should I set MaxConnPerRoute in webclient that is in the RestTemplate implementation?
Is keepAlive(true) accurate translation of setKeepAliveStrategy(DefaultConnectionKeepAliveStrategy.INSTANCE)
Appreciate your help.
Per this URL https://github.com/reactor/reactor-netty/issues/1159 from what I understood connection request timeout is renamed to responseTimeOut in webclient httpclient. Is that accurate?
Yes that's true. More about all timeouts that can be configured for the HttpClient you can find in the Reference Documentation.
How should I set MaxConnPerRoute in webclient that is in the RestTemplate implementation?
You can provide connection pool configuration per remote address (if that's what you mean with MaxConnPerRoute), see javadoc for forRemoteHost.
ConnectionProvider.builder("test")
.maxConnections(2) // default max connections
.forRemoteHost(<socket-address>,spec -> spec.maxConnections(1)) // max connections only for this socket address
.build();
Is keepAlive(true) accurate translation of setKeepAliveStrategy(DefaultConnectionKeepAliveStrategy.INSTANCE)
If you mean specifying whether the connection is persistent or not then YES this configuration has to be used. By default the connection IS persistent. If you mean SO_KEEPALIVE then NO and you have to use .option() configuration. You can find more in the Reference Documentation.
HttpClient.create()
.option(ChannelOption.SO_KEEPALIVE, true)
This configuration can be removed is you use the timeout settings provided by Reactor Netty:
.doOnConnected(conn -> conn.addHandler(new ReadTimeoutHandler(restClientprops.getReadTimeout(), TimeUnit.MILLISECONDS)))

Issue with HttpClient in AEM

I have an HttpClient code written that is from org.apache.commons.httpclient package.
In that I am setting connection time and socket time out this way.
final HttpClient http = new HttpClient(this.connectionManager);
http.getParams().setParameter("http.connection.timeout", this.connectionTimeout);
http.getParams().setParameter("http.socket.timeout", this.socketTimeout);
Now the Adobe Cloud has raised issue that timeout is not being set(which is not true).
They suggested to set timeouts using
#Reference
private HttpClientBuilderFactory httpClientBuilderFactory;
public void doThis() {
HttpClientBuilder builder = httpClientBuilderFactory.newBuilder();
RequestConfig requestConfig = RequestConfig.custom()
.setConnectTimeout(5000)
.setSocketTimeout(5000)
.build();
builder.setDefaultRequestConfig(requestConfig);
HttpClient httpClient = builder.build();
// do something with the client
}
Refer Link
But HttpClientBuilderFactory does not belong to **org.apache.commons.httpclient it belongs to org.apache.http.client**
And always returns Closable Http client.
How do I resolve this security issue? Can I add an annotation for exception? Or will I have to rewrite all my code?
This issue is with Adobe Experience Manager 6.5 instance.
Is it probably because you are not setting the right timeout parameter?
You are setting the property http.connection.timeout which is not available in the class org.apache.commons.httpclient.params.HttpClientParams.
http.getParams() returns an instance of HttpClientParams which has the socket timeout and connection manager timeout but not a connection timeout. You could probably use the constant HttpClientParams.CONNECTION_MANAGER_TIMEOUT to set a timeout for the connection manager?
On the other hand, the property http.connection.timeout is available for the class HttpConnectionParams.
Constant field values reference
The problem is Adobe has two versions of HttpClient the old 3.x that has package structure org.apache.commons.httpclient.HttpClient and the one that HttpClientBuilderFactory gives out that is 4.x org.apache.http.Httpclient.
I was breaking my head around this. Finally we were left with two options...
1) Rewrite all our commons http api(3.x) to the newer version of apache.http (4.x) that has the methods setTimeout and setConnectionTimeout
OR
2)#SuppressWarnings("CQRules:ConnectionTimeoutMechanism")
We chose Option number 2 as the effort arround this was huge and we are planning to go live soon.

Implementing a connection recreation mechanism on periodic DB password change

We are using a PostgreSQL database with AWS RDS IAM authorization feature – which means that our application needs to refresh the authorization token every 10 minutes or so (since the token is valid for 15 minutes). This token is used as a database password and I need to periodically update it. We are using the Dropwizard framework which is taking advantage of Apache Commons DBCP Component that handles connection pooling.
I was able to enhance the configuration class so that it performs an AWS API call to get the token instead of reading the password from configuration file. However this works only once, during application startup, for 15 minutes. I would like to call AWS API for the token perdiodically and handle the creation of connections as well as invalidating old ones.
import org.jooq.Configuration;
import org.jooq.impl.DefaultConfiguration;
import io.dropwizard.setup.Environment;
import org.example.myapp.ApplicationConfiguration;
// more less relevant imports...
#Override
public void run(ApplicationConfiguration configuration, Environment environment) {
Configuration postgresConfiguration = new DefaultConfiguration().set(configuration.getDbcp2Configuration()
.getDataSource())
.set(SQLDialect.POSTGRES_10)
.set(new Settings().withExecuteWithOptimisticLocking(true));
// this DSLContext object needs to be refreshed/recreated every 10 minutes with the new password!
KeysDAO.initialize(DSL.using(postgresConfiguration));
// rest of the app's config
}
How can I implement such a connection recreation mechanism? The org.jooq.ConnectionProvider looks promising, but I need some more guidance on how to inject the password on a periodic basis (and implement a custom ConnectionProvider). Any hints would be greatly appreciated.
EDIT:
This morning I was able to confirm that after a fresh deployment the database interaction is possible, and after exactly 15 minutes I'm getting first exceptions:
org.postgresql.util.PSQLException: FATAL: PAM authentication failed for user "jikg_service"
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:514)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:141)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:454)
at org.postgresql.Driver.connect(Driver.java:256)
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39)
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:256)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
at org.apache.commons.pool2.impl.GenericObjectPool.ensureIdle(GenericObjectPool.java:927)
at org.apache.commons.pool2.impl.GenericObjectPool.ensureMinIdle(GenericObjectPool.java:906)
at org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor.run(BaseGenericObjectPool.java:1046)
at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
at java.base/java.util.TimerThread.run(Timer.java:506)
Suppressed: org.postgresql.util.PSQLException: FATAL: pg_hba.conf rejects connection for host "172.30.19.218", user "my_db_user", database "my_db_development", SSL off
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:514)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:141)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:201)
... 12 common frames omitted
Those exceptions are repeated every minute.
I owe you all an explanation on this one. I forgot to mention one significant detail - we're actually using a modified version of Dropwizard developed in-house that uses bundled Apache Commons DBCP (which afaik is not officially part of Dropwizard) as well as other components. I ended up dropping Apache Commons DBCP in favor of HikariCP - which made it possible to update the pool configuration at runtime. Although not officially supported, the creator of the library hinted that it might work, and in our scenario it indeed worked. Below is a sample solution.
import org.jooq.Configuration;
import org.jooq.impl.DefaultConfiguration;
import io.dropwizard.setup.Environment;
import org.example.myapp.ApplicationConfiguration;
// more less relevant imports...
#Override
public void run(ApplicationConfiguration configuration, Environment environment) {
HikariDataSource hikariDataSource = loadDatabaseConfiguration(configuration.getDatabaseConfiguration());
new DbConfigurationLoader(hikariDataSource).start();
// this DSLContext object now has the reference to DataSource object that has an always-fresh password!
KeysDAO.initialize(DSL.using(hikariDataSource, SQLDialect.POSTGRES_10, new Settings().withExecuteWithOptimisticLocking(true)));
// rest of the app's config
}
private HikariDataSource loadDatabaseConfiguration(DatabaseConfiguration configuration) {
HikariDataSource hikariDataSource = new HikariDataSource();
hikariDataSource.setJdbcUrl(configuration.getJdbcUrl());
hikariDataSource.setDriverClassName(configuration.getDriverClassName());
hikariDataSource.setMinimumIdle(configuration.getMinimumIdle());
hikariDataSource.setMaximumPoolSize(configuration.getMaximumPoolSize());
hikariDataSource.setUsername(configuration.getJdbcUser());
return hikariDataSource;
}
private class DbConfigurationLoader extends Thread {
private final HikariDataSource hikariDataSource;
private final RdsTokenProvider rdsTokenProvider;
public DbConfigurationLoader(HikariDataSource hikariDataSource) {
this.rdsTokenProvider = new RdsTokenProvider();
this.hikariDataSource = hikariDataSource;
}
#Override
public void run() {
while (true) {
hikariDataSource.setPassword(rdsTokenProvider.getToken());
try {
Thread.sleep(/* token is valid for 15 minutes, so it makes sense to refresh it more often */);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
}
Hope this saves somebody some time in the future.

Sharing objects with all verticles instances

My application, an API server, is thought to be organized as follows:
MainVerticle is called on startup and should create all necessary objects for the application to work. Mainly a mongoDB pool of connections (MongoClient.createShared(...)) and a global configuration object available instance-wide. It also starts the HTTP Listener, several instances of a HttpVerticle.
HttpVerticle is in charge of receiving requests and, based the command xxx in the payload, execute the XxxHandler.handle(...) method.
Most of the XxxHandler.handle(...) methods will need to access the database. In addition, some others will also deploy additional verticles with parameters from the global conf. For example LoginHandler.handle(...) will deploy a verticle to keep user state while he's connected and this verticle will be undeployed when the user logs out.
I can't figure out how to get the global configuration object while being in XxxHandler.handle(...) or in a "sub"-verticle. Same for the mongo client.
Q1: For configuration data, I tried to use SharedData. In `MainVerticle.start() I have:
LocalMap<String, String> lm = vertx.sharedData().getLocalMap("conf");
lm.put("var", "val");
and in `HttpVerticle.start() I have:
LocalMap<String, String> lm = vertx.sharedData().getLocalMap("conf");
log.debug("var={}", lm.get("var"));
but the log output is var=null.... What am I doing wrong ?
Q2: Besides this basic example with a <String, String> map type, what if the value is a mutable Object like JsonObject which actually is what I would need ?
Q3: Finally how to make the instance of the mongo client available to all verticles?
Instead of getLocalMap() you should be using getClusterWideMap(). Then you should be able to operate on shared data accross the whole cluster and not just in one verticle.
Be aware that the shared operations are async and the code might look like (code in Groovy):
vertx.sharedData().getClusterWideMap( 'your-name' ){ AsyncResult<AsyncMap<String,String>> res ->
if( res.succeeded() )
res.result().put( 'var', 'val', { log.info "put succeeded: ${it.succeeded()}" } )
}
You should be able to use any Serializable objects in your map.

How to cache a Memcached connection using the java spymemcached client

I am learning how to cache objects using memcached with the spymemcached client from spymemcached examples
MemcachedClient c=new MemcachedClient(new InetSocketAddress("hostname", portNum));
// Store a value (async) for one hour
c.set("someKey", 3600, someObject);
// Retrieve a value (synchronously).
Object myObject=c.get("someKey");
I have noted that each time I want to cache or retrieve an object I create a new memcached client which am assuming is a new connection and that memcached has no connection pooling mechanism therefore users are advised to cache the connections to decrease overhead for reconnecting from this question opening closing and reusing connections.
My question is how do I cache this connection? Can someone please give me an example I can start from.
If you are wondering what I have tried, I tried to put my connection in the memcached but then I realized that I have to create a connection to get it. :)
Thanks in advance.
I have noted that each time I want to cache or retrieve an object I
create a new memcached client which am assuming is a new connection
Don't do this; spymemcache uses a single connection for ALL I/O to and from memcache; it's all done asychronously; from spymemcache docs...
Each MemcachedClient instance establishes and maintains a single
connection to each server in your cluster.
Just do the following once in your app; make sure the client is available to other services in your app so they can access it.
MemcachedClient memClient = new MemcachedClient(new InetSocketAddress(host, port));
Use memClient for I/O with memcache; no need to create a new instance of MemcachedClient each time; done. The link you provided answers all of your questions.
What is your deployment? web-app or standalone?
This just means that you should use reuse the connections that you open as opposed to opening a connection for each request. It doesn't make sense to store a connection instance in memcached.
Cacheing the connection in the case means caching it in your application (keeping it in memory and open), not actually storing the connection in memcached.
I did a little more research and stumbled on this question
Then I came up with my solution as,
first created a contextlistener
public class ContextListener implements ServletContextListener {
#Override
public void contextInitialized(ServletContextEvent sce) {
Memcached.createClient();
}
#Override
public void contextDestroyed(ServletContextEvent sce) {
}
}
then i added the listener to the deployment discriptor by adding these lines to web.xml
<listener>
<description>Used with memcached to initialize connection</description>
<listener-class>com.qualebs.managers.ContextListener</listener-class>
</listener>
I created a class Memcached and added these methods
static void createClient() {
try {
client = new MemcachedClient(new InetSocketAddress("localhost", 11211));
} catch (IOException ex) {
Logger.getLogger(Memcached.class.getName()).log(Level.SEVERE, ex.getMessage(), ex);
}
}
static MemcachedClient getClient() throws IOException {
return client;
}
Now anywhere I need to use memcached connection just call Memcached.getClient()
I hope that will help anybody else out there with the same question.