I have configured in my .NET Core 3.1 application the MongoDb waitQueueMultiple & maxPoolSize in the Mongo connection string.
I would like to set these parameters in the MongoClientSettings instead of in the connection string, but I read that from 2.12.3 version of MongoDb waitQueueSize will be deprecated and I don't understand what the alternative will be.
Do you have any suggestions?
This is how I configured now my code:
var url = new MongoUrl(_mongoDbConfiguration.ConnectionString);
var settings = MongoClientSettings.FromUrl(url);
settings.MaxConnectionPoolSize = _mongoDbConfiguration.MaxPoolSize;
settings.WaitQueueSize = _mongoDbConfiguration.WaitQueueMultiple;
var client = new MongoClient(settings);
var database = client.GetDatabase(url.DatabaseName);
return database;
Thanks,
Dave.
I found that documentation digging around.
This docs come from PyMongo but I think could be useful also for .NET one:
waitQueueMultiple has been deprecated without replacement. This option was a poor solution for putting an upper bound on queuing since it didn't affect queuing in other parts of the driver.
Once the pool reaches its maximum size, additional threads have to wait for sockets to become available. PyMongo does not limit the number of threads that can wait for sockets to become available and it is the application's responsibility to limit the size of its thread pool to bound queuing during a load spike. Threads are allowed to wait for any length of time unless waitQueueTimeoutMS is defined.
The default value for waitQueueTimeout is 2 minutes as per csharp driver docs.
So in my application I give the possibility to set MaxConnectionPoolSize and WaitQueueTimeout.
If not configured the application will be taken the default values.
Related
I'm reading the documentation of orion Context Broker and in the command line arguments I dont see any argument to set the read preference to my replicaset of mongoDB. In my application I need to set that the read preference have the option nearest to avoid bottle necks in high query traffic periods. Does anyone know if is possible?
Current Orion version (3.3.1) doesn't allow to set read preference. There is an open issue in the Orion repository about implement the -mongoUri CLI parameter to allow setting the MongoDB connection URI (so you could add for instance &readPreference=secondary to it).
Alternatively, you could hack the Orion source code to build an specific version for you with the readPreference value you want. Have a look to composeMongoUri() function. It seems it is a matter of just addding uri += optionPrefix + "readPreference=<whatever you want>"; at the end.
It is not a smart soluction (it is not flexible and you would need to rebuild Orion if you want to change the setting) but it could be a valid workaround while -mongoUri gets implemented.
This question is for pg-promise, its recommended usage pattern & based on following assumption,
It does-not make sense to create more than a single pgp instance, if they are connecting to same DB(also enforced by the good warning message of "Creating a duplicate database object for the same connection.")
Given:
I have 2 individual packages which need DB connection, currently they take connection string in constructor from outside and create connection object inside them, which leads to the warning of duplicate connection object and is fair as they both talk to same DB and there is a possibility for optimisation here(since i am in control of those packages).
Then: To prevent this, i thought of implementing dependency injection, for which i pass a resolve function in libraries constructor which gives them the DB connection object.
Issue: There are some settings which are at top level like parsers and helpers and transaction modes which may be different for each of these packages what is the recommendation for such settings or is there a better patterns to address these issues.
EG:
const pg = require('pg-promise');
const instance = pg({"schema": "public"});
instance.pg.types.setTypeParser(1114, str => str);//UTC Date which one library requires other doesnt
const constring = "";
const resolveFunctionPackage1 = ()=>instance(constring);
const resolveFunctionPackage2 = ()=>instance(constring);
To sum up: What is the best way to implement dependency injection for pg-promise?
I have 2 individual packages which need DB connection, currently they take connection string in constructor from outside and create connection object inside them
That is a serious design flaw, and it's is never gonna work well. Any independent package that uses a database must be able to reuse an existing connection pool, which is the most valuable resource when it comes to connection usage. Head-on duplication of a connection pool inside an independent module will use up existing physical connections, and hinder performance of all other modules that need to use the same physical connection.
If a third-party library supports pg-promise, it should be able to accept instantiated db object for accessing the database.
And if the third-party library supports the base driver only, it should at least accept an instantiated Pool object. In pg-promise, db object exposes the underlying Pool object via db.$pool.
what happens when they want to set conflicting typeparsers?
There will be a conflict, because pg.types is a singleton from the underlying driver, so it can only be configured in one way. It is an unfortunate limitation.
The only way to avoid it, is for reusable modules to never re-configure the parsers. It should only be done within the actual client application.
UPDATE
Strictly speaking, one should avoid splitting a database-access layer of an application into multiple modules, there can be a number of problems to follow that.
But specifically for separation of type parsers, the library supports setting custom type parsers on the pool level. See example here. Note that the update is just for TypeScript, i.e. in JavaScript clients it has been working for awhile.
So you still can have your separate module create its own db object, but I would advise that you limit its connection pool size to the minimum then, like 1:
const moduleDb = pgp({
// ...connection details...
max: 1, // set pool size to just 1 connection
types: /* your custom type parsers */
});
Note: I realise there is a similar question on SO but it talks about an old version of Casbah, plus, the behaviour explained in the answer is not what I see!
I was under the impression that Casbah's MongoClient handled connection pooling. However, doing lsof on my process I see a big and growing number of mongodb connections, which makes me doubt this pooling actually exists.
Basically, this is what I'm doing:
class MongodbDataStore {
val mongoClient = MongoClient("host",27017)("database")
var getObject1(): Object1 = {
val collection = mongoClient("object1Collection")
...
}
var getObject2(): Object2 = {
val collection = mongoClient("object2Collection")
...
}
}
So, I never close MongoClient.
Should I be closing it after every query? Implement my own pooling? What then?
Thank you
Casbah is a wrapper around the MongoDB Java client, so the connection is actually managed by it.
According to the Java driver documentation (http://docs.mongodb.org/ecosystem/drivers/java-concurrency/) :
If you are using in a web serving environment, for example, you should
create a single MongoClient instance, and you can use it in every
request. The MongoClient object maintains an internal pool of
connections to the database (default maximum pool size of 100). For
every request to the DB (find, insert, etc) the Java thread will
obtain a connection from the pool, execute the operation, and release
the connection. This means the connection (socket) used may be
different each time.
By the way, that's what I've experienced in production. I did not see any problem with this.
In my applicaiton I have a DB to which the code will periodically connect, but it will be quite rarely use (maybe once a day/week).
Can I create just connect while module (app) init and then use it across the module while application run lifecycle?
var conn = mongoose.createConnection(process.env.SOME_DB)
I'm not sure should I have a keep alive option as suggested in mongoose docs:
options.server.socketOptions = options.replset.socketOptions = { keepAlive: 1 };
mongoose.connect(uri, options);
or standard auto reconnect feature will be enough?
An i'm also not what is "long running applications"? Actually any real-time service is long running application, should keep alive be enabled for all such services in production?
Also not sure what are Connection pools and how they can affect.
There is a reference to this in the Mongoose documentation:
http://mongoosejs.com/docs/connections.html
And yes, it's generally a good idea.
Also in that document Connection pools are explained. But generally speaking, Mongoose is keeping several socket connections open to the server/replica-set/mongos instances rather than one to allow concurrent processing of requests. Yes, even with async call-backs on IO there is wait time, so Connection pools allow another channel to talk on while one is busy.
And yes, it's generally a good idea.
An asp.net application I am working on may have a couple hundred users trying to connect. We get an error that the maximum number of connections in the pool has been reached. I understand the concept of connection pools in ADO.NET although in testing I've found that a connection is left "sleeping" on the ms sql 2005 server days after the connection was made and the browser was then closed. I have tried to limit the connection lifetime in the connection string but this has no effect. Should I push the max number of connections? Have I completely misdiagnosed the real problem?
All of your database connections must either be wrapped in a try...finally:
SqlConnection myConnection = new SqlConnection(connString);
myConnection.Open();
try
{
}
finally
{
myConnection.Close();
}
Or...much better yet, create a DAL (Data Access Layer) that has the Close() call in its Dispose method and wrap all of your DB calls with a using:
using (MyQueryClass myQueryClass = new MyQueryClass())
{
// DB Stuff here...
}
A few notes: ASP.NET does rely on you to release your Connections. It will release them through GC after they've gone out of scope but you can not count on this behavior as it may take a very long time for this to kick in - much longer than you may be able to afford before your connection pool runs out. Speaking of which - ASP.NET actually pools your connections to the database as you request them (recycling old connections rather than releasing them completely and then requesting them anew from the database). This doesn't really matter as far as you are concerned: you still must call Close()!
Also, you can control the pooling by using your connection string (e.g. Min/Max pool size, etc.). See this article on MSDN for more information.