This question is for pg-promise, its recommended usage pattern & based on following assumption,
It does-not make sense to create more than a single pgp instance, if they are connecting to same DB(also enforced by the good warning message of "Creating a duplicate database object for the same connection.")
Given:
I have 2 individual packages which need DB connection, currently they take connection string in constructor from outside and create connection object inside them, which leads to the warning of duplicate connection object and is fair as they both talk to same DB and there is a possibility for optimisation here(since i am in control of those packages).
Then: To prevent this, i thought of implementing dependency injection, for which i pass a resolve function in libraries constructor which gives them the DB connection object.
Issue: There are some settings which are at top level like parsers and helpers and transaction modes which may be different for each of these packages what is the recommendation for such settings or is there a better patterns to address these issues.
EG:
const pg = require('pg-promise');
const instance = pg({"schema": "public"});
instance.pg.types.setTypeParser(1114, str => str);//UTC Date which one library requires other doesnt
const constring = "";
const resolveFunctionPackage1 = ()=>instance(constring);
const resolveFunctionPackage2 = ()=>instance(constring);
To sum up: What is the best way to implement dependency injection for pg-promise?
I have 2 individual packages which need DB connection, currently they take connection string in constructor from outside and create connection object inside them
That is a serious design flaw, and it's is never gonna work well. Any independent package that uses a database must be able to reuse an existing connection pool, which is the most valuable resource when it comes to connection usage. Head-on duplication of a connection pool inside an independent module will use up existing physical connections, and hinder performance of all other modules that need to use the same physical connection.
If a third-party library supports pg-promise, it should be able to accept instantiated db object for accessing the database.
And if the third-party library supports the base driver only, it should at least accept an instantiated Pool object. In pg-promise, db object exposes the underlying Pool object via db.$pool.
what happens when they want to set conflicting typeparsers?
There will be a conflict, because pg.types is a singleton from the underlying driver, so it can only be configured in one way. It is an unfortunate limitation.
The only way to avoid it, is for reusable modules to never re-configure the parsers. It should only be done within the actual client application.
UPDATE
Strictly speaking, one should avoid splitting a database-access layer of an application into multiple modules, there can be a number of problems to follow that.
But specifically for separation of type parsers, the library supports setting custom type parsers on the pool level. See example here. Note that the update is just for TypeScript, i.e. in JavaScript clients it has been working for awhile.
So you still can have your separate module create its own db object, but I would advise that you limit its connection pool size to the minimum then, like 1:
const moduleDb = pgp({
// ...connection details...
max: 1, // set pool size to just 1 connection
types: /* your custom type parsers */
});
Related
In Apple's TicTacFish sample project, Step 3 provides an example of a LocalNetworkActorSystem based on Bonjour. At various points in the implementation, reference is made to building up a mapping from actor IDs to Connections/Peers, such that we can identify the correct connections to send calls and replies on (the sample project just broadcasts to all connections.) I've tried various approaches but can't get past the simple hurdle that:
the methods required by DistributedActorSystem (e.g., SampleLocalNetworkActorSystem.remoteCall(on:target:invocation:throwing:returning:) do not allow for additional arguments, preventing me from passing along data about the originating connection that could be used to uniquely identify the peer
the peer initialization happens separately from the relevant ActorID methods (e.g., SampleLocalNetworkActorSystem.resolve(id:as:), SampleLocalNetworkActorSystem.assignID(_:), SampleLocalNetworkActorSystem.actorReady(_:)) which prevents me from knowing which connection this actor initialization corresponds to.
Has anyone figured out a way to get this to work?
Trying to do some simple things with mongodb my mind got stuck in something that feels kinda strange for me.
client = MongoClient(connection_string)
db = client.database
print(db)
client.close()
I thought that when make a connection it is used only this one along the rest of the code until the close() method. But it doesn't seem to work that way... I don't know how I ended up having 9 connections when it supposed to be a single one, and even if each 'request' is a connection there's too many of them
For now it's not a big problem, just bothers me the fact that I don't know exactly how this works!
When you do new MongoClient(), you are not establishing just one connection. In fact you are creating the client, that will have a connection pool. When you do one or multiple requests, the driver uses an available connection from the pool. When the use is complete, the connection goes back to the pool.
Calling MongoClient constructor every time you need to talk to the db is a very bad practice and will incur a penalty for the handshake. Use dependency injection or singleton to have MongoClient.
According to the documentation, you should create one client per process.
Your code seems to be the correct way if it is a single thread process. If you don't need any more connections to the server, you can limit the pool size by explicitly specifying the number:
client = MongoClient(host, port, maxPoolSize=<num>).
On the other hand, if the code might later use the same connection, it is better to simply create the client once in the beginning, and use it across the code.
I'm writing a client library for a custom TCP-based protocol. I'd like the library to easily permit users to make requests on multiple threads, which is the reason for my question. My library is intended to replace a slow, Python-based implementation which I've identified as a bottleneck in our pipeline, so request throughput is crucial here. So far, I've used tokio's networking facilities so that, once I've got concurrent sockets, I'll be able to execute requests in parallel. After the application opens a socket, the protocol allows it to re-use the connection for subsequent requests.
When a session begins, the applications sends an initial request and receives an authorization token in return, which it must include with future requests. Requests are frequent and small but irregular in their contents, so I'd like to pool my TCP sockets, like (I imagine) a web browser does to support the keep-alive mechanism. I very much want to avoid having each request spin up a new socket -- even though it could re-use the existing authorization token -- because of the delays associated with the TCP's three-way handshake and slow start.
The classic solution to this problem is a connection pool, like you'd use in a web application to connect to a database. The trouble is, I haven't been able to find a good connection pool for sockets, and I'd prefer to avoid introducing the complexity of one I write myself. There's r2d2 (see here), for instance, but that only supports database connections. Meanwhile, tk_pool (here) hasn't been updated in two years, which is not encouraging.
This feels like a common task, so I'm surprised I haven't yet found a simple way to do this. I'm new to Rust's async/await features and tokio, so I may well be missing something essential. Here's the question, simply:
How can I distribute many bits of IO across several sockets, each connected to the same host? Put another way, how can I have a pool of workers take temporary ownership of (or gain a mutable reference to) the first available of a set of equivalent resources?
I'm open to all manner of suggestions, but to avoid making this question opinion based, I think the central question is one of fact: What is the idiomatic, async Rust way to do connection pooling?
Thanks!
Here's some pseudo-code that outlines how I'm imagining my code would look, once I've identified the right way to do this:
struct Session {
pool: ConnectionPool<tokio::net::TcpStream>,
// authorization token, etc.
}
impl Session {
async fn make_request(&mut self, parameters...) -> Result<Response, Error> {
let sock = self.pool.borrow_socket(); // probably requires &mut self. Won't be able to distribute the Session object if it requires mutable references. Will I need to use Cell?
sock.write(format!("request: {}", parameters)).await?;
let results = sock.read().await?;
Ok(parse(results)?)
// sock is dropped, which returns it to the pool; alternatively, maybe you've got to call, e.g., sock.release().
}
}
When my server starts up, it will create a connection to a mongo database, grab that *mgo.Session and store it in a "server" struct for handlers defined on that struct to use to serve requests.
I see two ways to actually execute this.
1) Save the `*mgo.Session` in the struct.
The upside of this is that you can call Session.Copy in each handler before using the session to have connection isolation. The downside is that you need to call .DB("x").C("y") for a specific x and y for each handler. If you wanted to change this, you need to find each instance where you're using it and change it. That's less than ideal.
2) Store the `*mgo.Database` or even `*mgo.Collection` object on the server struct.
The upside is that you can configure it in one place and use it. The downside is that there appears to be no way to use Copy() on the thing for connection isolation.
How do you recommend storing the mongo session so that you can cleanly use the connection and provide isolation between requests?
DBIx::Class::Manual::Intro
suggests connecting to the database as follows
my $schema = MyApp::Schema->connect(...)
explicitly providing connection details such as the password.
I want to connect to the same database from multiple different scripts, and it would be unwise to code the same connection parameters into each of the programs separately.
What is the "official" way to create a connection method with fixed connection details?
I realize that I can write something like this
package MyApp::Schema;
use base qw/DBIx::Class::Schema/;
sub my_connect {
$_[0]::SUPER->connect(...);
}
1;
Is this approach recommended?
I realize that providing different connection details may be useful for testing scripts, but in reality we do not yet use testing scripts, so this is currently irrelevant for our team.
Put your connection details in a config file, create a utility to return the connection and read the config details like you showed, or as a factory type function. Make the config dependant on the environment and you'll have testing capabilities for free.