I am trying to understand the difference between 2 configurable parameters while creating a connection pool using r2dbc-pool.
I was able to configure the connection pool with the help of the below post:
Connection pool size with postgres r2dbc-pool
But wanted to understand the difference while configuring max size and initial size while creating
ConnectionFactory connectionFactory = ConnectionFactories.get(ConnectionFactoryOptions.builder()
.option(DRIVER, "pool")
.option(PROTOCOL, "postgresql")
.option(HOST, host)
.option(USER, user)
.option(PASSWORD, password)
.option(MAX_SIZE, 30)
.option(INITIAL_SIZE, 10)
.option(DATABASE, database)
.build());
ConnectionPoolConfiguration configuration = ConnectionPoolConfiguration.builder(connectionFactory)
.maxIdleTime(Duration.ofMinutes(30))
.initialSize(initialSize)
.maxSize(maxSize)
.initialSize(20)
.maxCreateConnectionTime(Duration.ofSeconds(1))
.build();
There isn't a difference really, these are just different ways to create a connection pool.
The initial size is the number of open connections to the database once the application starts.
You can check that with the query:
select * from pg_stat_activity;
You'll see exactly the number you specified as initial size.
Max size is the maximum nuber of connections that can be open at the same time (you do not control that directly, underlying pool implementation will scale the number of opened connections with higher load)
Related
I am trying to create a connection pool by using HikariCP for Postgres. The pool should always have one active session/connection and the remaining connections should be on-demand which Hikari will take care of. To do the same, configured Datasource like below
private static HikariDataSource dataSource = null;
HikariConfig config = new HikariConfig();
config.setJdbcUrl("jdbc:postgresql://hostname:port/dbname");
config.setUsername("USERNAME");
config.setPassword("PASSWORD");
config.setMinimumIdle(1);
config.setMaximumPoolSize(5);
config.setIdleTimeout(10000);
dataSource = new HikariDataSource(config);
MinimumIdle(1) always keeps the one active session/open connection to DB unless the program or server closed and the remaining 4 connections will be created on-demand if they do not exist already in the pool.
IdleTimeout(10000) removes any connection object which is idle in the pool for more than 10 seconds and observed that it won't apply to any of the connections set to MinimumIdle(Here it is 1).
Is my understanding correct and serves my requirement? Appreciate your suggestions here.
I have an Azure Durable Function that interacts with a PostgreSQL database, also hosted in Azure.
The PostgreSQL database has a connection limit of 50, and furthermore, my connection string limits the connection pool size to 40, leaving space for super user / admin connections.
Nonetheless, under some loads I get the error
53300: remaining connection slots are reserved for non-replication superuser connections
This documentation from Microsoft seemed relevant, but it doesn't seem like I can make a static client, and, as it mentions,
because you can still run out of connections, you should optimize connections to the database.
I have this method
private IDbConnection GetConnection()
{
return new NpgsqlConnection(Environment.GetEnvironmentVariable("PostgresConnectionString"));
}
and when I want to interact with PostgreSQL I do like this
using (var connection = GetConnection())
{
connection.Open();
return await connection.QuerySingleAsync<int>(settings.Query().Insert, settings);
}
So I am creating (and disposing) lots of NpgsqlConnection objects, but according to this, that should be fine because connection pooling is handled behind the scenes. But there may be something about Azure Functions that invalidates this thinking.
I have noticed that I end up with a lot of idle connections (from pgAdmin):
Based on that I've tried fiddling with Npgsql connection parameters like Connection Idle Lifetime, Timeout, and Pooling, but the problem of too many connections seems to persist to one degree or another. Additionally I've tried limiting the number of concurrent orchestrator and activity functions (see this doc), but that seems to partially defeat the purpose of Azure Functions being scalable. It does help - I get fewer of the too many connections error). Presumably If I keep testing it with lower numbers I may even eliminate it, but again, that seems like it defeats the point, and there may be another solution.
How can I use PostgreSQL with Azure Functions without maxing out connections?
I don't have a good solution, but I think I have the explanation for why this happens.
Why is Azure Function App maxing out connections?
Even though you specify a limit of 40 for the pool size, it is only honored on one instance of the function app. Note that that a function app can scale out based on load. It can process several requests concurrently in the same function app instance, plus it can also create new instances of the app. Concurrent requests in the same instance will honor the pool size setting. But in the case of multiple instances, each instance ends up using a pool size of 40.
Even the concurrency throttles in durable functions don't solve this issue, because they only throttle within a single instance, not across instances.
How can I use PostgreSQL with Azure Functions without maxing out connections?
Unfortunately, function app doesn't provide a native way to do this. Note that the connection pool size is not managed by the function runtime, but by npgsql's library code. This library code running on different instances can't talk to each other.
Note that, this is the classic problem of using shared resources. You have 50 of these resources in this case. The most effective way to support more consumers would be to reduce the time each consumer uses the resource. Reducing the Connection Idle Lifetime substantially is probably the most effective way. Increasing Timeout does help reduce errors (and is a good choice), but it doesn't increase the throughput. It just smooths out the load. Reducing Maximum Pool size is also good.
Think of it in terms of locks on a shared resource. You would want to take the lock for the minimal amount of time. When a connection is opened, it's a lock on one of the 50 total connections. In general, SQL libraries do pooling, and keep the connection open to save the initial setup time that is involved in each new connection. However, if this is limiting the concurrency, then it's best to kill idle connections asap. In a single instance of an app, the library does this automatically when max pool size is reached. But in multiple instances, it can't kill another instance's connections.
One thing to note is that reducing Maximum Pool Size doesn't necessarily limit the concurrency of your app. In most cases, it just decreases the number of idle connections - at the cost of - paying the initial setup time when a new connection will need to be established at a later time.
Update
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT might be useful. You can set this to 5, and pool size to 8, or similar. I would go this way if reducing Maximum Pool Size and Connection Idle Lifetime is not helping.
This is where Dependency Injection can be really helpful. You can create a singleton client and it will do the job perfectly. If you want to know more about service lifetimes you can read it here in docs
First add this nuget Microsoft.Azure.Functions.Extensions.DependencyInjection
Now add a new class like below and resolve your client.
[assembly: FunctionsStartup(typeof(Kovai.Serverless360.Functions.Startup))]
namespace MyFunction
{
class Startup : FunctionsStartup
{
public override void Configure(IFunctionsHostBuilder builder)
{
ResolveDependencies(builder);
}
}
public void ResolveDependencies(IFunctionsHostBuilder builder)
{
var conStr = Environment.GetEnvironmentVariable("PostgresConnectionString");
builder.Services.AddSingleton((s) =>
{
return new NpgsqlConnection(conStr);
}
}
}
Now you can easily consume it from any of your function
public FunctionA
{
private readonly NpgsqlConnection _connection;
public FunctionA(NpgsqlConnection conn)
{
_connection = conn;
}
public async Task<HttpResponseMessage> Run()
{
//do something with your _connection
}
}
Here's an example of using a static HttpClient, something which you should consider so that you don't need to explicitly manage connections, rather allow your client to do it:
public static class PeriodicHealthCheckFunction
{
private static HttpClient _httpClient = new HttpClient();
[FunctionName("PeriodicHealthCheckFunction")]
public static async Task Run(
[TimerTrigger("0 */5 * * * *")]TimerInfo healthCheckTimer,
ILogger log)
{
string status = await _httpClient.GetStringAsync("https://localhost:5001/healthcheck");
log.LogInformation($"Health check performed at: {DateTime.UtcNow} | Status: {status}");
}
}
I have a play application using Slick that I want to test using Spec2, but I keep getting the error org.postgresql.util.PSQLException: FATAL: sorry, too many clients already. I have tried to shut down the database connection by using
val mockApp = new GuiceApplicationBuilder()
val db = mockApp.injector.instanceOf[DBApi].database("default")
...
override def afterAll = {
db.getConnection().close()
db.shutdown()
}
But the error persists. The Slick configuration is
slick.dbs.default.driver="slick.driver.PostgresDriver$"
slick.dbs.default.db.driver="org.postgresql.Driver"
slick.dbs.default.db.url="jdbc:postgresql://db:5432/hygge_db"
slick.dbs.default.db.user="*****"
slick.dbs.default.db.password="*****"
getConnection of DbApi either gets connection from underlying data-source's (JdbcDataSource I presume) pool or creates a new one. I see no pool specified in your configuration, so I think it always creates a new one for you. So if you didn't close connection inside the test - getConnection won't help - it will just try to create a new one or take random connection from pool (if pooling is enabled).
So the solution is to either configure connection pooling:
When using a connection pool (which is always recommended in
production environments) the minimum size of the connection pool
should also be set to at least the same size. The maximum size of the
connection pool can be set much higher than in a blocking application.
Any connections beyond the size of the thread pool will only be used
when other connections are required to keep a database session open
(e.g. while waiting for the result from an asynchronous computation in
the middle of a transaction) but are not actively doing any work on
the database.
so you can just set maximum available connections number in your config:
connectionPool = 5
Or you can share same connection (you'll probably have to ensure sequentiality then):
object SharedConnectionForAllTests{
val connection = db.getConnection()
def close() = connection.close()
}
It's better to inject it with Spring/Guice of course, so you could conviniently manage connection's lifecycle.
Just looking for an explanation of rationale for this bit of code (PoolUtiltites:293 in version 2.2.4):
dataSource.setLoginTimeout((int) TimeUnit.MILLISECONDS.toSeconds(Math.min(1000L, connectionTimeout)));
This code and the setConnectionTimeout method means that I get this behaviour:
connectionTimeout == 0, then loginTimeout = Integer.MAX_VALUE
connectionTimeout > 0 && < 100, then HikariConfig throws IllegalArgumentException
connectionTimeout >= 100 && <= 1000, then loginTimeout = connectionTimeout
connectionTeimout > 1000, then loginTimeout = 1000
That looks really weird to me!
It's almost like the Math.min should be Math.max ???
In my current project I'd like to fail connections after 30s, which is impossible in the current setup.
I'm using the 4.1 postgres jdbc driver, but I think this is not relevant to the issue above.
Many thanks - and cool pooling library!!!
Ok, there are a couple of moving parts here. First, Math.min() is a bug, it should be Math.max(). In light of that (it will be fixed) consider the following:
It is important to note that connections are created asynchronously in the pool. The setConnectionTimeout() sets the maximum time (in milliseconds) that a call to getConnection() will wait for a connection before timing out.
The DataSource loginTimeout is the maximum time that physical connection initiation to the database can take before timing out. Because HikariCP obtains connections asynchronously, if the connection attempt fails, HikariCP will continue to retry, but your calls to getConnection() will timeout appropriately. We are using the connectionTimeout in kind of a double duty for loginTimeout.
For example, lets say the pool is completely empty, and you have configured a connectionTimeout of 30 seconds. When you call getConnection() HikariCP, realizing that there are no idle connections available, starts trying to obtain a new one. There is little point in having a loginTimeout exceeding 30 seconds, in this case.
The intent of the Math.max() call is to ensure that we never set loginTimeout to 0 if the user has configured connectionTimeout to 250ms. TimeUnit.MILLESECONDS.toSeconds() would return 0 without the Math.max(). If the user has configured a connectionTimeout of 0, meaning they never want to timeout, the time conversion of Integer.MAX_VALUE results in several thousand years as a timeout (virtually never).
Having said that, and in light of how HikariCP connections to the database are obtained asynchronously, even without the Math.max() fix, you should be able to achieve application-level connection timeouts of 30s. Unless physical connections to your database exceed 1000ms you would be unaffected by the Math.min().
We are putting out a 2.2.5-rc3 release candidate in the next few hours. I will slot this fix in.
I can't find any documentation for the node-postgres drive on setting the maximum connection pool size, or even finding out what it is if it's not configurable. Does anyone know how I can set the maximum number of connections, or what it is by default?
defaults are defined in node-postgres/lib/defaults https://github.com/brianc/node-postgres/blob/master/lib/defaults.js
poolSize is set to 10 by default, 0 will disable any pooling.
var pg = require('pg');
pg.defaults.poolSize = 20;
Note that the pool is only used when using the connect method, and not when initiating an instance of Client directly.
node.js is single threaded why want to have more then 1 connection to db per process ? Even when you will cluster node.js processes you should have 1 connection per process max. Else you are doing something wrong.