I have a problem with my database connections. Whenever my project runs a db.run from any of my microservices it creates a new connection to my db. I have a play-scala-slick project and I'm using Amazon Web Services (AWS).
Do I have to manually open and close connections e.g. for every query from any of my microservices or is there a clean, proper, smooth way to handle my connections? Thanks!
A solution is to steh the timeout variables in aws rds.
First you have to set up a Parameter Group
set global wait_timeout=3;
set global interactive_timeout=3;
Related
I deployed a RDS Aurora cluster for postgresql 11 in AWS. My lambda is talking to this cluster via IAM authentication. Since lambda is serverless, I have to create a connection to database every time my lambda is triggered and close the connection when it finishes. It is not great since creating db connection is heavy and takes time. I have used xray to observe the connection performance which takes 150ms to create a new connection. It also gives a lot load on db cluster since there will be many short lived connections on db.
After some searching I found RDS proxy is designed to solve the problem. So I deployed RDS proxy to use username/password to connect to my Aurora cluster. And my lambda connects to RDS proxy via IAM authentication.
When I observe the creating connection performance, it becomes worse. It takes more than 500ms to create a connection and sometimes it even takes more than 1 second.
How come it is worse when using RDS proxy? Is there anything I didn't configure in the proxy?
We're encountering issues with using Flyway for database migrations with multiple nodes in parallel, backed by a PostgreSQL database behind PgBouncer with transaction pooling.
The problem is that when multiple nodes start up at the same time, Flyway gets an exclusive lock but this seems to be a session lock, which isn't supported by PgBouncer transaction pooling (as multiple nodes may get the same session). This then causes each node to not start up because they've locked each other.
Is there anything we can change or configure in Flyway to support this? We'd prefer not to switch away from transaction pooling if possible, as that's our main motivation for using PgBouncer.
At the moment, Flyway doesn't support PgBouncer, so you're seeing errors because of that lack of support. No work arounds from the developers currently. I'd suggest opening an issue on the Community Github. That's the best way to get changes in.
As a workaround, we're currently configuring two data sources for our application - one to PgBouncer as normal, and another with a single connection that's used solely for Flyway that bypasses PgBouncer and connects directly to the PostgreSQL back-end.
I'm looking for a way to create a connection pool for many DBs on the same DB server (PostgreSQL Aurora).
This means that I need the ability of changing the target DB of a connection at run time.
Currently I'm using HikariCP for connection pooling, in a stack of Spring Boot and JHispter.
Background:
we need to deploy a multi-tenancy micro-service system with a single DB server (to be specific, a single AWS Aurora PostgreSQL instance)
our solution of multi-tenancy is that each tenant has a DB, in that DB we have many schema for each service. All the DBs are in the same AWS Aurora instance.
Our problem:
with this deployment, we have a connection pool for each (tenant x micro-service instance).
This leads to a huge number of connections.
Ie: with the pool size of 50 connections/pool. We need: 500 tenants x 20 micro-service instances x 50 connections/pool = 500000 connections.
The maximum connections allowed on any Aurora DB is 16000, and actually by default the "max_connections" parameter is typically set to something lower.
So now I'm looking for a way to make our pooling scope larger, so that many tenants can share the same pool. Since we use only 1 Aurora server instance, I think it's possible to create a connection pool that can be shared between many tenants.
Is there any way to have a connection pool that can switch the DB at run time?
Unless Aurora has done some customization on this, you cannot change the database of a connection once it is established in PostgreSQL. You can still use a pooler, but it will effectively be a separate pool for each database. This is pretty fundamental, there is nothing you can do about it.
I have read that you can replicate a Cloud SQL database to MySQL. Instead, I want to replicate from a MySQL database (that the business uses to keep inventory) to Cloud SQL so it can have up-to-date inventory levels for use on a web site.
Is it possible to replicate MySQL to Cloud SQL. If so, how do I configure that?
This is something that is not yet possible in CloudSQL.
I'm using DBSync to do it, and working fine.
http://dbconvert.com/mysql.php
The Sync version do the service that you want.
It work well with App Engine and Cloud SQL. You must authorize external conections first.
This is a rather old question, but it might be worth noting that this seems now possible by Configuring External Masters.
The high level steps are:
Create a dump of the data from the master and upload the file to a storage bucket
Create a master instance in CloudSQL
Setup a replica of that instance, using the external master IP, username and password. Also provide the dump file location
Setup additional replicas if needed
Voilà!
NewRelic gives nice database analyses, however it seems to track only the web app's transactions.
I have independently managed servers which query and load my Heroku postgresql database. Is there a way I can get diagnostics and analysis of the database activity so that it will include all connections to it?
New Relic application monitoring will only collect data on database queries that are part of a web transaction or background task that is being monitored. If you're using one of New Relic's supported languages to query your database, you may be able to track that code as a background task (see https://newrelic.com/docs/features/monitoring-background-processes). If you would like a general monitoring plugin for your postgresql database, you could check out the postgresql plugin for New Relic (created and supported by Boundless): http://newrelic.com/plugins/boundless/109.
You should also try Heroku PG Extras: https://github.com/heroku/heroku-pg-extras. That will give info about cache hit, indexes, long queries, etc.