We are new to Google Cloud SQL and have been trying to integrate pgbouncer with Google Cloud SQl Postgres and authenticate database users with SECURITY DEFINER function (which queries pg_shadow)
Our Configuration:
Server -> Pgbouncer + Cloud sql proxy (side car) -> Cloud SQL Postgres
Problem:
But as cloud sql postgres actually does not allow to read pg_shadow from a privileged user (i.e postgres user is not a superuser). This makes it impossible to setup pgbouncer with SECURITY DEFINER function.
Cloud SQL doesn't provide customers to use superuser (cloudsqladmin)
We've read through many articles (mostly cloud-proxy issues) where they have suggested to use pgbouncer but have not elaborated on the above problem.
Options not applicable:
Application level pooling (not feasible right now for us)
Authenticating using auth_file eg. users_list.txt (not recomended, needs manual management of database users)
What we are looking for:
We intend to run a single instance of cloudsql-proxy and pgbouncer which proxies and pools connections to cloudsql postgres database.
We would appreciate your help guys!
Related
Overall I'm trying to create a Datastream Connection to a Postgres database in Cloud SQL.
As I'm trying to configure it all through Terraform, I'm stuck on how I should create a Replication Slot. This guide explains how to do it through the Postgres Client and running SQL commands, but I thought there might be a way to do it in the Terraform configuration directly.
Example SQL that I would like to replicate in Terraform:
ALTER USER [CURRENT_USER] WITH REPLICATION;
CREATE PUBLICATION [PUBLICATION_NAME] FOR ALL TABLES;
SELECT PG_CREATE_LOGICAL_REPLICATION_SLOT('[REPLICATION_SLOT_NAME]', 'pgoutput');
If not, does anyone know how to run the Postgres SQL commands against the Cloud SQL database through Terraform?
I have setup the Datastream and Postgres connection for all other parts. I'm expecting that there is a Terraform setting I'm missing or a way to run Postgres commands against the Google Cloud SQL Postgres database.
Unfortunately, there is no terraform resource for specifying a replication slot on a google_sql_database_instance.
Specifically, I would like to see which users (coming from PostgreSQL Drivers / RDS API) are accessing the specific tables inside RDS.
I believe that finding users who are accessing RDS API can be done using CloudTrail. I'm unsure about how to go on about postgres driver users.
I was using Postgres addon on Heroku and was connecting to the database using DATABASE_URL env var.
I now need to switch to Google Cloud Platform Postgres. I've created an instance and successfully added my local connection to the Authorised networks. Yet it seems that Heroku does not provide a static IP for its apps.
My question is then – is it possible to connect my Heroku app to a Postgres database from the Google Cloud Platform? If yes – what's the best way to do it?
You will want to run the Cloud SQL Proxy alongside your application. This will allow your Heroku App to connect to Cloud SQL without the need to worry about changing and adding IPs to your Authorized Networks.
This thread might be useful for your use-case. Node and Cloud SQL with Heroku
My team and I are currently experiencing an issue where we can't connect to Cloud SQL's Postgres instance(s) from anything other than the psql cli tool. We get a too many connections for database "postgres" error (in PGAdmin, DBeaver, and our node typeorm/pg backend). It initially happened on our (only) Postgres database instance. After restarting, stopping and starting again, increasing machine CPU/memory proved to do nothing, I deleted the database instance entirely and created a new one from scratch.
However, after a few hours the problem came back. I know that we're not actually having too many connections as I am able to query pg_stat_activity from psql command line and see the following:
Only one of those (postgres username) connections is ours.
My coworker also can't connect at all - not even from psql cli.
If it matters, we are using PostgreSQL 13, europe-west2 (London), single zone availability, db-g1-small instance with 1.7GB memory, 10GB HDD, and we have public IP enabled and the correct IP addresses whitelisted.
I'd really appreciate if anyone has any insights into what's causing this.
EDIT: I further increased the instance size (to no longer be a shared core), and I managed to successfully connect my backend to it. However my psql cli no longer works - it appears that only the first client to connect is allowed to connect after a restart (even if it disconnects, other clients can't connect...).
From the error message, it is clear that the database "postgres" has a custom connection limit (set, for example, by ALTER DATABASE postgres CONNECTION LIMIT 1). And apparently, it is quite small. Why is everyone try to connect to that database anyway? Usually 'postgres' database is reserved for maintenance operations, and you should create other databases for daily use.
You can see the setting with:
select datconnlimit from pg_database where datname='postgres';
I don't know if the low setting is something you did, or maybe Google does it on its own for their cloud offering.
#jjanes had the right idea/mention.
I created another database within the Cloud SQL instance that wasn't named postgres and then it was fine.
It wasn't anything to do with maximum connection settings (as this was within Google Cloud SQL) or not closing connections (as TypeORM/pg does this already).
I have pulled the SHARED_DATABASE_URL from heroku config
SHARED_DATABASE_URL => postgres://username:xxxx#host.com/db_name
I am using pgAdmin to try to connect to it but it keeps on timing out. Do I need to specify a port? What am i missing?
You can use this command to connect to psql.
heroku pg:psql
If you are happy to change to postgres 9.1 you can use the newly launched development database, which permits connections via normal postgres tools. Read more at https://postgres.heroku.com/blog/past/2012/4/26/heroku_postgres_development_plan/
You cannot access the shared database using psql, pgadmin, etc.
Heroku offers you the choice of running on a shared or dedicated database package. The shared plan is suitable for development and staging applications. It runs Postgres 8.3. The dedicated plans are suitable for production scale applications. In addition, the dedicated databases offer a number of advantages, including direct access (via psql or any native postgres library), stored procedures, and Postgres 9 support.[source]