I could not find any way to limit replication slot in Cloud SQL postgres flags (max_slot_wal_keep_size) ? Is there any way I can limit it ?
Related
Overall I'm trying to create a Datastream Connection to a Postgres database in Cloud SQL.
As I'm trying to configure it all through Terraform, I'm stuck on how I should create a Replication Slot. This guide explains how to do it through the Postgres Client and running SQL commands, but I thought there might be a way to do it in the Terraform configuration directly.
Example SQL that I would like to replicate in Terraform:
ALTER USER [CURRENT_USER] WITH REPLICATION;
CREATE PUBLICATION [PUBLICATION_NAME] FOR ALL TABLES;
SELECT PG_CREATE_LOGICAL_REPLICATION_SLOT('[REPLICATION_SLOT_NAME]', 'pgoutput');
If not, does anyone know how to run the Postgres SQL commands against the Cloud SQL database through Terraform?
I have setup the Datastream and Postgres connection for all other parts. I'm expecting that there is a Terraform setting I'm missing or a way to run Postgres commands against the Google Cloud SQL Postgres database.
Unfortunately, there is no terraform resource for specifying a replication slot on a google_sql_database_instance.
We are new to Google Cloud SQL and have been trying to integrate pgbouncer with Google Cloud SQl Postgres and authenticate database users with SECURITY DEFINER function (which queries pg_shadow)
Our Configuration:
Server -> Pgbouncer + Cloud sql proxy (side car) -> Cloud SQL Postgres
Problem:
But as cloud sql postgres actually does not allow to read pg_shadow from a privileged user (i.e postgres user is not a superuser). This makes it impossible to setup pgbouncer with SECURITY DEFINER function.
Cloud SQL doesn't provide customers to use superuser (cloudsqladmin)
We've read through many articles (mostly cloud-proxy issues) where they have suggested to use pgbouncer but have not elaborated on the above problem.
Options not applicable:
Application level pooling (not feasible right now for us)
Authenticating using auth_file eg. users_list.txt (not recomended, needs manual management of database users)
What we are looking for:
We intend to run a single instance of cloudsql-proxy and pgbouncer which proxies and pools connections to cloudsql postgres database.
We would appreciate your help guys!
I can't find it in Supported Flags.
Are there no way to increase it?
I wanted to know if there are any Postgres config parameter settings available for CTEs in specific? The reason I am asking this is because we migrated a DB from AWS RDS Postgres to Aurora Postgres with the same major version (Postgres v12). Both the DB's have the same structure/indices/etc., however when we run the same CTE on Aurora, it runs very slow and shows very high IOPS v/s when we run it on RDS, it runs fast and no IOPS issue there. I am assuming that there might be some config setting which is different in RDS v/s Aurora which is causing the issue.
Any suggestions/recommendations?
Thanks
I have a requirement of replicate data from AWS RDS Postgres(9.6) Database to On-Premise Postgres(9.5) Database. I have found stuff about replication from On-premise to On-premise. But How can we implement it for AWS RDS to On-premise?
I do this using Bucardo.
Check-out this: https://bucardo.org/Bucardo/
With Bucardo you can replicate RDS postgres instance to a slave postgres present somewhere, only configuring slave, so without the needs to configure RDS stuff.
Also you can do this with zero downtime.
Anyway I am not sure this will work using different versions of Postgresql. You should use same version if possible. I tested it with 9.4.x and it is working.
UPDATE
I can confirm that this is working also using different version of Postgres, for example I was able to replicate with these versions:
AWS RDS postgresql 9.4.x
On-premise postgresql 9.6.x
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.FeatureSupport.LogicalReplication
Beginning with PostgreSQL version 9.4, PostgreSQL supports the
streaming of WAL changes using logical replication slots. Amazon RDS
supports logical replication for a PostgreSQL DB instance version
9.4.9 and higher and 9.5.4 and higher. Using logical replication, you can set up logical replication slots on your instance and stream
database changes through these slots to a client like pg_recvlogical.
Logical slots are created at the database level and support
replication connections to a single database.
mind possible problems eg https://dba.stackexchange.com/questions/173267/aws-rds-postgres-logical-replication