I am testing replication from Citus(Cloud Hosted) to my RDS Aurora Postgres with ref https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Replication.Logical.html#AuroraPostgreSQL.Replication.Logical.Configure
Everything ran successful but on subscribers, no data is coming so what may be wrong? how do I troubleshoot
SELECT count(*) FROM LogicalReplicationTest;
Note: on Citus DB I can See Publication:
But in my RDS can't see Subscriptions in the list in spite of CREATE SUBSCRIPTION testsub CONNECTION was successful:
I observed that when I run SELECT count(*) FROM LogicalReplicationTest; there is no data but only column name showing Lock symbol as shown in this screenshot! any idea what this lock symbol means apart from the Read-only Column? why it's not listing data inside that table which is in publisher DB Is there any permission I have to delegate to the table when creating a publication?
Related
Attempting to migrate from AWS Aurora PostgreSQL 13.4 to Google Cloud SQL PostgreSQL 13.
Migration job gives this error:
finished setup replication with errors: failed to drop database "postgres": generic::unknown: retry budget exhausted (10 attempts): pq: database "postgres" is being accessed by other users
The user the DMS is using only has SELECT permissions on the source database(Aurora)
I'm very confused as to why it is trying to drop the "postgres" database at all. Not sure if it is trying to drop the database in the source or destination. Not sure what I'm missing.
I've installed necessary extensions in the destination DB(pg_cron). No difference.
User in source database has SELECT on all tables/schemas outlined in the docs(including pglogical schema)
I've tried various PostgreSQL versions in the destination cluster( 13.x, 14.x). No difference.
The "Test connection" tool when creating the migration job, shows no errors. (There is a warning about a few tables not having Primary keys, but that's it.)
I would like to find the date for the most latest changes on a postgres table . I see understand that we can get this info from accessing table data files. But my postgres database is hosted as AWS RDS aurora (Postgres database engine) . I guess we cant access the internals of RDS , how do i get this info , any help would be appreciated.
That is futile, even if you could get the files from Amazon, which you cannot.
The data files are also modified if no DML operation is going on: both autovacuum and HOT chain pruning would do that.
Trying to drop a db from AWS RDS (postgres) is blocked only for superuser (which is reserved to AWS only).
trying to run this command on my instance:
DROP DATABASE "dbname"
results in error:
psycopg2.InternalError: DROP DATABASE cannot run inside a transaction block
I saw this issue in AWS forums, which seems to have been active by a lot of people, but no AWS representative giving a valid solution.
How can I drop my db without taking down the whole instance and raising it again?
Worked for me:
Connect to postgres DB, disconnect from DB you want to DROP!
select pg_terminate_backend(pid) from pg_stat_activity where datname='yourdb';
drop database 'yourdb';
Step 2 and 3 execute fast in order to prevent user rdsadmin reconnect
Try using ISOLATION_LEVEL_AUTOCOMMIT, a psycopg2 extensions:
No transaction is started when command are issued and no commit() or
rollback() is required.
The connection must be in autocommit mode. You can way can set it using psycopg2 is through the autocommit attribute:
import psycopg2
con = psycopg2.connect(...)
con.autocommit = True
cur = con.cursor()
cur.execute('DROP DATABASE db;')
I have configured SQL Replication between two our two IBM DB2 databases ( database A and database B). The replication is working fine and all the data changes in one database are being replicated to the other database.
The issue I am facing is that if I add a new table in database A, then I have to manually add this table in Capture Control Server and Apply Control Server.
Even If I add a new column in a table for whom SQL Replication is already configured, I have to register that column in Capture Control Server and Apply Control Server.
Is there any way of replicating the schema changes automatically from one database to the other so that we don't have to manually add the new tables and columns in Capture Control Server and Apply Control Server for SQL Replication?
Regards,
Babar Hussain
I have two databases on Amazon RDS, both Postgres. Database 1 and 2
I need to restore an instance from a snapshot of Database 1 for my Staging environment. (Database 2 is my current Staging DB).
However, I want the data from a few of the tables in Database 2 to overwrite the tables in the newly restored snapshot. What is the best way to do this?
When restoring RDS from a Snapshot, a new database instance is created. If you only wish to copy a portion of the snapshot:
Restore the snapshot to a new (temporary) database
Connect to the new database and dump the desired tables using pg_dump
Connect to your staging server and restore the tables using pg_restore (most probably deleting any matching existing tables first)
Delete the temporary database
pg_dump actually outputs SQL commands that are then used to recreate tables and restore data. Look at the content of a dump to understand how the restore process actually works.
I hope this still works for someone else.
With my team we faced a similar issue. We also had 2 Postgres databases and we also just needed to backup some tables from db1 to db2.
What we did is to use a lambda function using Python (from AWS lambda ofc) that connected to both databases and validates if db1.table1 has the same data as db2.table1, if not, then the lambda function should write the missing data from db1.table1 into db2.table1. The approach of using lambda was because we wanted to automate the process due to the main db (let's say db1) is constantly being updated. In addition, it allowed us to only backup our desired tables (let's say 3 tables out of 10), instead of backing up the whole database.
Note: Maybe you want to do these writes using temporary tables to avoid issues with any constraints you have in your tables.