I have an OLTP DB in Oracle and a downstream OLAP System in PostgreSQL on-premises. The data from Oracle is pumped into PostgreSQL using Oracle_FDW.
I am exploring the possibility of moving the PostgreSQL to AWS, but none of the RDS have Oracle_fdw capability. One way out is to install PG on an EC2 instance but that would leave some of the features like read-replica provided natively by AWS. Is there a better workaround?
Also is there a way to fetch the data in Oracle RDS from Postgres RDS in AWS?
With PostgreSQL on Amazon RDS your choice of extensions is limited to the extensions they explicitly support. As far as I'm aware there's no way around this limitation.
Like you mentioned, the general option in this case would be to host PostgreSQL yourself on EC2 instead of RDS. You lose automatic backup/replication/management features, but you get the power and flexibility you need. This will certainly work but will require some leg work to replace what you're losing by not using RDS.
The only alternative to this I can think of is that you may be able to host a different (otherwise empty) PostgreSQL server with the oracle-fdw extension installed and use the postgres-fdw extension (which is supported by RDS) to proxy requests from your RDS hosted database, through your proxy PostgreSQL database, to your Oracle database and back. If the amount of data you're retrieving is substantial, or if the number of queries per minute is high this is probably a terrible idea. But it might be worth testing to see if it works for your use case.
I did a quick search around and I haven't been able to find any references to anyone actually layering foreign data wrappers like this but I also couldn't find anything in the manual or online saying it wasn't supported either. In theory it should work, but if you do try it make sure you thoroughly test it prior to using it to do anything important.
Oracle_FDW is now supported in recent versions - https://aws.amazon.com/about-aws/whats-new/2021/07/amazon-rds-for-postgresql-supports-oracle-fdw-extension-for-accessing-data-in-oracle-databases/
Related
Since Citus is not going to be available as a Managed Service in AWS, I am trying move the database to RDS (not the whole history but only the transactional portion as an OLTP). The migration from Citus is not clear because the data does not reside in a single node. I want to check the options we might have to move data from Citus to RDS.
Amazon DMS: This option is good for the supported databases (PostgreSQL) but we do not know what behavior this will have in Citus from the distributed nature of the engine. Has someone migrated the data to S3, to another DB or something in these lines?
I saw this paper from AWS https://d1.awsstatic.com/whitepapers/aws-cloud-data-ingestion-patterns-practices.pdf?did=wp_card&trk=wp_card on how to ingest data from different sources and DMS seems like a good option but I do not know the internals of Citus that well to tell if we will get all the data and gather the CDC correctly.
A Custom migration: Via a support ticket, we can access the S3 buckets that Citus uses for Disaster recovery where the WAL logs are available and we could use something like WAL-G to take those logs and replicate them in a Postgres instance. The issue here is that this is a very custom migration and the development time might be too high.
Is there any other option to move data from Citus to RDS or Aurora in AWS, what looks like a good path to make the database migration? All the documents refer to move data the other way around, from Aurora or RDS to Citus.
Sumedh from Citus Cloud here. Please go ahead and open a support ticket with us to further investigate solutions. We can evaluate if using DMS is a viable approach for your use-case.
I am seeing some slow performance on a couple of my queries that run against my db2 on cloud instance. When I had a local db2, I would try these tools to see if I could improve performance. Now, with db2 on cloud, I believe I can run them using admin_cmd, however, if they are already being run automatically on my db objects, there is no point, but I am not sure how to tell.
Yes, Db2 on Cloud does auto reorgs and runstats automatic. We do recommend running them manually, if you are running a lot of data loads to better the performance.
As you stated, Db2 on Cloud is a managed (as a Service) database offering. But this is for the general part, not for application-specific stuff. Backup / restore can be done without any application insights, but creating indexes, running runstats or performing reorgs is application-specific.
Runstats can be invoked using admin_cmd. The same is true for running reorg on tables and indexes.
We are managing to use GCP CloudSQL for our PostgreSQL database,
at this moment one of our applications uses large objects and i was wondering how to perform a vacuumlo operation on such platforms (question might be valid for AWS RDS or any other cloud postgresql provider).
Does making custom queries/procedures to perform the same task is the only solution?
Since vacuumlo is a client tool, it should work just fine with hosted databases.
We have a quite large (at least to us) database that has over 20.000 tables, which is running in an AWS EC2 Instance, but due to several reasons, we'd like to move it into an AWS RDS instance. We've tried a few different approaches for migrating into RDS but as per the data volume involved (2TB) and RDS' restrictions (users and permissions) and compatibility issues, we haven't been able to accomplish it.
Given the above facts, I was wondering if PostgreSQL actually supports something like mapping a remote schema into a database, if that would be possible we could try to tinker individual per schema migrations and, not the whole database at once, which would actually make the process less painful.
I've read about the IMPORT FOREIGN SCHEMA feature which seems to be supported from version 9.5 and, that seems to do the trick, but is there something like that for 9.4.9?
You might want to look at the AWS Database Migration tool, and the associated Schema Migration tool.
This can move data from an existing database into RDS, and convert - or at least report on what would need to be changed - the schema and associated objects.
You can run this in AWS, point it at your existing EC2-based database as the source, and use a new RDS instance as the destination.
We are using the same PostgreSQL 9.3 Server both for production and development.
So we would like to get the copy of the existing production database for the development purposes. To be precise, all the INSERT/UPDATE/DELETE events that come to production should be also placed into its copy. Reverse replication is not needed. How can we do that?
PS: Please take into account that full master-slave replication is not suitable. We have no opportunity to deploy one more PostgreSQL server at the moment.
UPD: pg_dump/pg_restore is not the case too, because this will crash all the updates done by developers in their database.
Sounds like you are looking for a replication system like Slony or Bucardo
Slony uses triggers to replicate the data so that should work without much hassle here. And Bucardo uses NOTIFY to do pretty much the same.
For just a local setup I would recommend Slony but if you would like to offer the developers a local database (i.e. local machine) I would recommend Bucardo instead as it offers asynchronous replication.