We have a production postgresql db available only in glue catalog. What's the best practices to ETL some tables in this database and load the data into another postgresql instance in the same aws account?
This production database is our transactional db and we don't need most of it's tables. We already have some glue ETL's creating tables in S3 (so accessible via Athena) but the goal here is to load into another postgresql instance.
Thanks
Related
I have my Old postgres database which is not a cloud based. And I want to migrate the data from the old database to new database which is in aws.
So can this be done via dblink or what is the other best practises to do this.
You can migrate DBs to AWS via AWS Database Migration Service. It's fully managed tool to help you move your data from on premises to AWS. You can read more about it here: https://aws.amazon.com/dms/?nc=sn&loc=1.
We have a Redshift cluster that needs one table from one of our RDS / postgres databases. I'm not quite sure the best way to export that data and bring it in, what the exact steps should be.
In piecing together various blogs and articles the consensus appears to be using pg_dump to copy the table to a csv file, then copying it to an S3 bucket, and from there use the Redshift COPY command to bring it in to a new table-- that's my high level understanding, but am not sure what the command line switches should be, or the actual details. Is anyone doing this currently and if so, is what I have above the 'recommended' way to do a one-off import into Redshift?
It appears that you want to:
Export from Amazon RDS PostgreSQL
Import into Amazon Redshift
From Exporting data from an RDS for PostgreSQL DB instance to Amazon S3 - Amazon Relational Database Service:
You can query data from an RDS for PostgreSQL DB instance and export it directly into files stored in an Amazon S3 bucket. To do this, you use the aws_s3 PostgreSQL extension that Amazon RDS provides.
This will save a CSV file into Amazon S3.
You can then use the Amazon Redshift COPY command to load this CSV file into an existing Redshift table.
You will need some way to orchestrate these operations, which would involve running a command against the RDS database, waiting for it to finish, then running a command in the Redshift database. This could be done via a Python script that connects to each database (eg via psycopg2) in turn and runs the command.
I have a Postgres database created on Heroku but looks like it's spun up on AWS based on the server name. Unfortunately, I can't tell if it's an Aurora or RDS instance.
I'm trying to create some data pipelines on Azure Data Factory to do some ETL work moving Blob files onto the Postgres Database but am having a tough time setting it up. The default Postgres option isn't working - looks like DataFactory tries to make a JDBC connection.
I'm stuck and am clueless as to how I should set up the connection.
I have created a AWS RDS Postgresql db instance in one VPC and I need to "move" it to another VPC.
I created a snapshot of the original db instance.
Re-created it in a new VPC (using terraform).
How can I retrieve one particular database from my initial db instance snapshot into the new DB Instance?
When you restore snapshot all databases are restored.
If you want to copy particular database, AWS service Database Migration Service can be used if postgres version is higher than 9.4.
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html
If postgres version is lower, then only manual export/import of database can be used. Another instance where export files will be stored is needed.
Is there a way to backup or restore a specific schema or table on a Cloud SQL server? Backing up the entire set of data, but then being able to restore only certain schemas or tables would be very helpful for multi-tenant systems.
Not via backup/restore. You can export a specific schema or table to Google Cloud Storage, but that's probably not what you're looking for.
The Cloud SQL use MySQL database with some limitations. You can check out the unsupported features and functions at the following link:
https://cloud.google.com/sql/faq#supportmysqlfeatures
With this in mind any backup/restore tool that are being used for MySQL should work for Google Cloud SQL as well. Using mysqldump for import/export of Cloud SQL is covered at this document:
https://cloud.google.com/sql/docs/import-export
You can use mysqldump to backup specific table or backup entire database and restore only specific tables using the solutions offered in these links:
Can I restore a single table from a full mysql mysqldump file?
How to take backup of a single table in a MySQL database?
You can restore the backup to a different instance and then replace the data on the original instance with the data from the backup instance.