I have a database in PostgreSQL. Now we need to import whole database into DynamoDB. Data Migration service need to be used for this purpose or any other service can be used. Please explain in detail.
What is the strategy to be followed? I have studied many blogs but I couldn't get any proper way to migrate the whole database from PostgreSQL to DynamoDB. Only through DMS Service is it possible to migrate or any other service can be used to migrate or any script should be run to migrate the PostgreSQL to to DynamoDB.
The AWS Database Migration Service offers everything you would need to migrate data from any relational DB into AWS - whether the target is DynamoDB (NoSQL offering from AWS) or any of the DB flavors from AWS RDS service.
You can find multiple migration playbooks and step-by-step guides on the Resources page of this AWS service
Related
Since Citus is not going to be available as a Managed Service in AWS, I am trying move the database to RDS (not the whole history but only the transactional portion as an OLTP). The migration from Citus is not clear because the data does not reside in a single node. I want to check the options we might have to move data from Citus to RDS.
Amazon DMS: This option is good for the supported databases (PostgreSQL) but we do not know what behavior this will have in Citus from the distributed nature of the engine. Has someone migrated the data to S3, to another DB or something in these lines?
I saw this paper from AWS https://d1.awsstatic.com/whitepapers/aws-cloud-data-ingestion-patterns-practices.pdf?did=wp_card&trk=wp_card on how to ingest data from different sources and DMS seems like a good option but I do not know the internals of Citus that well to tell if we will get all the data and gather the CDC correctly.
A Custom migration: Via a support ticket, we can access the S3 buckets that Citus uses for Disaster recovery where the WAL logs are available and we could use something like WAL-G to take those logs and replicate them in a Postgres instance. The issue here is that this is a very custom migration and the development time might be too high.
Is there any other option to move data from Citus to RDS or Aurora in AWS, what looks like a good path to make the database migration? All the documents refer to move data the other way around, from Aurora or RDS to Citus.
Sumedh from Citus Cloud here. Please go ahead and open a support ticket with us to further investigate solutions. We can evaluate if using DMS is a viable approach for your use-case.
We are managing to use GCP CloudSQL for our PostgreSQL database,
at this moment one of our applications uses large objects and i was wondering how to perform a vacuumlo operation on such platforms (question might be valid for AWS RDS or any other cloud postgresql provider).
Does making custom queries/procedures to perform the same task is the only solution?
Since vacuumlo is a client tool, it should work just fine with hosted databases.
We are in the process of refactoring our databases. As part of that we have modified split our data which used to exist in single Postgres table into a new Postgres table schema and DynamoDB tables. What is the best way to migrate the data from the old schema into this new hybrid schema? We are thinking of writing a Java program to do it. But wanted to check if we can leverage some AWS offering to do this in an efficient way.
Check out the AWS Database Migration Service.
AWS Database Migration Service helps you migrate databases to AWS quickly and securely. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database. The AWS Database Migration Service can migrate your data to and from most widely used commercial and open-source databases.
See https://aws.amazon.com/dms/
I want to migrate AWS PostgreSQL to google cloud SQL. I can perform such by some basic strategy such as extract the AWS data, Create Database in GCP and Restore the extracted data in GCP. But I was wondering is there any more sophisticated way to so such as using terraform or similar.
Yes. See https://cloud.google.com/solutions/migrating-postgresql-to-gcp/
For migrating MySQL there are more options available, however at the time of the writing, these only apply to MySQL:
https://cloud.google.com/sql/docs/mysql/migrate-data
https://cloud.google.com/sql/docs/mysql/replication/replication-from-external
Actually I am not very familiar with Dynamodb and I would like to launch a NoSQL database with local Dynamodb (downloadable version) but not hosted on amazon AWS. I would appreciate it if someone could let me know is it possible to make such a clustered or does downloadable version of Dynamodb support to be cluster locally ?
You can very easily run DynamoDB locally, but it only supports running a single instance—not a cluster. It's intended to be used for local testing/debugging.
DynamoDB is provided as a hosted service. Does not exist a DynamoDB code that you can download and install to use as a host or service provider.
As part of SDK for a lot of languages, AWS Team developed some wrappers that permits you to execute local versions of DynamoDB to test your particular code. These wrappers respect the DynamoDB API contract. In that case you can code to the DynamoDB interface and get the responses like it were hosted in AWS environment. But you can't host any database or even serve data as a service using this solutions.