I am using prometheus-operator in Kubernetes and am using EBS volumes as backing store through VolumeClaimTemplates. I would like to instead use Amazon RDS PostgreSQL as a backing store so that I wouldn't have to worry about running out of storage and monitoring storage etc.
I came across remote storage adapters for InfluxDB, Graphite and OpenstDB here but they don't have an adapter for PostgreSQL or MySQL.
Does anyone have any experience making prometheus backup samples to PostgreSQL/MySQL in production environments?
I came across prometheus-postgresql-adapter here but am not sure how it will work with Amazon RDS. If you have any pointers to make it work with RDS, that too will be much appreciated.
Short answer: you can't. Amazon RDS doesn't support that yet. The GitHub issue here says it all: https://github.com/timescale/prometheus-postgresql-adapter/issues/10.
Currently, if you want to use the prometheus-postgresql-adapter, you would need to run the TimescaleDB.
Related
Since Citus is not going to be available as a Managed Service in AWS, I am trying move the database to RDS (not the whole history but only the transactional portion as an OLTP). The migration from Citus is not clear because the data does not reside in a single node. I want to check the options we might have to move data from Citus to RDS.
Amazon DMS: This option is good for the supported databases (PostgreSQL) but we do not know what behavior this will have in Citus from the distributed nature of the engine. Has someone migrated the data to S3, to another DB or something in these lines?
I saw this paper from AWS https://d1.awsstatic.com/whitepapers/aws-cloud-data-ingestion-patterns-practices.pdf?did=wp_card&trk=wp_card on how to ingest data from different sources and DMS seems like a good option but I do not know the internals of Citus that well to tell if we will get all the data and gather the CDC correctly.
A Custom migration: Via a support ticket, we can access the S3 buckets that Citus uses for Disaster recovery where the WAL logs are available and we could use something like WAL-G to take those logs and replicate them in a Postgres instance. The issue here is that this is a very custom migration and the development time might be too high.
Is there any other option to move data from Citus to RDS or Aurora in AWS, what looks like a good path to make the database migration? All the documents refer to move data the other way around, from Aurora or RDS to Citus.
Sumedh from Citus Cloud here. Please go ahead and open a support ticket with us to further investigate solutions. We can evaluate if using DMS is a viable approach for your use-case.
I have setup a postgres database inside the Kubernetes cluster and now I would like to backup the database and don't know how it is possible.
Can anyone help me to get it done ?
Thanks
Sure you can backup your database. You can setup a CronJob to periodically run pg_dump and upload the dumped data into a cloud bucket. Check this blog post for more details.
However, I recommend you to use a Kubernetes native disaster recovery tool like Velero, Stash, Portworx PX-Backup, etc.
If you use an operator to manage your database such as zalando/postgres-operator, CrunchyData, KubeDB, etc. You can use their native database backup functionality.
Disclosure: I am one of the developer of Stash tool.
I have ECS at Alibaba cloud, I want to add PostgreSQL, but I can't find any Tutorials on the Internet
how to add PostgreSQL to ECS Alibaba Cloud
There are several ways on using PostgreSQL on Alibaba Cloud:
ApsaraDB RDS for PostgreSQL, which is PaaS solution for PostgreSQL on Alibaba Cloud, so you don't have to worry about installing and configuring PostgreSQL from scratch. It comes with a lot of additional features such as, high availability, disaster recovery, backup, etc. You can find their documentation on creating your PostgreSQL instance.
ApsaraDB for PolarDB, also a PaaS, which is Alibaba Cloud's homegrown RDB fully compatible with MySQL and PostgreSQL. It can support higher storage capacity, nodes clustering, and it's designed for high performance. Check out their documentation on how to create a PostgreSQL cluster.
Self-managed PostgreSQL on ECS - of cause you can still run PostgreSQL on your own ECS. There're plenty of resources on how to install and configure your own PostgreSQL. Check out the DigitalOcean's tutorial on installing PostgreSQL on Ubuntu 20.04.
You have two ways to do it.
you would just take the ECS as a Linux server. you build the PostgreSQL by yourself. it may request higher skills.
you would use the PaaS service, polardb(PostgreSQL) ,you do not need build it step by step ,just use it in 2-3 mins.
the polardb links as below:
https://www.alibabacloud.com/product/polardb?spm=a3c0i.20899616.6791778070.dbannerarelationaldb1.53fd2accf4slGC
I need to set up real time sync process between a on premise postgresql instance with cloud postgresql instance. Please let me know what are all the options available through which i can achieve it.
Do i have to use any specific tool or it can be managed through replication .
Please advice
Use PgPool
http://www.pgpool.net/mediawiki/index.php/Main_Page
from their web page:
pgpool-II can manage multiple PostgreSQL servers. Using the replication function enables creating a realtime backup on 2 or more physical disks, so that the service can continue without stopping servers in case of a disk failure.
Can we replicate data from one RDS server to another? Or can we set master slave relationship between two RDS servers?
Should we replicate data from non RDS instance to RDS instance?
RDS can replicate from external mysql and also be a master of an external slave. It depends on your usecase if you "should" do it.
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Procedural.Importing.External.Repl.html
While i guess you could setup replication between two RDS instances yourself I don't see why you should since starting a RDS read replica is just a few clicks in AWS console or an api call.
It can be possible to replicate data from RDS to RDS. It is also possible to replicate data from RDS to some other MySQL server.
Steps:
You can go creating your ec2 server and install MySQL.
Change configuration to replicate data.
That will require additional work to manage ec2 instance in case if your data is increasing and crossing the server limits
Then you have to do all the manual work again to replicate data as we can't increase storage in ec2 server.
RDS provides an easy mechanism to create Read replica via a few clicks. (Note: replica is quite a costlier option.)
But going with that you will save manual work one person salary who will be managing the database and doing these setups regularly.
If you are using postgresql database on RDS then you can use bucardo for asynchronous replication. You need to create a EC2 or use can use local system also but it will not be fast enough.
Use the following tutorial if you want to use bucardo.
https://www.installvirtual.com/how-to-install-bucardo-for-postgres-replication/
I think you can using snapshot to clone another rds database