What's the best approach for backing up a Postgres on Kubernetes?
My first guess would have been to create a master-slave architecture with enabling replication. Doing an initial pg_basebackup and then fetching the WAL-logs. Once in a month I'd have scheduled another pg_basebackup with a cron, however containerized environemnts don't like cron daemons (no systemd available). How do you schedule base backups?
The best approach is to use the Kubernetes Cronjob resource:
You can use a CronJob to run Jobs on a time-based schedule. These
automated jobs run like Cron tasks on a Linux or UNIX system.
Cron jobs are useful for creating periodic and recurring tasks, like
running backups or sending emails.
You basically need to create a custom Linux image to run in your container jobs. This image will need a Postgres Client (so you can connect to your database with psql, pg_dump or pg_basebackup); and the credentials that can be configured as a secret.
You may want to upload the backup to external storage, so you can install and use awscli for AWS S3, gsutil for Google Cloud Storage, etc...
Here is some references:
Creating a Kubernetes Cron Job to backup Postgres DB
Simple backup of postgres database in kubernetes
Back up databases using Kubernetes CronJobs
Related
How to connect A Cron job to Redshift host and S3 bucket?
Expecting Cron job will connect to Redshift hostname and S3 bucket
This depends on what your cronjob is running. Start by looking into the awscli (for shell level commands) - https://docs.aws.amazon.com/cli/index.html and boto3 (SDK for python connectivity) - https://github.com/boto/boto3. There are other SDKs available for the software you are running but hopefully these resources will give you a start. As for Redshift you can connect using the above API calls or by installing/using a jdbc client.
More description on what you are trying to achieve is needed to give you more specific advice.
I have setup a postgres database inside the Kubernetes cluster and now I would like to backup the database and don't know how it is possible.
Can anyone help me to get it done ?
Thanks
Sure you can backup your database. You can setup a CronJob to periodically run pg_dump and upload the dumped data into a cloud bucket. Check this blog post for more details.
However, I recommend you to use a Kubernetes native disaster recovery tool like Velero, Stash, Portworx PX-Backup, etc.
If you use an operator to manage your database such as zalando/postgres-operator, CrunchyData, KubeDB, etc. You can use their native database backup functionality.
Disclosure: I am one of the developer of Stash tool.
I'm trying to schedule a function to periodically run and delete records from my google cloudsql (Postgresql) database. I want this to run a couple of times a day and will run under 10 minutes. What options do I have to schedule this function?
Thanks
Ravi
Your best option will be to use Cloud Scheluder to schedule a job that publishes to a Pub/Sub topic. Then, have a Cloud Function subscribed to this topic so it get's triggered by the message sent.
You can configure this job to run as a Daily routine x times a day.
Try pgAgent
pgAgent is a job scheduling agent for Postgres databases, capable of running multi-step batch or shell scripts and SQL tasks on complex schedules.
pgAgent is distributed independently of pgAdmin. You can download pgAgent from the download area of the pgAdmin website.
I wanted to automate taking snapshots of the volume attached to an EC2 instance running the primary node of our production MongoDB replicaSet. While trying to gauge the pitfalls and best practices over Google, I came across the fact that data inconsistency and corruption are very much possible while creating a snapshot but not of journaling is enabled, which it is in our case.
So my question is - is it safe to go ahead and execute aws ec2 create-snapshot --volume-id <volume-id> to get clean backups of my data?
Moreover, I plan on running the same command via a cron job that runs once every week. Is that a good enough plan to have scheduled backups?
For MongoDB on an EC2 instance I do the following:
mongodump to a backup directory on the EBS volume
zip the mongodump output directory
copy the zip file to an S3 bucket (with encryption and versioning enabled)
initiate a snapshot of the EBS volume
I write a script to perform the above tasks, and schedule it to run daily via cron. Now you will have a backup of the database on the EC2 instance, in the EBS snapshot, and on S3. You could go one step further by enabling cross region replication on the S3 bucket.
This setup provides multiple levels of backup. You should now be able to recover your database in the event of an EC2 failure, an EBS failure, an AWS Availability Zone failure or even a complete AWS Region failure.
I would recommend reading the official MongoDB documentation on EC2 deployments:
https://docs.mongodb.org/ecosystem/platforms/amazon-ec2/
https://docs.mongodb.org/ecosystem/tutorial/backup-and-restore-mongodb-on-amazon-ec2/
I would like to deploy an application using docker and would like to use a postgresql container to hold my data.
However I am worried about losing data, so I need back-ups.
I know I could run a cron job on the host to dump the data out from the container, however this approach is not containerized and when I deploy to a new location, I have to remember to add the cronjob.
What is a good , preferably containerized, approach to implement rotating data backups from a postgresql docker container?
Why not deploy a second container that is linked to the PostgreSQL one that does the backups?
It can contain a crontab within, together with instructions on how to upload the backup to Amazon S3, or some other secure storage in the cloud that will not fail even in case of an atomic war :)
Here's some basic information on linking containers: https://docs.docker.com/userguide/dockerlinks/
You can also use Docker Compose in order to deploy a fleet of containers (at least 2, in your case). If your "backup container" uploads stuff to the cloud, make sure you don't put your secrets (such as AWS keys) into the image at build time. Put them into the container at run-time. Here's more information on managing secrets using Docker.