Is automatic Backup performed regardless of the Cloud SQL instance State?
we want to stop the instance dayly so we can keep costs to the minimum
Automatic backups are not taken if the instance is off.
Related
I'm using AWS RDS and have a need to replicate "database_a" in an RDS instance to "database_a" in a different RDS instance. The replication only needs to be once every 24 hours.
I'm currently solving this with pg_dump and pg_restore but am wondering if there is a better (ie faster/more efficient) way I can go about things.
Using wal-e/g and RDS, is it at all possible for my use case to simply push the latest changes from the last say 24 hours? The 2 RDS cannot speak to each other so all connection would be by S3. I'm not clear what the docs mean by 'When uploading backups to S3, the user should pass in the path containing the backup started by Postgres:' - does this mean i can create a pg backup to my EC2 and then point wal-g at this backup?
Finally, is it at all possible to just use wal-e/g for complete backups (ie non incremental) just as i am doing now with pg_dump/pg_restore and in doing so would I see a speed improvement by switching?
Thanks in advance,
In a word, yes.
On a system using dump/restore, you're consuming a lot more CPU and network resources (therefore costs) which you could reduce notably by using the WALs for incremental backups, and only doing an image perhaps once a week. This is especially true if your database is mostly data that doesn't change. It might be incorrect if your database is not growing but is made of records that are updated many times per 24 hours (e.g. stock prices).
Once you are publishing WALs to S3 frequently, then you'll have a far more up to date backup than nightly backups.
When publishing WALs you can recover to any point in time
WAL-E and WAL-G both have built in encryption
There is also differential backup support, but not something I've played with
I am using the PostreSQL pg_try_advisory_lock function to coordinate a cluster of micro-services, sharing the same database, to ensure that only one instance performs an ad hoc task.
Does this function contend with any internal locks PostresSQL may be using. I am concerned that using a database wide locking mechanism such as this may impact the system resources used within the database engine.
PostgreSQL does not use advisory locks internally, so there is no danger of blocking anything in the system by using them.
I have restored a snapshot of a PostgreSQL instance as a new instance with exactly the same configuration as the original instance. However, running queries takes much longer on the new instance. A query that takes less than 0.5 ms to execute on the original instance, takes over 1.2 ms on the new one. A nightly Python script that runs in 20 minutes on the old instance is now taking over an hour with the new one. This has been going on for several days now.
I run VACUUM(ANALYZE, DISABLE_PAGE_SKIPPING); after our nightly snapshot restores for our Staging DB to get everything running smoothly again
Unfortunately this is normal but it should go away after a while.
Snapshots are stored on S3, and when you create a new EBS volume with one, the volume only pulls in blocks of data as they are requested, causing degraded performance until the whole volume is initialized. See these AWS docs for confirmation.
Those docs suggest using dd to force all the data to load, but on RDS you have no way to do that. You might want to try SELECTing everything you can instead, although that will still miss some things (like indexes).
I am trying to restore large *.sql dump (~4 GB) to one of my DB on RDS. Last time tried to restore it using Workbench and it took about 24+ hours until the whole process is complete.
I wonder if there is a quicker way to do this. Please help and share your thoughts
EDIT: i have my sql dump on my local computer by the way.
At the moment i have 2 options in mind:
Follow this link
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Procedural.Importing.NonRDSRepl.html (with low confidence)
dump the DB and compress it, and then upload the compressed dump to one of my EC2 instance, and then SSH to my EC2 instance and do
mysql> source backup.sql;
I prefer the second approach (simply because i have more confidence in that), as well as it would fastened the upload time since the entire dump is first uploaded, un-compressed and finally restored.
My suggestion is to take table-wise backup of large tables and restore them by disabling indexes. Which inserts records quickly (at more than double speed) and simply enable the indexes after restore completes.
Before restore command:
ALTER TABLE `table_name` DISABLE KEYS;
After restore completes:
ALTER TABLE `table_name` ENABLE KEYS;
Also add these extra commands at the top of the file to avoid a great deal of disk access:
SET FOREIGN_KEY_CHECKS = 0;
SET UNIQUE_CHECKS = 0;
SET AUTOCOMMIT = 0;
And add these at the end:
SET UNIQUE_CHECKS = 1;
SET FOREIGN_KEY_CHECKS = 1;
COMMIT;
I hope this will work, thank you.
Your intuition about using an EC2 intermediary is correct, but I really think the biggest thing that will benefit you is actually being inside AWS. I occasionally perform a similar procedure to move the contents of a 6GB database from one RDS DB instance to another.
My configuration:
Source DB instance and Target DB instance are in the same region and availability zones. Both are db.m3.large.
EC2 instance is in the same region and availability zone as the DB instances.
The EC2 instance is compute optimized. (I use c3.xlarge but would recommend the c4 family if I were to start again from scratch).
From here, I just use the EC2 instance to perform a very simple mysqldump from the source instance, and then a very simple mysqlrestore to the target instance.
Being in the same region and availability zones really make a big difference, because it reduces network latency during the procedure. The data transfer is also free or near-free in this situation. The instance class you choose for both your EC2 and RDS instances is also important -- if you want this to be a fast operation, you want to be able to maximize I/O.
External factors like network latency and CPU can (and in my experience, have) bottleneck the operation if you don't provide enough resources. In my example with the db.m3.large instance, the MySQLDump takes less than 5 minutes and the MySQLRestore takes about 15 minutes. I've attempted to restore to a db.m3.medium RDS instance before and the restore time took a little over an hour because the CPU bottlenecked -- it wasn't able to keep up with the EC2 instance. Back when I was restoring from my local machine, being outside of their network caused whole process to take over 4 hours.
This EC2 intermediary shouldn't need to be available 24/7. Turn it off or terminate it when you're done with it. I only have to pay for an hour of c3.xlarge time. Also remember that you can scale your RDS instance class up/down temporarily to increase resources available during your dump. Try to get your local machine out of the equation, if you can.
If you're interested in this strategy, AWS themselves has provided some documentation on the matter.
Command is always fast than Workbench.
Try to use this command to restore your database.
To Restore :
mysql -u root -p YOUR_DB_NAME < D:\your\file\location\dump.sql
To Take Dump :
mysqldump -u root -p YOUR_DB_NAME > D:\your\file\location\dump.sql
I'm considering deploying an app to Heroku along with a Postres Standard database plan. I'm keen on ensuring data integrity and ensuring in no case that my customer's data can be lost if the database becomes corrupted or some other similar issue. I also want to ensure a smooth recovery process in tis even. So I have the following questions:
First, I'm assuming with Continuos there's a still a possibility
that a database can become corrupted. Is this true?
What's provides more
integrity, protection, and ease of recovery if a database becomes
corrupted: Standard DB / with Continuos Protection or Standard DB
with a Follower DB.
If by chance the DB
becomes corrupted, or an database integrity issues arise, how will
Heroku remediate (given the database is a "managed" service). Is it
automated or I have to work with Support manually to remediate?
I would love to hear your thoughts on this. My experience in the past has been with MySQL but not Postgres, which I hear great things about.
Thanks
Caveat: I have some experience with Postgresql, but I don't have any experience with Heroku as such.
What Heroku calls 'Continuous Protection' and 'follower' databases are implemented using Postgresql's Continuous Archiving and streaming replication functionality. They have provided a range of administrative tools and infrastructure around these functions to make them easier to use.
Both of these functions make use of the fact that Postgresql writes all updates that it is making to databases in a Write-Ahead Log (WAL).
With Continuous Archiving, one takes a complete copy of all of the underlying files in the database - this is referred to as the base backup. One also collects all WAL files produced by the database, both during and after production of the base backup. Note that you do not need to stop the database in order to make the base backup - it is a fairly unobtrusive process.
If the worst happens, and it is necessary to recover the database from the backup, you just restore the base dump, configure the database so it knows where to find the archived WAL files, and start it up. It will then replay the WAL files in sequence until it is fully up to date.
Note that you can also stop the replay early. This can be extremely useful, as you will see in my answer to your first question:
First, I'm assuming with Continuos there's a still a possibility that
a database can become corrupted. Is this true?
Yes, of course. Database corruption can happen for a number of reasons: hardware failure, a software fault in the database, a fault in your application, or even operator error.
One of the benefits of continuous archiving, though, is that you can replay the WAL files up to a particular point in time, so you can effectively rewind back to the point immediately before the database became corrupted.
As mentioned above, a Follower DB uses Postgresql's 'Streaming Replication' function. With this function, you restore your base backup onto another server, configure it to connect to the master database and fetch WAL files in real time as they are produced. The follower then keeps up to date with any changes made on the master.
Whats provides more integrity, protection, and ease of recovery if a
database becomes corrupted: Standard DB / with Continuos Protection or
Standard DB with a Follower DB.
Ease of recovery is the difference.
If you have a Follower DB, it is a hot standby - if the master fails for some reason, you can switch your application over to the follower with minimal downtime. On the other hand, if you have a large database and you have to restore it from the last base backup and then replay all the WAL files produced since - well that could take a long time, days even if it was a really large database.
Note that also, however, that a follower DB will be of no use if your database becomes corrupted due to, for example, an administrator accidentally dropping the wrong table. The table will be dropped in the follower only a few seconds later. They are like lemmings going over a cliff. The same applies if your application corrupts the database due to a bug, or a hacker, or whatever. Even with the follower, you must have a proper backup in place, either a Continuous Archive or a normal pg_dump.
If by chance the DB becomes corrupted, or an database integrity issues
arise, how will Heroku remediate (given the database is a "managed"
service). Is it automated or I have to work with Support manually to
remediate?
Their documentation indicates that premium plans do feature automated failover. This would be useful in the event of a hardware or platform failure and most kinds of database failure, where the system can detect that the master database has gone down and initiate a failover.
In the case where the database becomes corrupted by the application itself (or a hasty admin) then I suspect you would have to manually initiate failover.