My PostgreSQL database storage is expanding way too much relative to my real database storage. I am assuming it is making logs for every actions against the database. If so, how do I turn it off?
All tables' storage size in the database:
table_name | pg_size_pretty
-----------------+----------------
matches | 3442 MB
temp_matches | 3016 MB
rankings | 262 MB
atp_matches | 41 MB
players | 11 MB
injuries | 4648 kB
tournaments | 1936 kB
temp_prematches | 112 kB
locations | 104 kB
countries | 16 kB
(10 rows)
My storage usage should be around 10GB.
Your PostgreSQL instance may have Point-in-time recovery (PITR) enabled.
To add explanation, PITR uses write ahead logs (WAL). It is necessary to archive the WAL files for instances it is enabled on. This archiving is done automatically on the backend and will consume storage space (even if the instance is idle) and therefore using this feature would result on an increased storage space on your DB instance.
Here's a similar issue: Google Cloud SQL - Postgresql storage keeps growing
You can stop the storage increase by disabling Point-in-time recovery: https://cloud.google.com/sql/docs/postgres/backup-recovery/pitr#disablingpitr
First, I recommend you verify if you have set the value "Enable automatic storage increases" since your instance's storage will continue increasing and your pocket will be affected.
Please keep in mind that you can increase storage size, but you cannot decrease it; the storage increases are permanent for the life of the instance. When you enable this setting, a spike in storage requirements can permanently increase storage costs (incrementally) for your instance.
On the other hand, If you have the Point-in-time recovery ( PITR ) enabled, I recommend you disable it in order to delete the logs. If not, I think that It would be necessary contact the GCP support team in order that they inspect your instance carefully.
Related
Trying to save a PostgreSQL backup (~20 Tb) to Google Cloud Storage for the long-term, and I am currently piping PostgreSQL pg_dump() command to an streaming transfer through gsutil.
pg_dump -d $DB_NAME -b --format=t \
| gsutil cp - gs://$BUCKET_NAME/$BACKUP_FILE
However, I am worried that the process will crash because of GCS' 5Tb object size limit.
Is there any way to upload larger than 5Tb objects to Google Cloud Storage?
EDITION: using split?
I am considering to pipe pg_dump to Linux's split utility and the gsutil cp.
pg_dump -d $DB -b --format=t \
| split -b 50G - \
| gsutil cp - gs://$BUCKET/$BACKUP
Would something like that work?
You generally don't want to upload a single object in the multi-terabyte range with a streaming transfer. Streaming transfers have two major downsides, and they're both very bad news for you:
Streaming Transfers don't use Cloud Storage's checksum support. You'll get regular HTTP data integrity checking, but that's it, and for periodic 5 TB uploads, there's a nonzero chance that this could eventually end up in a corrupt backup.
Streaming Transfers can't be resumed if they fail. Assuming you're uploading at 100 Mbps around the clock, a 5 TB upload would take at least 4 and a half days, and if your HTTP connection failed, you'd need to start over from scratch.
Instead, here's what I would suggest:
First, minimize the file size. pg_dump has a number of options for reducing the file size. It's possible something like "--format=c -Z9" might produce a much smaller file.
Second, if possible, store the dump as a file (or, preferably, a series of split up files) before uploading. This is good because you'll be able to calculate their checksums, which gsutil can take advantage of, and also you'd be able to manually verify that they uploaded correctly if you wanted. Of course, this may not be practical because you'll need a spare 5TB of hard drive space, but unless your database won't be changing for a few days, there may not be an easy alternative to retry in case you lose your connection.
As mentioned by Ferregina Pelona, guillaume blaquiere and John Hanley. There is no way to bypass the 5-TB limit implemented by Google, as mentioned in this document:
Cloud Storage 5TB object size limit
Cloud Storage supports a maximum single-object size up to 5 terabytes.
If you have objects larger than 5TB, the object transfer fails for
those objects for either Cloud Storage or Transfer for on-premises.
If the file surpasses the limit (5 TB), the transfer fails.
You can use Google's issue tracker to request this feature, within the link provided, you can check the features that were requested or request a feature that satisfies your expectations.
I've had a replica slave set up for about two weeks now. It has been failing replication due to configuration issues, but still growing in the size of the master each day (about 5gb a day).
Until today, binary logs were disabled. And if I go to Monitoring -> slave instance, under Backup Configuration, it says "false".
How do I determine why this is growing each day?
I noticed in monitoring in the InnoDB Pages Read/Write section, there are upticks of Write each day, but no read. But what is it writing to? The DB hasn't changed. and there are no binary logs.
I noticed in the docs, it says "Point-in-time recovery is enabled by default when you create a new Cloud SQL instance."
But there has never been a "Backup" listed in the Operations list on the instance. And when I do gcloud sql instances describe my-instance, it's not listed under backUpConfiguration
The issue you are having could possibly happen due to Point-in-time recovery, it will show an increase to your storage constantly.
There, you will be able to keep automated backups enabled while disabling point-in-time recovery. Once you disable it, the binary logs will be deleted and you will notice an immediate reduction in storage usage.
Here are the steps to disable Point-in-time recovery:
Select your instance
Select Backups
Under Settings, select Edit
Uncheck box for point-in-time recovery
To add an explanation of Point-in-time recovery, I will add Google Cloud SQL documentation with Postgres and MySQL.
It is necessary to archive the WAL files for instances it is enabled on. This archiving is done automatically on the backend and will consume storage space (even if the instance is idle), and, consequently, using this feature would cause an increased storage space on your DB instance.
Ive got mysql 5.7 cloud instance in europe-west1 zone with 1vCPU and 4Gb RAM without redundancy enabled.
my db always wasnt above 30Gb, but suddenly became about 11TB.
I logged to instance and notify that DB size only 15,8Gb.
can somebody explain what happened and how i can reduce space using, which costs me 70$/day?
If you don't see an abnormal increment of binary logs while running the command SHOW BINARY LOGS; try to check the size of temporary tablespace by running the following command:
SELECT FILE_NAME, TABLESPACE_NAME, ENGINE, INITIAL_SIZE, TOTAL_EXTENTS*EXTENT_SIZE
AS TotalSizeBytes, DATA_FREE, MAXIMUM_SIZE FROM INFORMATION_SCHEMA.FILES
WHERE TABLESPACE_NAME = 'innodb_temporary'\G
You can also try to check the size of the general logs if you enabled them by running:
SELECT ROUND(SUM(LENGTH(argument)/POW(1024,2),2) AS GB from mysql.general_log;
If you want to solve this issue quickly to avoid more charges you can try to restart your instance (if temporal logs are filling your disk this will delete them) and export your database to a new instance with a smaller disk and then delete your old instance.
You can also try to contact Google Cloud support if you need further help with your Cloud SQL instance
I have a RDS Postgres database on version 9.6.6, and was tasked with upgrading it to version 11.
To do this, I created a snapshot and then provisioned a new 9.6.6 database. Next, I clicked "modify" in the RDS console and upgraded this new instance to version 10.11. Once it was on major version 10, RDS then allowed me to upgrade again to version 11.6.
However, within an hour of completing this process, the new 11.6 database seized up and entered into the storage-full status. The following CloudWatch event was produced right before this happened...
The free storage capacity for DB Instance: my_new_db is low at 2% of the provisioned storage [Provisioned Storage: 4.78 GB, Free Storage: 77.81 MB]. You may want to increase the provisioned storage to address this issue.
Both the old and the new database are allotted 5GiB of disk space. I don't believe the size of my database tables are a factor, since the following query returns 330MB, which is only 7% of the allotted space.
SELECT pg_size_pretty(pg_database_size('my_old_db'));
Since the size of my database tables clearly isn't the culprit, what could have possibly caused my RDS instance to run out of disk space?
Specs:
Instance class: db.t2.medium
Ram: 4GB
vCPU: 2
Encryption: Enabled
Storage type: General Purpose (SSD)
Storage: 5GiB
I had a similar issue recently when upgrading a 9.6.22 DB instance in-place to 13.3.
The instance had only 5 GB storage, with a DB that was roughly 2.5 GB, and disk space decreased rapidly after upgrade to about 1 GB (within an hour or so), then stabilised. There was no disk-full error but storage was about 80% used.
I went through the RDS 'disk full' troubleshooting page which didn't help as the extra storage was not within the Postgres database cluster (i.e. set of DBs in this instance).
The mitigation was to resize this into a DB instance with over 20 GB storage.
I believe both issues may be down to the increased minimum size of storage for RDS instances - at some point, AWS must have increased this from 5 GB to 20 GB. See this section of AWS docs and the AWS Console for RDS when creating a new instance.
I would say there is an RDS issue here, as the console or API should tell you that you need to increase the size of storage before upgrading - similarly to how it may tell you that current instance type is not supported for upgrade.
I'm trying to setup a production mongo system on Amazon to use as a datastore for a realtime metrics system,
I initially used the MongoDB AMIs[1] in the Marketplace, but I'm confused in that there is only one data EBS. I've read that Mongo recommends RAID 10 on EBS storage (8 EBS on each server). Additionally, I've read that the bare minimum for production is a primary/secondary with an arbiter. Is RAID 10 still the recommended setup, or is one provisioned IOPS EBS sufficient?
Please Advise. We are a small shop, so what is the bare minimum we can get away with and still be reasonably safe?
[1] MongoDB 2.4 with 1000 IOPS - data: 200 GB # 1000 IOPS, journal: 25 GB # 250 IOPS, log: 10 GB # 100 IOPS
So, I just got off of a call with an Amazon System Engineer, and he had some interesting insights related to this question.
First off, if you are going to use RAID, he said to simply do striping, as the EBS blocks were mirrored behind the scenes anyway, so raid 10 seemed like overkill to him.
Standard EBS volumes tend to handle spiky traffic well (it may be able to handle 1K-2K iops for a few seconds), however eventually it will tail off to an average of 100 iops. One suggestion was to use many small EBS volumes and stripe them to get better iops throughput.
Some of his customers use just the ephemeral storage on the EC2 images, but then have multiple (3-5) nodes in the availability set. The ephemeral storage is the storage on the physical machine. Apparently, if you use the EC2 instance with the SSD storage, you can get up to 20K iops.
Some customers will do a huge EC2 image w/ssd for the master, then do a smaller EC2 w/ EBS for the secondary. The primary machine is performant, but the failover is available but has degraded performance.
make sure you check 'EBS Optimized' when you spin up an instance. That means you have a dedicated channel to the EBS storage (of any kind) instead of sharing the NIC.
Important! Provisioned IOPS EBS is expensive, and the bill does not shut off when you shut down the EC2 instances they are attached to. (this sucks while you are testing) His advice was to take a snapshot of the EBS volumes, then delete them. When you need them again, just create new provisioned IOPS EBS volumes, restore the snapshot, then reconfigure your EC2 instances to attache the new storage. (it's more work than it should be, but it's worth it not to get sucker punched with the IOPS bill.
I've got the same question. Both Amazon and Mongodb try to market a lot on provisioned IOPs chewing over its advantages over a standard EBS volume. We run prod instances on m2.4xlarge aws instances with 1 primary and 2 secondaries setup per service. In the highest utilized service cluster, apart from a few slow queries the monitoring charts do not reveal any drop on performance at all. Page faults are rare occurrences and that too between 0.0001 and 0.0004 faults once or twice a day. Background flushes are in milliseconds and locks and queues are so far at manageable levels. I/O waits on the Primary node at any time ranges between 0 to 2 %, mostly less than 1 and %idle steadily stays above 90% mark. Do I still need to consider provisioned IOPs given we've a budget still to improve any potential performance drag? Any guidance will be appreciated.