I upgraded my Cloud SQL machine from a 'db-f1-micro' 0.6GB RAM machine to a 'db-n1-standard-1' 3.75GB RAM machine last week. Running:
SELECT ##innodb_buffer_pool_size;
The output is:
1375731712
which I believe is 1.38GB. Here's the memory utilization for the primary and replica:
This seems oddly low for this machine type but researching (How to set innodb_buffer_pool_size in mysql in google cloud sql?) it doesn't appear I can alter the innodb_buffer_pool_size. Is this somehow dynamically set and slowly increasing over time? Doesn't appear to be near the 75-80% range google appears to aim for on these.
What is the value of innodb_buffer_pool_chunk_size and innodb_buffer_pool_instances?
innodb_buffer_pool_size must always be equal to or a multiple of the product of these two values, and will be automatically adjusted to be so. Chunk size can only be modified at startup, as explained in the docs page for InnoDB Buffer Pool Size configuration.
For Google CloudSQL in particular, not only the absolute, but also the relative size of the innodb_buffer_pool_size depends on instance type. I work for GCP support, and after some research in our documentation, I can tell that pool size is automatically configured based on an internal formula, which is subject to change. Improvements are being made to make instances more resilient against OOMs, and the buffer pool size has an important role in this.
So, it is expected behaviour that with your new instance type, and possibly different innodb_buffer_pool_chunk_size and innodb_buffer_pool_instances, you might get a quite changed memory usage. Currently, the user does not have control over the innodb_buffer_pool_size.
Related
What is the easiest way to move Google Cloud SQL instance(Postgres 9.6) from one google project to another with minimum or zero downtime? Instance size is about 20 GB
There is a service called "Migration job" which looks very relevant https://cloud.google.com/database-migration/docs/postgres/create-migration-job . But I cannot understand whether it can be used to move instance from one google project to another.
Simple restoring from backup is not really my case because I want to achieve minimum possible downtime, so I'm looking for something like 2 running instances with the synced real-time data.
PS. I also have configured VM with pgbouncer
Yes, Database Migration Service could be used to move one Cloud SQL instance from one GCP project to another. This is a cheaper way than the next approach, and although it requires more setup, it should be faster too. A connection profile can be created for the existing Cloud SQL instance, and a Cloud SQL target must be created in the destination project, but once everything is set up, most of the migration will be automatic. This is a well documented procedure, of which you can find information in our documentation.
Developers sometimes want to migrate their (normal) relational database with “zero” downtime. While downtime can be reduced, migration cannot be done without any impact on applications (that is, zero downtime). Replication causes replication lag.
The instant the decision is made to “migrate” all applications from one replica to another, applications (and therefore developers) have to wait (that is, downtime) for at least as long as the “replication lag” before using the new database. In practice, the downtime is a few orders of magnitude higher (minutes to hours) because:
Database queries can take multiple seconds to complete, and in flight queries must be completed or aborted at the time of migration.
The database has to be “warmed up” if it has substantial buffer memory - common in large databases.
If database shards have duplicate tables, some writes may need to be paused while the shards are being migrated.
Applications must be stopped at source and restarted in GCP, and connection to the GCP database instance must be established.
Network routes to the applications must be rerouted. Based on how DNS entries are set up, this can take some time.
All of these can be reduced with some planning and “cost” (some operations not permitted for some time before/after migration).
Decreasing the load on the source DB until the migration completes might help, and downtime might be less disruptive.
Other considerations:
Increase the machine types to increase network throughput.
Increase SSD size for higher IOPS/MBps.
More about.
The most intuitive way would be to export the data from the Cloud SQL instance to a GCS bucket, and import it to a new instance in the new project. This would imply some downtime, and you would have to manually create the instance in the target project with the same configuration as the original; it does require some manual steps, but it would be a simple and verifiable way to copy the data across an instance in a different project.
I have a free tier RDS PostgreSQL database db.t2.micro (1 core, 1 Gib). My architecture did the RDS collapse because it reaches the max number of concurrent connections.
When I query select * from pg_settings where name='max_connections', the result was 87.
I found this formula about the capacity of the max_connection attribute for each instance based on the memory:
LEAST({DBInstanceClassMemory/9531392},5000)
For my instance, the number was 104, and I modified the parameter group to this value, and still my RDS collapse.
I made a last attempt updating the max_connections to 500 believing that it'll not work because the limit is 104. But to my surprise the database worked and could handle all the concurrent connections (above 104):
Obviously, I'm missing something.
Is really there a limit of max_connections for my instance?
If I change the max_connection setting the pricing for my instance change?
Also, I'm curious about What represents the horizontal line in the graphic, because it is at the level of my initial max_connection setting, and before the change, it was not present
Thanks!
Based on the information at https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html#RDS_Limits.MaxConnections, you should've hit the limit there.
That said, I guess one of two possible things are happening here:
AWS do not enforce the limits. I guess they could be suggesting the limits based on instance size here.
There is some kind of "burst" allowed. Similar to IOPS there could be a small "burst" balance that you can scale to temporarily without affecting service.
I recently upgraded a Postgres 9.6 instance to 11.1 on Google Cloud SQL. Since then I've begun to notice a large number of the following error across multiple queries:
org.postgresql.util.PSQLException: ERROR: could not resize shared
memory segment "/PostgreSQL.78044234" to 2097152 bytes: No space left
on device
From what I've read, this is probably due to changes that came in PG10, and the typical solution involves increasing the instance's shared memory. To my knowledge this isn't possible on Google Cloud SQL though. I've also tried adjusting work_mem with no positive effect.
This may not matter, but for completeness, the instance is configured with 30 gigs of RAM, 120 gigs of SSD hd space and 8 CPU's. I'd assume that Google would provide an appropriate shared memory setting for those specs, but perhaps not? Any ideas?
UPDATE
Setting the database flag random_page_cost to 1 appears to have reduced the impact the issue. This isn't a full solution though so would still love to get a proper fix if one is out there.
Credit goes to this blog post for the idea.
UPDATE 2
The original issue report was closed and a new internal issue that isnt viewable by the public was created. According to a GCP Account Manager's email reply however, a fix was rolled out by Google on 8/11/2019.
This worked for me, I think google needs to change a flag on how they're starting the postgres container on their end that we can't influence inside postgres.
https://www.postgresql.org/message-id/CAEepm%3D2wXSfmS601nUVCftJKRPF%3DPRX%2BDYZxMeT8M2WwLSanVQ%40mail.gmail.com
Bingo. Somehow your container tech is limiting shared memory. That
error is working as designed. You could figure out how to fix the
mount options, or you could disable parallelism with
max_parallel_workers_per_gather = 0.
show max_parallel_workers_per_gather;
-- 2
-- Run your query
-- Query fails
alter user ${MY_PROD_USER} set max_parallel_workers_per_gather=0;
-- Run query again -- query should work
alter user ${MY_PROD_USER} set max_parallel_workers_per_gather=2;
-- -- Run query again -- fails
You may consider increasing Tier of the instance, that will have influence on machine memory, vCPU cores, and resources available to your Cloud SQL instance. Check available machine types
In Google Cloud SQL PostgreSQL is also possible to change database flags, that have influence on memory consumption:
max_connections: some memory resources can be allocated per-client, so the maximum number of clients suggests the maximum possible memory use
shared_buffers: determines how much memory is dedicated to PostgreSQL to use for caching data
autovacuum - should be on.
I recommend lowering the limits, to lower memory consumption.
We plan to use MongoDB in production for our website which will be hosted (for the start) on a single ec2 instance.
What is the recommandation for a MongoDB, which will have around 25k documents for the start with low traffic? So far i am not used to AWS, therefore a have no comparison to other dedicated hosters.
The "storagesize" of the collection in question will be arond 400MB, "totalindexsize" maybe around 20MB
If it's not a production instance, you can start with t2.medium instance. If its a production instance, start with m5.large.
Attach a new ebs volume of size 10GB and configure mongodb to use this new volume as the data directory. This helps to scale up your storage easily at later point of time.
Make sure you format your ebs volume to have xfs file system before installing mongodb which is required for best performance by mongo.
Also, later if you feel like increasing the instace size wheb your traffic increases, just use the "instance modify" option to get it done.
I cannot give you a specific answer, but the cloud provides you with the option to test your setup very quickly. Just start with some instance (e.g. m3.medium) and you can create an Amazon Machine Image of your running MongoDB instance anytime and just start it on a larger or smaller instance type.
You can find the instance types here: https://aws.amazon.com/ec2/instance-types/
Deeper thought about the instance type choice can be found here: https://www.mongodb.com/blog/post/maximizing-mongodb-performance-on-aws
If you have any doubt about sizing, err on the side of going larger and then scaling down as needed. Undersizing your instances when deploying MongoDB can, and likely will, create performance problems. In order to achieve the best performance, your MongoDB working set should fit in memory.
The working set is the portion of data and related indexes that your clients access most frequently.
I am going to configure mongodb on a small number of cloud servers.
I am coming from mysql, and I remember that if I needed to change settings like RAM, etc. I would have to modify "my.cnf" file. This came useful while resizing each cloud server.
Now, how can I check or modify how much RAM or disk space the database is going to take for each node?
thank you in advance.
I don't think there are any built in broad stroke limitation tools or flags in mongodb per se and that is most likely because this is something you should be doing at the operating system level.
Most modern multi-user operating systems have built in ways to set quotas on disk space, etc per user so you could probably set up a mongo user and place the limits on them if you really wanted to. MongoDB works best when it has enough memory to hold the working set of data and indexes in memory and it does a good job of managing that on its own.
However, if you want to get granular you can take a look at the help output of mongod --help
I see the following options that you could tweak:
--nssize arg (=16) .ns file size (in MB) for new databases
--quota limits each database to a certain number of files (8 default)
--quotaFiles arg number of files allower per db, requires --quota