My Server has following resources :
[postgres#srv2813 ~]$ free -m
total used free shared buffers cached
Mem: 15929 15118 810 142 12 219
-/+ buffers/cache: 14885 1043
Swap: 8031 2007 6024
[postgres#srv2813 ~]$ cat /proc/cpuinfo | grep processor | wc -l
8
[root#srv2813 postgres]# sysctl kernel.shmall
kernel.shmall = 4194304
[root#srv2813 postgres]# sysctl kernel.shmmax
kernel.shmmax = 17179869184
and My PostgreSQL conf :
default_statistics_target = 100
maintenance_work_mem = 1GB
checkpoint_completion_target = 0.9
effective_cache_size = 12GB
work_mem = 32MB
wal_buffers = 16MB
shared_buffers = 3840MB
max_connections = 500
fsync = off
temp_buffers=32MB
But its getting "too many connection" error. The nginx_status page of the webserver shows around 500 active connections when this happens. The server hosts an api severver, so every "http request" invariably initiate a database "read". Its not a "write" heavy thing, but very "read" heavy.
Its possible that i maxed out our sever, but still i expected a little more from a 16G/8 core box considering the "read only" nature of the application. Can i push the PostgreSQL in any other possible direction?
PostgreSQL is process based vs thread based so it does not generally work well with a lot of connections.
I would look at using something like PgBouncer. PgBouncer is a lightweight connection pooler for PostgreSQL.
Related
I started a project that is already in production and we have a few clients already registered and paid for the service (SaaS business).
We use an Ubuntu VPS with Postgresql with default postgresql.conf settings.
Based on PGtune, to optimize the database, I should set these settings in postgresql.conf:
# DB Version: 10
# OS Type: linux
# DB Type: web
# Total Memory (RAM): 1 GB
# CPUs num: 1
# Data Storage: ssd
max_connections = 200
shared_buffers = 256MB
effective_cache_size = 768MB
maintenance_work_mem = 64MB
checkpoint_completion_target = 0.7
wal_buffers = 7864kB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 655kB
min_wal_size = 1GB
max_wal_size = 4GB
At the moment I have the following settings:
max_connections = 100
shared_buffers = 128MB
And all other settings recommended by PGTune are commented out.
Is it safe to change/adust these settings while the Database is in production and up and running?
You cannot change max_connections and shared_buffers without restarting PostgreSQL, so you will need a short down time.
Raising max_connections above 100 is a bad idea. If you need many client connections, use pgBouncer.
I have pgbouncer.ini file with the below configuration
[databases]
test_db = host=localhost port=5432 dbname=test_db
[pgbouncer]
logfile = /var/log/postgresql/pgbouncer.log
pidfile = /var/run/postgresql/pgbouncer.pid
listen_addr = 0.0.0.0
listen_port = 5433
unix_socket_dir = /var/run/postgresql
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
admin_users = postgres
#pool_mode = transaction
pool_mode = session
server_reset_query = RESET ALL;
ignore_startup_parameters = extra_float_digits
max_client_conn = 25000
autodb_idle_timeout = 3600
default_pool_size = 250
max_db_connections = 250
max_user_connections = 250
and I have in my postgresql.conf file
max_connections = 2000
does it effect badly on the performance ? because of max_connections in my postgresql.conf ? or it doesn't mean anything and already the connection handled by the pgbouncer ?
one more question. in pgpouncer configuration, does it right listen_addr = 0.0.0.0 ? or should to be listen_addr = * ?
Is it better to set default_pool_size on PgBouncer equal to the number of CPU cores available on this server?
Shall all of default_pool_size, max_db_connections and max_user_connections to be set with the same value ?
So the idea of using pgbouncer is to pool connections when you can't afford to have a higher number of max_connections in PG itself.
NOTE: Please DO NOT set max_connections to a number like 2000 just like that.
Let's start with an example, if you have a connection limit of 20 and then your app or organization wants to have a 1000 connections at a given time, that is where pooler comes into picture and in this specific case you want the 20 connections to pool that 1000 coming in from the application.
To understand how it actually works let's take a step back and understand what happens when you do not have a connection pooler and only rely on PG config setting for the max connections which in our case is 20.
So when a connection comes in from a client\application etc. the main process of postgresql(PID, i.e. parent ID) spawns a child for that. So each new connection spawns a child process under the main postgres process, like so:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24379 postgres 20 0 346m 148m 122m R 61.7 7.4 0:46.36 postgres: sysbench sysbench ::1(40120)
24381 postgres 20 0 346m 143m 119m R 62.7 7.1 0:46.14 postgres: sysbench sysbench ::1(40124)
24380 postgres 20 0 338m 137m 121m R 57.7 6.8 0:46.04 postgres: sysbench sysbench ::1(40122)
24382 postgres 20 0 338m 129m 115m R 57.4 6.5 0:46.09 postgres: sysbench sysbench ::1(40126)
So now once a connection request is sent, it is received by the POSTMASTER process and creates a child process at OS level under the main parent process. This connection then has a life span of "unlimited" unless close by the application or you have a time out set for idle connections in postgresql.
Now here comes the situation where it can be a very costly affair to manage the connections with a given compute, if they exceed a certain limit. Meaning n number of connections when served have a given compute cost and after some time the OS won't be able to handle a situation with HUGE connections and will in turn cause contentions at different compute level(i.e. Memory, CPU, I/O).
What if you can use the presently spawned child processes(backends) if they are not doing any work. You will save time on getting the child process(backends) and the additional cost as well(this can be different at times). This is where the pool of connections that are always open help to serve different client requests comes in and is also called pooling.
So basically now you have only n connections available but the pooler can manage n+i number of connections to serve the client requests.
This where pg-bouncer helps to reuse the connections. It can be configured with 3 types of pooling i.e Session pooling, Statement pooling and Transaction pooling. Basically bouncer returns the connection back to the pool once it has done, statement level work or transaction level work etc. Only during session pooling it keeps the connections unless it disconnects.
So basically lower down the number of connections at PG conf file level and tune all settings in the bouncer.ini.
To answer the second part:
one more question. in pgpouncer configuration, does it right listen_addr = 0.0.0.0 ? or should to be listen_addr = * ?
It depends if you have a standalone deployment, server etc.
basically if its on the server itself and you want it to allow connections from everywhere(incoming) use "*" if you want only the local network to be allowed use "127.0.0.0".
For the rest of your questions check this link: pgbouncer docs
I have tried to share a little of what I know, feel free to ask away if anything was unclear or or correct if it was incorrectly mentioned.
I had a PostgreSQL based web application that run into no connection left problem. so I used pgbouncer to solve this problem. It works very much better with pgbouncer but I'm still having some problems with connection limit...
Right now my postgresql.conf file contains:
max_connections = 100
shared_buffers = 128MB
#temp_buffers = 8MB
#max_prepared_transactions = 0
#work_mem = 1MB
#maintenance_work_mem = 16MB
#max_stack_depth = 2MB
#temp_file_limit = -1
#max_files_per_process = 1000
#shared_preload_libraries = ''
and my pgbouncer.ini file:
pool_mode = session
max_client_conn = 100
default_pool_size = 45
reserve_pool_size = 1
It still encounters some connection limit problems at peak time of site.
Can you help me configure this settings to better configuration.
Thanks in advance.
I got many times an error of POSKeyError. I think our setting is not enough parameters of PostgreSQL. Because the system chenged storage from MySQL to PostgreSQL. I got the error many times before the chenging.
Please let me know the specific setting or any points.
Using version:
Plone 4.3.1
RelStorage 1.5.1 with PostgreSQL on RDS, AWS
shared-blob-dir true (stored on the filesystem)
Plone Quick Upload 1.8.2
Here are some PostgreSQL tune-ups within postgresql.conf:
# shared_buffers and effective_cache_size should be 30%-50%
# of your machine free memory
shared_buffers = 3GB
effective_cache_size = 2GB
checkpoint_segments = 64
checkpoint_timeout = 1h
max_locks_per_transaction = 512
max_pred_locks_per_transaction = 512
# If you know what you're doing you can uncomment and adjust the following values
#cpu_tuple_cost = 0.0030
#cpu_index_tuple_cost = 0.0001
#cpu_operator_cost = 0.0005
And here are they explained by Jens W. Klein:
most important: shared_buffers = 3GB (set it to 30%-50% of your
machine free memory)
checkpoint_segments = 64,
checkpoint_timeout = 1h (decreases logging overhead)
max_locks_per_transaction = 512,
max_pred_locks_per_transaction = 512 (relstorage needs lots of them)
effective_cache_size = 4GB (adjust to ~50% of your memory)
just for import you could disable fsync in the config, then it should be really fast, but don't switch off the machine
CPU tweaks. We didn't touch the default values for these, but if you
know what you're doing, go for it. Bellow are some recommended
values:
cpu_tuple_cost = 0.0030,
cpu_index_tuple_cost = 0.001,
cpu_operator_cost = 0.0005 (query planning optimizations, the defaults are some year old, so current cpus are faster, these are better estimates, but i don't know how to get here "real" values)
You should also read https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
And here is our buildout.cfg:
[instance1]
recipe = plone.recipe.zope2instance
rel-storage =
type postgresql
host 10.11.12.13
dbname datafs
user zope
password secret
blob-dir /var/sharedblobstorage/blobs
blob-cache-size 350MB
poll-interval 0
cache-servers 10.11.12.14:11211
cache-prefix datafs
I have 3GB ram assigned to VMBox. Doing similarity check on column of a table consisting of about 3885 entries, so in total 3885*3885 comparisons. But I get this error:
[23636.611505] Out of memory: Kill process 987 (postgres) score 848 or sacrifice child
[23636.612762] Killed process 987 (postgres) total-vm:5404836kB, anon-rss:2772756kB, file-rss:828kB
server closed the connection unexpectedly
This probably means the server terminated abnormally
before of while processing the request
The connection to the server was lost. Ateempting reset: Succeeded.
Still no idea why 15,093,225 comparisons cause "Out of memory" on 3gb ram. Any solutions?
Edit: I increased
Shared_buffers = from 128MB to 1000MB
Working_mem = from 4MB to 50MB
But still same error.
Edit2: I did explain and same error. Cartesian, because I have to compare sentences that are imported in cells.