I have tested pgtune on my postgres.config so I know what I can change there, but when I try to change max_connections or shared_buffers I can't restart postgres. I just get an error, but there is nothing in the log specifying the error. (Not sure where those logs go, but they are not in regular pg_log dir.)
My settings is:
shared_buffers = 24MB # (pgtune wizard 2013-04-11 = 120MB)
max_connections = 120 # (pgtune wizard 2013-04-11 = 200)
Im on a 512 linode which only runs postgresql. If I change shared_buffers beyond 24MB or max_connections beyond 120, I can't restart postgres.
I'm running on a Linode xen instance with Ubuntu 12.04.2 LTS:
Ubuntu 12.04.2 LTS (GNU/Linux 3.8.4-x86_64-linode31 x86_64)
Anyone know if postgres it self determine that 24MB and 120 connections is max for my system?
It sounds like you're probably exceeding a very low default limit for shared memory.
This is covered in the manual - see operating system resource limits. For Linux, see kernel.shmmax.
On a side-note, increasing max_connections is often the wrong answer. Most PostgreSQL instances will work best with a relatively small number of actively working connections. It's often best to use connection pooling to queue up work; you'll get better overall throughput with lower resource use. If your application doesn't have a connection pool built-in you can use PgBouncer as an external connection pool.
Related
We are using, PostgreSQL Version 14.
Our PC configuration at production
Windows 2016 Server
32 GB RAM
8 TB Hard disk
8 Core CPU
In my PostgreSQL.conf file
Shared_buffer 8 GB
Work_MEM 1 GB
Maintenance_work_mem 1 GB
Max_Connections 1000
I wish to handle 5000 connection at a time. Somebody suggest me to go with PGBouncer.
But we initially starts with PostgreSQL without PGBouncer.
I need to know, is my configuration is OK with the 5000 connection or we need to increase RAM or any other...
This is our first, PostgreSQL implementation. So Please suggest me to start with PostgreSQL with out PGBouncer.
Thank you
Note:
In SQL if we set -1, it will handle more number of connections. Like this is there any configuration is available in PostgreSQL
My Server Config :
CPU - 16 core
RAM - 64 GB
Storage : 2 TB
OS : CentOs 64 Bit
I have DB and java application on the same server.
My postgres config file has the following:
max_connections = 9999
shared_buffers = 6GB
However, when i check DB via show max_connections it shows only 500.
How can i increase the max_connections value ?
Either you forgot to remove the comment (#) at the beginning of the postgresql.conf line, or you didn't restart PostgreSQL.
But a setting of 500 is already much too high, unless you have some 100 cores in the machine and an I/O system to match. Use a connection pool.
We have a hosted PostgreSQL, with no access to the system or *.conf files.
I do have a admin access and can connect to it using Oracle SQL developer.
Can I run any command to increase the max_connections. All other parameters seems to be ok shared mem and buffers can hold more connections so there is not problem there.
Changing max_connection parameter needs a Postgres restart
Commands
Check max_connection just to keep current value in mind
SHOW max_connections;
Change max_connection value
ALTER SYSTEM SET max_connections TO '500';
Restart PostgreSQL server
Apparently, the hosted Postgres we are using does not provide this option. (compose.io)
So the work around is to use a pgbouncer to manage you connections better.
I am using Postgres DB for my product. While doing the batch insert using slick 3, I am getting an error message:
org.postgresql.util.PSQLException: FATAL: sorry, too many clients already.
My batch insert operation will be more than thousands of records.
Max connection for my postgres is 100.
How to increase the max connections?
Just increasing max_connections is bad idea. You need to increase shared_buffers and kernel.shmmax as well.
Considerations
max_connections determines the maximum number of concurrent connections to the database server. The default is typically 100 connections.
Before increasing your connection count you might need to scale up your deployment. But before that, you should consider whether you really need an increased connection limit.
Each PostgreSQL connection consumes RAM for managing the connection or the client using it. The more connections you have, the more RAM you will be using that could instead be used to run the database.
A well-written app typically doesn't need a large number of connections. If you have an app that does need a large number of connections then consider using a tool such as pg_bouncer which can pool connections for you. As each connection consumes RAM, you should be looking to minimize their use.
How to increase max connections
1. Increase max_connection and shared_buffers
in /var/lib/pgsql/{version_number}/data/postgresql.conf
change
max_connections = 100
shared_buffers = 24MB
to
max_connections = 300
shared_buffers = 80MB
The shared_buffers configuration parameter determines how much memory is dedicated to PostgreSQL to use for caching data.
If you have a system with 1GB or more of RAM, a reasonable starting
value for shared_buffers is 1/4 of the memory in your system.
it's unlikely you'll find using more than 40% of RAM to work better
than a smaller amount (like 25%)
Be aware that if your system or PostgreSQL build is 32-bit, it might
not be practical to set shared_buffers above 2 ~ 2.5GB.
Note that on Windows, large values for shared_buffers aren't as
effective, and you may find better results keeping it relatively low
and using the OS cache more instead. On Windows the useful range is
64MB to 512MB.
2. Change kernel.shmmax
You would need to increase kernel max segment size to be slightly larger
than the shared_buffers.
In file /etc/sysctl.conf set the parameter as shown below. It will take effect when postgresql reboots (The following line makes the kernel max to 96Mb)
kernel.shmmax=100663296
References
Postgres Max Connections And Shared Buffers
Tuning Your PostgreSQL Server
Adding to Winnie's great answer,
If anyone is not able to find the postgresql.conf file location in your setup, you can always ask the postgres itself.
SHOW config_file;
For me changing the max_connections alone made the trick.
EDIT: From #gies0r: In Ubuntu 18.04 it is at
/etc/postgresql/11/main/postgresql.conf
If your postgres instance is hosted by Amazon RDS, Amazon configures the max connections for you based on the amount of memory available.
Their documentation says you get 112 connections per 1 GB of memory (with a limit of 5000 connections no matter how much memory you have), but we found we started getting error messages closer to 80 connections in an instance with only 1 GB of memory. Increasing to 2 GB let us use 110 connections without a problem (and probably more, but that's the most we've tried so far.) We were able to increase the memory of an existing instance from 1 GB to 2 GB in just a few minutes pretty easily.
Here's the link to the relevant Amazon documentation:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html#RDS_Limits.MaxConnections
change max_connections variable
in postgresql.conf file located in
/var/lib/pgsql/data or /usr/local/pgsql/data/
Locate postgresql.conf file by below command
locate postgresql.conf
Edit postgresql.conf file by below command
sudo nano /etc/postgresql/14/main/postgresql.conf
Change
max_connections = 100
shared_buffers = 24MB
to
max_connections = 300
shared_buffers = 80MB
I have installed postgresql on an Azure VM and am running tests to see if postgresql can support the expected load. I have increased the max_connections value to 1000 but when I run ab -c 300, postgresql stops responding. Are there any other settings I should be changing?
Thanks, Kate.
PostgreSQL will perform best with a lot less than 1000 connections on most hardware. Usually less than 100. If your application cannot queue work using a connection pool, you should put an external connection pool like PgBouncer between your application and PostgreSQL.
See: https://wiki.postgresql.org/wiki/Number_Of_Database_Connections