Suddenly we are having a memory issue on our server and I think it's due to postgres
Our replicating slave went down and still down, and I think it causes postgres to choak?
when I do ps -ef I see lots of the following lines
postgres 31161 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.22.123(53570) INSERT waiting for 26C/F5875798
postgres 31175 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.30.130(36126) INSERT
postgres 31176 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.30.130(36124) INSERT waiting for 26C/F645AB88
postgres 31177 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.30.130(36128) INSERT waiting for 26C/F644A6B8
postgres 31210 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.29.17(51158) INSERT
postgres 31250 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.19.52(36284) UPDATE
postgres 31251 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.19.52(36286) INSERT waiting for 26C/F5C32028
postgres 31252 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.19.52(36288) INSERT
postgres 31387 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.22.54(42988) INSERT
postgres 31389 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.22.54(42986) INSERT
postgres 31434 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.30.116(33648) UPDATE
postgres 31859 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.30.13(33494) INSERT
postgres 31861 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.30.13(33496) INSERT
postgres 31886 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.29.48(34408) UPDATE
postgres 31888 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.29.48(34410) INSERT waiting
postgres 31936 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.26.115(50432) INSERT waiting for 26C/F5C59038
postgres 31937 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.26.115(50428) UPDATE waiting
postgres 31938 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.26.115(50430) UPDATE waiting for 26C/F5C4AA48
postgres 32233 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.29.17(51240) INSERT
postgres 32297 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.24.193(56344) INSERT waiting for 26C/F58CCD90
postgres 32298 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.24.193(56342) INSERT waiting
postgres 32299 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.24.193(56346) INSERT waiting
postgres 32703 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.24.99(50452) INSERT
postgres 32705 13564 0 Aug22 ? 00:00:00 postgres: momsplanner littlehome_db x.x1.24.99(50454) INSERT waiting
7 of them seem to be waiting for syncrep (presumably from your defunct replica). Do you intentionally have synchronous replication set up? You either need to get your replica back running ASAP, or turn off synchronous replication. Another 5 are waiting on locks, maybe ones held by those first 7.
On the other hand, 12 of them don't seem to be blocked. It is hard to say what is going on with them. If you were using synchronous replication consistently, I would expect almost every connection to soon be piled up on it, not just half of them.
It is also odd to see 11 different IP addresses connected to a database named "littlehome_db". Are all those connections authorized and welcome?
Related
I want to restart my postgres using command
sudo service postgresql restart
However I'm getting the following error:
[warn] No PostgreSQL clusters exist; see "man pg_createcluster" ...
(warning).
I already have postgres running, showing through ps aux:
# ps aux| grep postgres
postgres 1 0.0 1.1 287484 24328 ? Ss 21:58 0:00 postgres
postgres 77 0.0 0.4 287608 8588 ? Ss 21:58 0:00 postgres: checkpointer process
postgres 78 0.0 0.1 287484 3884 ? Ss 21:58 0:00 postgres: writer process
postgres 79 0.0 0.4 287484 8680 ? Ss 21:58 0:00 postgres: wal writer process
postgres 80 0.0 0.3 287896 6640 ? Ss 21:58 0:00 postgres: autovacuum launcher process
postgres 81 0.0 0.1 142612 3120 ? Ss 21:58 0:00 postgres: stats collector process
root 1253 0.0 0.0 12788 972 pts/0 S+ 22:31 0:00 grep postgres
Now searching online, I found this post. I followed the solution to create a cluster:
sudo pg_createcluster 9.6 main
sudo service postgresql stop
sudo service postgresql start
After this, I am able to "restart" posgres through
sudo service postgresql restart
HOWEVER, this is not on the original cluster. As if I run ps-aux, I see a new cluster setup, this is not what I want.
# ps aux| grep postgres
postgres 1 0.0 1.1 287484 24328 ? Ss 21:58 0:00 postgres
postgres 77 0.0 0.4 287608 8588 ? Ss 21:58 0:00 postgres: checkpointer process
postgres 78 0.0 0.1 287484 3884 ? Ss 21:58 0:00 postgres: writer process
postgres 79 0.0 0.4 287484 8680 ? Ss 21:58 0:00 postgres: wal writer process
postgres 80 0.0 0.3 287896 6640 ? Ss 21:58 0:00 postgres: autovacuum launcher process
postgres 81 0.0 0.1 142612 3120 ? Ss 21:58 0:00 postgres: stats collector process
postgres 1292 0.0 1.2 287628 25680 ? S 22:42 0:00 /usr/lib/postgresql/9.6/bin/postgres -D /var/lib/postgresql/9.6/main -c config_file=/etc/postgresql/9.6/main/postgresql.conf
postgres 1294 0.0 0.1 287628 3944 ? Ss 22:42 0:00 postgres: 9.6/main: checkpointer process
postgres 1295 0.0 0.1 287628 3940 ? Ss 22:42 0:00 postgres: 9.6/main: writer process
postgres 1296 0.0 0.1 287628 3944 ? Ss 22:42 0:00 postgres: 9.6/main: wal writer process
postgres 1297 0.0 0.3 288056 6464 ? Ss 22:42 0:00 postgres: 9.6/main: autovacuum launcher process
postgres 1298 0.0 0.1 142620 3172 ? Ss 22:42 0:00 postgres: 9.6/main: stats collector process
root 1329 0.0 0.0 12788 968 pts/0 S+ 22:42 0:00 grep postgres
All the start/restart/stop is all applied to the new cluster. I just want to restart the original cluster. How can I do that?
I found that this happened because the postgresql db was running inside a container. To restart the database, I'll need to restart the docker container
I tried to profile plpgsql functions.
And I use pg_stat_statements to do that.
I can do profile, but after that I can not restart postgresql.
Result of ps -aux is below
postgres 22129 0.0 0.1 1276896 37276 ? Ss 07:13 0:00 postgres: dbname: checkpointer process
postgres 22134 0.0 0.0 178008 4360 ? Ss 07:13 0:00 postgres: dbname: stats collector process
postgres 23030 18.5 0.9 1436048 316336 ? Ss 07:17 4:53 postgres: dbname: postgres sakura 127.0.0.1(56496) idle waiting for D1E/CEA4D9C8
I think stats process may be lock the server to restart.
I think it's locking to stop server, because after that I have to wait long time to only shutdown the PC. May be shutdown process is waiting until stats process stop.
Is there any way to stop stats process properly?
Or, the way to force top stats process?
I tried kill -9, but it does not help.
I installed PostgreSQL 9.6 on my Ubuntu 16.04 system. It seems like it's up and running, but when I try to make a connection it just fails. When I run the status command I see that it's always saying exited for some reason. I see lots of things in a google search but none seem to be helping.
I changed the first entry in the pg_hba.conf to be local all postgres trust
% ps augxw | grep postg
postgres 769 0.0 0.3 303964 24384 ? S 22:13 0:00 /usr/lib/postgresql/9.6/bin/postgres -D /var/lib/postgresql/9.6/main -c config_file=/etc/postgresql/9.6/main/postgresql.conf
postgres 772 0.0 0.0 303964 3956 ? Ss 22:13 0:00 postgres: 9.6/main: checkpointer process
postgres 773 0.0 0.0 303964 3956 ? Ss 22:13 0:00 postgres: 9.6/main: writer process
postgres 774 0.0 0.0 303964 3956 ? Ss 22:13 0:00 postgres: 9.6/main: wal writer process
postgres 775 0.0 0.0 304408 6572 ? Ss 22:13 0:00 postgres: 9.6/main: autovacuum launcher process
postgres 776 0.0 0.0 158964 3204 ? Ss 22:13 0:00 postgres: 9.6/main: stats collector process
ubuntu 1492 0.0 0.0 12944 936 pts/0 S+ 22:14 0:00 grep postg
%
% sudo su - postgres
$ psql -h localhost
psql: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
$ exit
logout
% systemctl status postgresql
● postgresql.service - PostgreSQL RDBMS
Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
Active: active (exited) since Thu 2017-03-30 22:13:57 PDT; 1min 19s ago
Process: 901 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
Main PID: 901 (code=exited, status=0/SUCCESS)
Tasks: 0
Memory: 0B
CPU: 0
CGroup: /system.slice/postgresql.service
Mar 30 22:13:57 ip-172-31-9-223 systemd[1]: Starting PostgreSQL RDBMS...
Mar 30 22:13:57 ip-172-31-9-223 systemd[1]: Started PostgreSQL RDBMS.
You are connecting using TCP/IP socket, but local refers to unix-domain socket.
The first field is the connection type: "local" is a Unix-domain socket, "host" is either a plain or SSL-encrypted TCP/IP socket, "hostssl" is an SSL-encrypted TCP/IP socket, and "hostnossl" is a plain TCP/IP socket.
I did install postgresql-9.6 from that repo and:
host all postgres 127.0.0.1/32 trust does what you want to do
postgres#lkaminski-ubuntu-desk:~$ psql works out of the box, no need to change config. It is using unix-domain socket that is already trusted. So you can just drop -h localhost and no need to edit configs.
$ sudo grep -e "^[^#]" /etc/postgresql/9.6/main/pg_hba.conf
local all postgres peer
host all postgres 127.0.0.1/32 trust
local all all peer
host all all 127.0.0.1/32 md5
host all all ::1/128 md5
The issue ended up being that version 9.5 had the normal 5432 port and so when 9.6 got installed/started, it pointed to 5433. Deleting the 9.5 install and editing the config file to point at 5432 fixed the issue.
I'm trying to determine if Postgres 9.3 still has a logger process. It isn't referenced anywhere in the "PostgreSQL 9.3.4 Documentation". And I can't find it in my cluster's process list (see below). Also, does anyone know of a good general overview the memory structures in 9.3?
postgres 21397 1 0 20:51 pts/1 00:00:00 /opt/PostgreSQL/9.3/bin/postgres
postgres 21399 21397 0 20:51 ? 00:00:00 postgres: checkpointer process
postgres 21400 21397 0 20:51 ? 00:00:00 postgres: writer process
postgres 21401 21397 0 20:51 ? 00:00:00 postgres: wal writer process
postgres 21402 21397 0 20:51 ? 00:00:00 postgres: autovacuum launcher process
postgres 21403 21397 0 20:51 ? 00:00:00 postgres: archiver process last was 0001000004000092
postgres 21404 21397 0 20:51 ? 00:00:00 postgres: stats collector process
Thanks
Jim
Postgres has a logging collector process which is controlled through a config parameter,
logging_collector.
So in your postgresql.conf file, you would make sure this is set:
logging_collector = on
The blurb on this param from the postgres doc:
This parameter enables the logging collector, which is a background
process that captures log messages sent to stderr and redirects them
into log files. This approach is often more useful than logging to
syslog, since some types of messages might not appear in syslog
output. (One common example is dynamic-linker failure messages;
another is error messages produced by scripts such as
archive_command.) This parameter can only be set at server start.
It will show up in the process list with the following description:
postgres: logger process
For more info: http://www.postgresql.org/docs/current/static/runtime-config-logging.html
Regarding the memory structures, I'm not sure offhand, but would recommend you post that as a separate question.
When i want insert a row to table.
It always hangs on waiting can not commit successful.
postgres 17941 2092 0 16:42 ? 00:00:00 /usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main -c config_file=/etc/postgresql/9.1/main/postgresql.conf
postgres 17943 17941 0 16:42 ? 00:00:00 postgres: writer process
postgres 17944 17941 0 16:42 ? 00:00:00 postgres: wal writer process
postgres 17945 17941 0 16:42 ? 00:00:00 postgres: autovacuum launcher process
postgres 17946 17941 0 16:42 ? 00:00:00 postgres: stats collector process
postgres 18326 17941 0 16:51 ? 00:00:00 postgres: admin gt_development 127.0.0.1(40309) COMMIT waiting for 0/1217CA98
Who can help me?
Thanks for reply.
I have fixed by pure reinstall the postgresql 9.1