Pipelinedb gets stuck under high load, worker process eats 100% CPU doing nothing - pipelinedb

Note: I am looking for any hints on how to debug this issue, not necessarily a direct answer to this specific problem.
I am measuring the performance of PipelineDB for use in our system.
I have defined a few continuous views (Calculating sums, top-K and such), feeding from a single stream (Which has ~20 columns, some text, mostly integers and booleans).
The test is written in Python, and I am using the psycopg2 cursor.copy_from() function, to achieve max performance.
PipelineDB behaves nicely when there the work specified by the continuous views is not too complicated.
However, when I ask it to calculate many top-K results, or many percentile_cont() values, the test hangs with the following symptoms:
The (single) 'worker0' process starts eating 100% CPU
The input process shows that it is running the COPY command, never changing to IDLE (During normal work, it changes between COPY and IDLE).
Test hangs (i.e. the copy_from() function call does not return)
Below is the output of a 'ps -ef' command showing all pipelinedb processes, after about a minute or running the test. Note that the worker0 process is consuming 100% CPU since the beginning of the test. It never resumes normal work ('top' shows that it is consuming exactly 100% CPU)
Test logs show that it is running OK for the first ~1 second, inserting ~30,000 events (In batches of 100), and then it hangs, because a call to the copy_from() function does not return.
When I reduce the amount of work PipelineDB has (By removing some of the continuous views), the test works OK, achieving up to 20,000 inserts per second, sustained for at least one minute.
I would like to note that all events have the same time stamp, and all views have a "GROUP BY minute" clause, hence a single row should be created/updated in every continuous view, during the test.
I have played with some configuration parameters, specifically those related to memory buffer sizes, sync methods, time intervals, max_wait and such, amount of workers, and could not find any combination that avoids the problem.
I do not know if I am hitting a PipelineDB issue, or a PostgreSQL issue.
Certainly it is not expected behavior, and cannot be tolerated in a real application.
Any hints, guesses, gut feelings etc. are welcome.
[orens#rd10 ~]$ps -ef | grep pipelinedb
UID PID PPID C STIME TTY TIME CMD
orens 3005 3004 0 11:17 ? 00:00:00 pipelinedb: logger process
orens 3007 3004 0 11:17 ? 00:00:00 pipelinedb: checkpointer process
orens 3008 3004 0 11:17 ? 00:00:00 pipelinedb: writer process
orens 3009 3004 0 11:17 ? 00:00:00 pipelinedb: wal writer process
orens 3010 3004 0 11:17 ? 00:00:00 pipelinedb: autovacuum launcher process
orens 3011 3004 0 11:17 ? 00:00:00 pipelinedb: stats collector process
orens 3012 3004 0 11:17 ? 00:00:00 pipelinedb: pipelinedb scheduler process
orens 3014 3004 0 11:17 ? 00:00:00 pipelinedb: bgworker: reaper0 [pipeline]
orens 3015 3004 0 11:17 ? 00:00:00 pipelinedb: bgworker: queue0 [pipeline]
orens 3016 3004 0 11:17 ? 00:00:00 pipelinedb: bgworker: combiner1 [pipeline]
orens 3017 3004 0 11:17 ? 00:00:00 pipelinedb: bgworker: combiner0 [pipeline]
orens 3018 3004 0 11:17 ? 00:00:00 pipelinedb: bgworker: worker0 [pipeline]
orens 3046 3004 0 11:17 ? 00:00:00 pipelinedb: bgworker: reaper0 [db1]
orens 3050 3004 0 11:17 ? 00:00:00 pipelinedb: bgworker: queue0 [db1]
orens 3052 3004 0 11:17 ? 00:00:00 pipelinedb: bgworker: combiner0 [db1]
orens 3056 3004 90 11:17 ? 00:01:06 pipelinedb: bgworker: worker0 [db1]
orens 3132 3004 1 11:17 ? 00:00:01 pipelinedb: ut_user db1 ::1(58830) COPY
[orens#rd10 ~]$

Related

Postgres walwriter and background writer using memory

I just did a large import of renderd tiles for an osm server. I want to start my next process (running the import of nominatim) but it takes a lot of memory. The problem I have is that walwriter, background writer, checkpointer are consuming 131GB of memory. I checked pg_top and the processes are sleeping. Is there any way to clear these processes safely or just force postgres to complete the walwriter and background writer?
I am using Postgres v12, and shared_buffers is set to 128GB.
HTOP:
pg_top:
last pid: 628600; load avg 0.08, 0.03, 0.04; up 1+00:31:38 02:22:22
5 5 sleeping
CPU states: 0.0% user, 0.0% nice, 0.0% system, 100% idle, 0.0% iowait
Memory: 487G used, 16G free, 546M buffers, 253G cached
DB activity: 0 tps, 0 rollbs/s, 0 buffer r/s, 100 hit%, 43 row r/s, 0 row w/s -
DB I/O: 0 reads/s, 0 KB/s, 0 writes/s, 0 KB/s
DB disk: 3088.7 GB total, 2538.8 GB free (17% used)
Swap: 45M used, 8147M free, 588K cached
627692 postgres 20 0 131G 4368K sleep 0:00 0.00% 0.00% postgres: 12/main: background writer
627691 postgres 20 0 131G 6056K sleep 0:00 0.00% 0.00% postgres: 12/main: checkpointer
627693 postgres 20 0 131G 4368K sleep 0:00 0.00% 0.00% postgres: 12/main: walwriter
628601 postgres 20 0 131G 11M sleep 0:00 0.00% 0.00% postgres: 12/main: postgres postgres [local] idle
627695 postgres 20 0 131G 6924K sleep 0:00 0.00% 0.00% postgres: 12/main: logical replication launcher
pg_wal directory:
Everything is just fine, and htop is lying to you.
Of course the background processes that access shared buffers will use that memory, and since it is shared memory, it is reported for each of these processes. In reality, it is allocated only once.
The shared memory allocated by PostgreSQL is slightly bigger than shared_buffers, so if that parameter is set to 128GB, you reported data make sense and are perfectly normal.
If you set max_wal_size = 32GB, it is normal to have a lot of WAL segments.

Is it mandatory parameter recover_target_timeline='latest' in switchover and switchback in PostgreSQL 9.4.1?

I have followed below steps for switchover and switchback.
Step 1:-
Disconnect application services from 10.x.x.10 and do the following
#Master(10.x.x.10)
pg_ctl -D /DATA_VEC/pgdata stop --mode=fast
#DR(20.x.x.20)
promote DR as read write mode
Step 2:- Start master as DR from new primary
#Master(10.x.x.10)
create recovery.conf
standby_mode = 'on'
primary_conninfo = 'user= password= host=20.x.x.20 port=9999
trigger_file = '/tmp/node1'
restore_command = 'cp /DATA_VEC/restore/%f "%p"'
pg_ctl -D /DATA_VEC/pgdata start
after promotion new standby, old primary not getting syncing with new primary server(old standby)
Logs from DR server which was primary.
2019-12-01 18:46:56 IST LOG: database system was shut down in recovery at 2019-12-01 18:46:53 IST
2019-12-01 18:46:56 IST LOG: entering standby mode
cp: cannot stat `/DATA_VEC/restore/00000002.history': No such file or directory
2019-12-01 18:46:56 IST LOG:
2019-12-01 18:46:56 IST LOG: restored log file "00000002000000000000000C" from archive
2019-12-01 18:46:57 IST LOG: consistent recovery state reached at 0/C000090
2019-12-01 18:46:57 IST LOG: record with zero length at 0/C000090
2019-12-01 18:46:57 IST LOG: database system is ready to accept read only connections
2019-12-01 18:46:57 IST LOG: started streaming WAL from primary at 0/C000000 on timeline 2
2019-12-01 18:46:57 IST LOG: replication terminated by primary server
2019-12-01 18:46:57 IST DETAIL: End of WAL reached on timeline 2 at 0/C000090.
2019-12-01 18:46:57 IST LOG: restored log file "00000002000000000000000C" from archive
2019-12-01 18:46:57 IST LOG: record with zero length at 0/C000090
2019-12-01 18:46:57 IST LOG: restarted WAL streaming at 0/C000000 on timeline 2
2019-12-01 18:46:57 IST LOG: replication terminated by primary server
2019-12-01 18:46:57 IST DETAIL: End of WAL reached on timeline 2 at 0/C000090.
#Master(10.x.x.10)
Pg_xlog content
-bash-4.1$ cd pg_xlog
-bash-4.1$ ll
total 65552
-rw------- 1 postgres postgres 302 Dec 1 12:52 00000002000000000000000A.00000028.backup
-rw------- 1 postgres postgres 16777216 Dec 1 13:52 00000002000000000000000B
-rw------- 1 postgres postgres 16777216 Dec 1 14:28 00000002000000000000000C
-rw------- 1 postgres postgres 16777216 Dec 1 12:52 00000002000000000000000D
-rw------- 1 postgres postgres 16777216 Dec 1 12:52 00000002000000000000000E
-rw------- 1 postgres postgres 41 Dec 1 13:57 00000002.history
-rw------- 1 postgres postgres 83 Dec 1 13:57 00000003.history
drwx------ 2 postgres postgres 4096 Dec 1 13:57 archive_status
#in restore_command location content:-
-bash-4.1$ cd /DATA_VEC/restore/
-bash-4.1$ ll
total 49156
-rw------- 1 postgres postgres 16777216 Dec 1 18:45 00000002000000000000000A
-rw------- 1 postgres postgres 16777216 Nov 30 21:22 00000002000000000000000B
-rw------- 1 postgres postgres 16777216 Dec 1 18:45 00000002000000000000000C
-rw------- 1 postgres postgres 83 Dec 1 18:45 00000003.history
-bash-4.1$
as per_pg_xlog timeline history file 00000003.historyarrived at standby still not starting streaming from new primary.
Question:-
1. Is it mandatory parameter recover_target_timeline='latest' in recovery.conf file" to get latest timeline id from new primary through streaming to start streaming replication?
2.If yes, is it for all PostgreSQL Version? like from 9.3 to 11.5
If you want switch-back functionality, you will have to set recovery_target_timeline='latest', as any promotion will increment the timeline. Using a fixed target timeline is usually reserved for very specific cases (i.e., you need to recover changes after a split-brain, diverged-timeline scenario).
To answer your specific questions:
Yes
Yes

LSF job states for a given job

Let's say my job was running for some time and it went to suspend state due to machine overloading and became running after sometime and got completed.
Now the status acquired by this job were RUNNING -> SUSPEND -> RUNNING
How to get all states acquired by a given job ?
bjobs -l If the job hasn't been cleaned from the system yet.
bhist -l Otherwise. You might need -n, depending on how old the job is.
Here's an example of bhist -l output when a job was suspended and later resumed because the system load temporarily exceeded the configured threshold.
$ bhist -l 1168
Job <1168>, User <mclosson>, Project <default>, Command <sleep 10000>
Fri Jan 20 15:08:40: Submitted from host <hostA>, to
Queue <normal>, CWD <$HOME>, Specified Hosts <hostA>;
Fri Jan 20 15:08:41: Dispatched 1 Task(s) on Host(s) <hostA>, Allocated 1 Slot(
s) on Host(s) <hostA>, Effective RES_REQ <select[type == any] or
der[r15s:pg] >;
Fri Jan 20 15:08:41: Starting (Pid 30234);
Fri Jan 20 15:08:41: Running with execution home </home/mclosson>, Execution CW
D </home/mclosson>, Execution Pid <30234>;
Fri Jan 20 16:19:22: Suspended: Host load exceeded threshold: 1-minute CPU ru
n queue length (r1m)
Fri Jan 20 16:21:43: Running;
Summary of time in seconds spent in various states by Fri Jan 20 16:22:09
PEND PSUSP RUN USUSP SSUSP UNKWN TOTAL
1 0 4267 0 141 0 4409
At 16:19:22 the jobs was suspended because r1m exceeded the threshold. Later at 16:21:43 the job resumes.

Postgresql Logger Process

I'm trying to determine if Postgres 9.3 still has a logger process. It isn't referenced anywhere in the "PostgreSQL 9.3.4 Documentation". And I can't find it in my cluster's process list (see below). Also, does anyone know of a good general overview the memory structures in 9.3?
postgres 21397 1 0 20:51 pts/1 00:00:00 /opt/PostgreSQL/9.3/bin/postgres
postgres 21399 21397 0 20:51 ? 00:00:00 postgres: checkpointer process
postgres 21400 21397 0 20:51 ? 00:00:00 postgres: writer process
postgres 21401 21397 0 20:51 ? 00:00:00 postgres: wal writer process
postgres 21402 21397 0 20:51 ? 00:00:00 postgres: autovacuum launcher process
postgres 21403 21397 0 20:51 ? 00:00:00 postgres: archiver process last was 0001000004000092
postgres 21404 21397 0 20:51 ? 00:00:00 postgres: stats collector process
Thanks
Jim
Postgres has a logging collector process which is controlled through a config parameter,
logging_collector.
So in your postgresql.conf file, you would make sure this is set:
logging_collector = on
The blurb on this param from the postgres doc:
This parameter enables the logging collector, which is a background
process that captures log messages sent to stderr and redirects them
into log files. This approach is often more useful than logging to
syslog, since some types of messages might not appear in syslog
output. (One common example is dynamic-linker failure messages;
another is error messages produced by scripts such as
archive_command.) This parameter can only be set at server start.
It will show up in the process list with the following description:
postgres: logger process
For more info: http://www.postgresql.org/docs/current/static/runtime-config-logging.html
Regarding the memory structures, I'm not sure offhand, but would recommend you post that as a separate question.

PostgreSQL insert data always fail process hang up

When i want insert a row to table.
It always hangs on waiting can not commit successful.
postgres 17941 2092 0 16:42 ? 00:00:00 /usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main -c config_file=/etc/postgresql/9.1/main/postgresql.conf
postgres 17943 17941 0 16:42 ? 00:00:00 postgres: writer process
postgres 17944 17941 0 16:42 ? 00:00:00 postgres: wal writer process
postgres 17945 17941 0 16:42 ? 00:00:00 postgres: autovacuum launcher process
postgres 17946 17941 0 16:42 ? 00:00:00 postgres: stats collector process
postgres 18326 17941 0 16:51 ? 00:00:00 postgres: admin gt_development 127.0.0.1(40309) COMMIT waiting for 0/1217CA98
Who can help me?
Thanks for reply.
I have fixed by pure reinstall the postgresql 9.1