Im trying to parse a PostgreSQL log file with pgbadger. Its a custom log prefix. But when I try to parse its always showing queryes=0.
sample log:
'2020-03-07T16:09:43Z UTC [ db=dev user=rdsdb pid=16929 userid=1 xid=7382 ]' LOG: SELECT 1
'2020-03-07T16:09:43Z UTC [ db=dev user=rdsdb pid=16929 userid=1 xid=7383 ]' LOG: SET statement_timeout TO 120000
Pgbadger log:
/usr/local/bin/pgbadger -v pglog --prefix "'%t [ db=%d user=%u pid=%p userid=%p xid=%x ]'"
DEBUG: timezone not specified, using 0 seconds
DEBUG: Output 'html' reports will be written to out.html
DEBUG: Autodetected log format 'stderr' from pglog
DEBUG: pgBadger will use log format stderr to parse pglog.
DEBUG: Starting progressbar writer process
DEBUG: Processing log file: pglog
DEBUG: Starting reading file pglog...0 bytes of 205 (0.00%), queries: 0, events: 0
DEBUG: Start parsing postgresql log at offset 0 of file pglog to 205
DEBUG: the log statistics gathering took: 2 wallclock secs ( 0.00 usr + 0.01 sys = 0.01 CPU)
DEBUG: Output 'html' reports will be written to out.html
LOG: Ok, generating html report...
DEBUG: building reports took: 0 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU)
DEBUG: the total execution time took: 2 wallclock secs ( 0.01 usr + 0.01 sys = 0.02 CPU)
Related
I had created an empty database called nominatim on my DB cluster, pointed the DSN to my database with the dbname, host, port, user and pass and tried to import the map of Portugal with the following params:
nohup <path/to/file>/build/utils/setup.php \
--threads 4 \
--setup-db \
--import-data \
--create-functions \
--enable-diff-updates \
--enable-debug-statements \
--create-tables \
--create-partition-tables \
--create-partition-functions \
--import-wikipedia-articles \
--load-data \
--import-tiger-data \
--calculate-postcodes \
--index \
--index-noanalyse \
--create-search-indices \
--create-country-names \
--osm-file <path/to/file>/portugal-latest.osm.pbf >> <path/to/file>/logImportPortugal.txt 2>&1
But it is giving me the following error:
WARNING: Starting rank 25
Traceback (most recent call last):
File "/srv/nominatim/Nominatim-3.5.1/nominatim/nominatim.py", line 370, in <module>
Indexer(options).run()
File "/srv/nominatim/Nominatim-3.5.1/nominatim/nominatim.py", line 215, in run
self.index(RankRunner(self.maxrank))
File "/srv/nominatim/Nominatim-3.5.1/nominatim/nominatim.py", line 259, in index
thread = next(next_thread)
File "/srv/nominatim/Nominatim-3.5.1/nominatim/nominatim.py", line 300, in find_free_thread
if thread.is_done():
File "/srv/nominatim/Nominatim-3.5.1/nominatim/nominatim.py", line 178, in is_done
if self.conn.poll() == psycopg2.extensions.POLL_OK:
psycopg2.errors.UndefinedTable: missing FROM-clause entry for table "parent"
LINE 1: parent.place_id
^
QUERY: parent.place_id
CONTEXT: PL/pgSQL function find_parent_for_address(text,text,smallint,geometry) line 25 at RAISE
PL/pgSQL function find_parent_for_poi(character,bigint,smallint,geometry,text,text,boolean) line 31 at assignment
PL/pgSQL function placex_update() line 178 at assignment
ERROR: error status 1 running nominatim!
string(33) "error status 1 running nominatim!"
The flag --setup-db shouldn't create all the relations? What am I missing?
Below is the full log:
nohup: ignoring input
2023-01-10 13:00:49 == module path: /srv/nominatim/build/module
2023-01-10 13:00:49 == Setup DB
Postgres version found: 14
Postgis version found: 3.2
set_config
------------
(1 row)
2023-01-10 13:00:54 == Import data
osm2pgsql version 1.2.0 (64 bit id space)
Mid: loading persistent node cache from /srv/nominatim/flatnode.file
Mid: pgsql, cache=0
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Parsing gazetteer style file '/srv/nominatim/Nominatim-3.5.1/settings/import-full.style'.
Using projection SRS 4326 (Latlong)
NOTICE: table "place" does not exist, skipping
Reading in file: /home/izadmin/planet_full/portugal-latest.osm.pbf
Using PBF parser.
^MProcessing: Node(50k 50.0k/s) Way(0k 0.00k/s) Relation(0 0.00/s)^MProcessing: Node(150k 75.0k/s) Way(0k 0.00k/s) Relation(0 0.00/s)^MProcessing: Node(310k 103.3k/s) Way(0k 0.00k/s) Relation>Node stats: total(36476942), max(10423206981) in 223s
Way stats: total(3678221), max(1129408209) in 88s
Relation stats: total(75907), max(15101706) in 7s
Stopping table: planet_osm_nodes
Stopped table: planet_osm_nodes in 0s
Stopping table: planet_osm_ways
Stopped table: planet_osm_ways in 0s
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels
Stopped table: planet_osm_rels in 0s
Osm2pgsql took 319s overall
2023-01-10 13:06:14 == Create Functions
2023-01-10 13:06:14 == Create Tables
2023-01-10 13:06:14 == Create Functions
2023-01-10 13:06:14 == Create Tables
2023-01-10 13:06:15 == Create Partition Tables
2023-01-10 13:06:16 == Create Partition Functions
2023-01-10 13:06:16 == Importing wikipedia articles and redirects
2023-01-10 13:08:08 == Drop old Data
...............................................................................................................................................................................................>2023-01-10 13:08:13 == Loading word list
count
--------
318141
(1 row)
count
-------
14556
(1 row)
2023-01-10 13:09:01 == Load Data
2023-01-10 13:43:34 == Reanalysing database
Latest data imported from 2023-01-09T21:15:53Z.
2023-01-10 13:43:37 == Import Tiger data
2023-01-10 13:43:37 == Found 0 SQL files in path /srv/nominatim/Nominatim-3.5.1/data/tiger
2023-01-10 13:43:37 == WARNING: Tiger data import selected but no files found in path /srv/nominatim/Nominatim-3.5.1/data/tiger
2023-01-10 13:43:37 == Calculate Postcodes
2023-01-10 13:44:47 == Index ranks 0 - 4
'/srv/nominatim/Nominatim-3.5.1/nominatim/nominatim.py' --database nominatim --port 5432 --threads 4 -v --host '127.0.0.1' --user postgres --maxrank 4WARNING: Starting indexing rank (0 to 4) >WARNING: Starting rank 0
WARNING: Done 0/0 in 0 # 0.000 per second - FINISHED rank 0
WARNING: Starting rank 1
WARNING: Done 0/0 in 0 # 0.000 per second - FINISHED rank 1
WARNING: Starting rank 2
WARNING: Done 0/0 in 0 # 0.000 per second - FINISHED rank 2
WARNING: Starting rank 3
WARNING: Done 0/0 in 0 # 0.000 per second - FINISHED rank 3
WARNING: Starting rank 4
WARNING: Done 2/2 in 0 # 3.477 per second - FINISHED rank 4
2023-01-10 13:44:47 == Index ranks 5 - 25
WARNING: Starting indexing rank (5 to 25) using 4 threads
WARNING: Starting rank 5
WARNING: Done 0/0 in 0 # 0.000 per second - FINISHED rank 5
WARNING: Starting rank 6
WARNING: Done 0/0 in 0 # 0.000 per second - FINISHED rank 6
WARNING: Starting rank 7
WARNING: Done 0/0 in 0 # 0.000 per second - FINISHED rank 7
WARNING: Starting rank 8
WARNING: Done 2/2 in 0 # 7.341 per second - FINISHED rank 8
WARNING: Starting rank 9
WARNING: Done 0/0 in 0 # 0.000 per second - FINISHED rank 9
WARNING: Starting rank 10
WARNING: Done 0/0 in 0 # 0.000 per second - FINISHED rank 10
WARNING: Starting rank 11
WARNING: Done 0/0 in 0 # 0.000 per second - FINISHED rank 11
WARNING: Starting rank 12
WARNING: Done 37/37 in 0 # 139.460 per second - FINISHED rank 12
WARNING: Starting rank 13
WARNING: Done 0/0 in 0 # 0.000 per second - FINISHED rank 13
WARNING: Starting rank 14
INFO: Done 100 in 0 # 389.785 per second - rank 14 ETA (seconds): 0.53
WARNING: Done 308/308 in 0 # 321.383 per second - FINISHED rank 14
WARNING: Starting rank 15
WARNING: Done 1/1 in 0 # 661.813 per second - FINISHED rank 15
WARNING: Starting rank 16
INFO: Done 100 in 0 # 587.337 per second - rank 16 ETA (seconds): 5.38
INFO: Done 687 in 1 # 516.197 per second - rank 16 ETA (seconds): 4.99
INFO: Done 1203 in 2 # 507.508 per second - rank 16 ETA (seconds): 4.06
INFO: Done 1710 in 3 # 488.050 per second - rank 16 ETA (seconds): 3.18
INFO: Done 2198 in 4 # 455.463 per second - rank 16 ETA (seconds): 2.34
INFO: Done 2653 in 5 # 450.394 per second - rank 16 ETA (seconds): 1.35
INFO: Done 3103 in 7 # 425.337 per second - rank 16 ETA (seconds): 0.37
WARNING: Done 3262/3262 in 8 # 388.872 per second - FINISHED rank 16
WARNING: Starting rank 17
INFO: Done 100 in 0 # 1959.248 per second - rank 17 ETA (seconds): 0.01
WARNING: Done 127/127 in 0 # 1647.831 per second - FINISHED rank 17
WARNING: Starting rank 18
INFO: Done 100 in 0 # 1458.789 per second - rank 18 ETA (seconds): 2.29
INFO: Done 1558 in 0 # 1776.158 per second - rank 18 ETA (seconds): 1.06
INFO: Done 3334 in 1 # 1774.513 per second - rank 18 ETA (seconds): 0.06
WARNING: Done 3447/3447 in 1 # 1777.266 per second - FINISHED rank 18
WARNING: Starting rank 19
INFO: Done 100 in 0 # 3007.609 per second - rank 19 ETA (seconds): 1.98
INFO: Done 3107 in 1 # 2444.868 per second - rank 19 ETA (seconds): 1.20
INFO: Done 5551 in 2 # 2570.094 per second - rank 19 ETA (seconds): 0.19
WARNING: Done 6053/6052 in 2 # 2644.112 per second - FINISHED rank 19
WARNING: Starting rank 20
INFO: Done 100 in 0 # 2110.150 per second - rank 20 ETA (seconds): 13.73
INFO: Done 2210 in 0 # 2969.273 per second - rank 20 ETA (seconds): 9.04
INFO: Done 5179 in 2 # 2321.562 per second - rank 20 ETA (seconds): 10.29
INFO: Done 7500 in 3 # 2142.735 per second - rank 20 ETA (seconds): 10.07
INFO: Done 9642 in 4 # 2055.257 per second - rank 20 ETA (seconds): 9.45
INFO: Done 11697 in 5 # 2019.092 per second - rank 20 ETA (seconds): 8.60
INFO: Done 13716 in 6 # 1998.935 per second - rank 20 ETA (seconds): 7.68
INFO: Done 15714 in 7 # 1993.985 per second - rank 20 ETA (seconds): 6.70
INFO: Done 17707 in 8 # 1990.804 per second - rank 20 ETA (seconds): 5.71
INFO: Done 19697 in 9 # 1978.861 per second - rank 20 ETA (seconds): 4.74
INFO: Done 21675 in 11 # 1960.704 per second - rank 20 ETA (seconds): 3.77
INFO: Done 23635 in 12 # 1936.453 per second - rank 20 ETA (seconds): 2.81
INFO: Done 25571 in 13 # 1929.992 per second - rank 20 ETA (seconds): 1.81
INFO: Done 27500 in 14 # 1939.698 per second - rank 20 ETA (seconds): 0.81
WARNING: Done 29070/29067 in 14 # 1942.903 per second - FINISHED rank 20
WARNING: Starting rank 21
INFO: Done 100 in 0 # 2234.986 per second - rank 21 ETA (seconds): 0.48
WARNING: Done 1167/1167 in 0 # 1538.119 per second - FINISHED rank 21
WARNING: Starting rank 22
INFO: Done 100 in 0 # 1638.646 per second - rank 22 ETA (seconds): 18.97
INFO: Done 1738 in 0 # 1982.608 per second - rank 22 ETA (seconds): 14.85
INFO: Done 3720 in 2 # 1732.995 per second - rank 22 ETA (seconds): 15.85
INFO: Done 5452 in 3 # 1742.091 per second - rank 22 ETA (seconds): 14.77
INFO: Done 7194 in 4 # 1723.857 per second - rank 22 ETA (seconds): 13.92
INFO: Done 8917 in 5 # 1717.254 per second - rank 22 ETA (seconds): 12.97
INFO: Done 10634 in 6 # 1712.904 per second - rank 22 ETA (seconds): 12.00
INFO: Done 12346 in 7 # 1716.312 per second - rank 22 ETA (seconds): 10.98
INFO: Done 14062 in 8 # 1718.356 per second - rank 22 ETA (seconds): 9.96
INFO: Done 15780 in 9 # 1720.232 per second - rank 22 ETA (seconds): 8.95
INFO: Done 17500 in 10 # 1727.692 per second - rank 22 ETA (seconds): 7.92
INFO: Done 19227 in 11 # 1727.651 per second - rank 22 ETA (seconds): 6.92
INFO: Done 20954 in 12 # 1723.779 per second - rank 22 ETA (seconds): 5.93
INFO: Done 22677 in 13 # 1715.023 per second - rank 22 ETA (seconds): 4.96
INFO: Done 24392 in 14 # 1717.552 per second - rank 22 ETA (seconds): 3.95
INFO: Done 26109 in 15 # 1697.694 per second - rank 22 ETA (seconds): 2.99
INFO: Done 27806 in 16 # 1659.334 per second - rank 22 ETA (seconds): 2.04
INFO: Done 29465 in 17 # 1662.434 per second - rank 22 ETA (seconds): 1.03
INFO: Done 31127 in 18 # 1667.951 per second - rank 22 ETA (seconds): 0.03
WARNING: Done 31184/31183 in 18 # 1668.224 per second - FINISHED rank 22
WARNING: Starting rank 23
WARNING: Done 0/0 in 0 # 0.000 per second - FINISHED rank 23
WARNING: Starting rank 24
INFO: Done 100 in 0 # 2041.400 per second - rank 24 ETA (seconds): 2.52
INFO: Done 2141 in 1 # 1956.016 per second - rank 24 ETA (seconds): 1.59
INFO: Done 4097 in 2 # 1911.586 per second - rank 24 ETA (seconds): 0.60
WARNING: Done 5248/5245 in 2 # 1908.391 per second - FINISHED rank 24
WARNING: Starting rank 25
Traceback (most recent call last):
File "/srv/nominatim/Nominatim-3.5.1/nominatim/nominatim.py", line 370, in <module>
Indexer(options).run()
File "/srv/nominatim/Nominatim-3.5.1/nominatim/nominatim.py", line 215, in run
self.index(RankRunner(self.maxrank))
File "/srv/nominatim/Nominatim-3.5.1/nominatim/nominatim.py", line 259, in index
thread = next(next_thread)
File "/srv/nominatim/Nominatim-3.5.1/nominatim/nominatim.py", line 300, in find_free_thread
if thread.is_done():
File "/srv/nominatim/Nominatim-3.5.1/nominatim/nominatim.py", line 178, in is_done
if self.conn.poll() == psycopg2.extensions.POLL_OK:
psycopg2.errors.UndefinedTable: missing FROM-clause entry for table "parent"
LINE 1: parent.place_id
^
QUERY: parent.place_id
CONTEXT: PL/pgSQL function find_parent_for_address(text,text,smallint,geometry) line 25 at RAISE
PL/pgSQL function find_parent_for_poi(character,bigint,smallint,geometry,text,text,boolean) line 31 at assignment
PL/pgSQL function placex_update() line 178 at assignment
ERROR: error status 1 running nominatim!
string(33) "error status 1 running nominatim!"
I am using Nominatim 3.5.1
A full import of https://download.geofabrik.de/europe-latest.osm.pbf gets stuck as soon as it gets to ways.
Processing: Node(2605985k 456.3k/s) Way(638k 0.18k/s) Relation(0 0.0/s)
I have tried it with both v1.20 of osm2pgsql that comes with Nominatim v3.5.1 and also osm2pgsql 1.3.0 which is the most up to date version.
I'm running the import on a 96core machine with 50GB of RAM and 750GB SSD. It has worked previously very quickly for other regions. My --osm2pgsql-cache is set to 25000 which is enough to hold the entire pbf file in memory.
My postgres.conf is:
# Defaults from postgres db init
max_connections = 100 # (change requires restart)
dynamic_shared_memory_type = posix # the default is the first option
min_wal_size = 80MB
log_timezone = 'Etc/UTC'
datestyle = 'iso, mdy'
timezone = 'Etc/UTC'
lc_messages = 'C.UTF-8' # locale for system error message
lc_monetary = 'C.UTF-8' # locale for monetary formatting
lc_numeric = 'C.UTF-8' # locale for number formatting
lc_time = 'C.UTF-8' # locale for time formatting
# Based on https://wiki.openstreetmap.org/wiki/PostgreSQL
shared_buffers = 4GB
maintenance_work_mem = 10GB
autovacuum_work_mem = 1GB
work_mem = 256MB
checkpoint_timeout = 10min
max_wal_size = 25GB
Output of top as well:
top - 03:21:56 up 7:36, 0 users, load average: 1.10, 1.09, 1.04
Tasks: 19 total, 1 running, 18 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.0 sy, 0.0 ni, 99.7 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 87092.1 total, 30475.4 free, 27253.4 used, 29363.3 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 54874.0 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
38 postgres 20 0 4404008 4.1g 4.1g S 1.7 4.8 1:30.25 postgres
169 postgres 20 0 4392784 4.1g 4.1g D 1.3 4.8 4:41.63 postgres
165 nominat+ 20 0 27.2g 24.7g 11836 S 1.0 29.1 31:39.34 osm2pgsql
39 postgres 20 0 4391716 4.0g 4.0g S 0.3 4.7 0:23.79 postgres
1 root 20 0 8700 3572 3276 S 0.0 0.0 0:00.01 init.sh
36 postgres 20 0 4391580 127672 125776 S 0.0 0.1 0:00.97 postgres
40 postgres 20 0 4391580 21996 20080 S 0.0 0.0 1:31.52 postgres
41 postgres 20 0 4392120 8544 6328 S 0.0 0.0 0:00.33 postgres
42 postgres 20 0 71768 4808 2912 S 0.0 0.0 0:13.07 postgres
43 postgres 20 0 4392004 6920 4828 S 0.0 0.0 0:00.00 postgres
62 nominat+ 20 0 78824 25384 19972 S 0.0 0.0 0:00.03 setup.php
67 postgres 20 0 4410852 36408 32900 S 0.0 0.0 0:00.01 postgres
164 nominat+ 20 0 2612 608 540 S 0.0 0.0 0:00.00 sh
168 postgres 20 0 4410044 4.1g 4.1g S 0.0 4.8 62:42.30 postgres
171 postgres 20 0 4427500 1.9g 1.9g S 0.0 2.3 4:04.50 postgres
I am running a simulation which displays a list of parameters at every time step and displays it on the command window. It also generates a separate log file which shows all the parameters at each time step.Is there a way where i can export into a .csv file in a structured manner.There is no structure in which the output is displayed.Every row has different number of parameters.
Part of output looks like this
Time = 80.0000 sec =======================================
rmc_gr 22 DTT_pos_in: 5 Veh_Spd(mph): 1.9 fhtyi_st_pr_des: 3 ttgyda_pc_dd: 0.000
rmip_pc_dot_dd: -0.000 rmssh__thld: 0.000 rmssh_tm_tin_exit: 0.000 rmssh_gr_des: 1
tin_ne: 676 tdc_t_bar: 602 tdc_nbar: 407 tdc_n_isb_bar: 0
tfp_nerq: 7000 tcrpmbar: 408 tpm__bar: 89
r56sc_gr_t: 22 sacr_d_ele: 0 saprc_p_lnp: 180
saseq_gr: 22 sautl_gr_end: 0 sautl_gr_strt: 0 sautl_gr_end: 22
sautl_grstrt: 0 sa_slp: 0 satq_tqhld: 0.000000 sacorst: 0
I have a network program written in C using TCP sockets. Sometimes the client program hangs forever expecting input from server. Specifically, the client hangs on select() call set on an fd intended to read characters sent by server.
I am using strace to know where the process got stuck. However, sometimes when I attach the hung client process to strace, it immediately resumes it's execution and properly exits. Not all hung processes exhibit this behavior, some processes stuck in the select() even if I attach them to strace. But most of the processes resume their execution when attached to strace.
I am curious what causing the processes resume when attached to strace. It might give me clues to know why client processes are getting hung.
Any ideas? what causes a hung process to resume it's execution when attached to strace?
Update:
Here's the output of strace on hung processes.
> sudo strace -p 25645
Process 25645 attached - interrupt to quit
--- SIGSTOP (Stopped (signal)) # 0 (0) ---
--- SIGSTOP (Stopped (signal)) # 0 (0) ---
[ Process PID=25645 runs in 32 bit mode. ]
select(6, [3 5], NULL, NULL, NULL) = 2 (in [3 5])
read(5, "\0", 8192) = 1
write(2, "", 0) = 0
read(3, "====Setup set_oldtempbehaio"..., 8192) = 555
write(1, "====Setup set_oldtempbehaio"..., 555) = 555
select(6, [3 5], NULL, NULL, NULL) = 2 (in [3 5])
read(5, "", 8192) = 0
read(3, "", 8192) = 0
close(5) = 0
kill(25652, SIGKILL) = 0
exit_group(0) = ?
Process 25645 detached
_
> sudo strace -p 14462
Process 14462 attached - interrupt to quit
[ Process PID=14462 runs in 32 bit mode. ]
read(0, 0xff85fdbc, 8192) = -1 EIO (Input/output error)
shutdown(3, 1 /* send */) = 0
exit_group(0) = ?
_
> sudo strace -p 7517
Process 7517 attached - interrupt to quit
--- SIGSTOP (Stopped (signal)) # 0 (0) ---
--- SIGSTOP (Stopped (signal)) # 0 (0) ---
[ Process PID=7517 runs in 32 bit mode. ]
connect(3, {sa_family=AF_INET, sin_port=htons(300), sin_addr=inet_addr("100.64.220.98")}, 16) = -1 ETIMEDOUT (Connection timed out)
close(3) = 0
dup(2) = 3
fcntl64(3, F_GETFL) = 0x1 (flags O_WRONLY)
close(3) = 0
write(2, "dsd13: Connection timed out\n", 30) = 30
write(2, "Error code : 110\n", 17) = 17
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
exit_group(1) = ?
Process 7517 detached
Not just select(), but the processes(of same program) are stuck in various system calls before I attach them to strace. They suddenly resume after attaching to strace. If I don't attach them to strace, they just hang there forever.
Update 2:
I learned that strace could start a process which was previously stopped (process in T sate). Now I am trying to understand why did these processes go to 'T' state, what's the cause. Here's the /proc//status information:
> cat /proc/12554/status
Name: someone
State: T (stopped)
SleepAVG: 88%
Tgid: 12554
Pid: 12554
PPid: 9754
TracerPid: 0
Uid: 5000 5000 5000 5000
Gid: 48986 48986 48986 48986
FDSize: 256
Groups: 9149 48986
VmPeak: 1992 kB
VmSize: 1964 kB
VmLck: 0 kB
VmHWM: 608 kB
VmRSS: 608 kB
VmData: 156 kB
VmStk: 20 kB
VmExe: 16 kB
VmLib: 1744 kB
VmPTE: 20 kB
Threads: 1
SigQ: 54/73728
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000006
SigCgt: 0000000000004000
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
Cpus_allowed: 00000000,00000000,00000000,0000000f
Mems_allowed: 00000000,00000001
strace uses ptrace. The ptrace man page has this:
Since attaching sends SIGSTOP and the tracer usually suppresses it,
this may cause a stray EINTR return from the currently executing system
call in the tracee, as described in the "Signal injection and
suppression" section.
Are you seeing select return EINTR?
Why does it take so long to print a newline? Is this just my machine, or do others see the same effect?
With the newline:
#!/usr/bin/perl
use strict;
use Benchmark;
timethis(100000,'main();');
sub main {
print "you are the bomb. \n";
}
# outputs:
# timethis 100000: 8 wallclock secs ( 0.15 usr + 0.45 sys = 0.60 CPU) # 166666.67/s (n=100000)
W/o the newline:
#!/usr/bin/perl
use strict;
use Benchmark;
timethis(100000,'main();');
sub main {
print "you are the bomb. ";
}
# outputs:
# timethis 100000: 0 wallclock secs ( 0.09 usr + 0.04 sys = 0.13 CPU) # 769230.77/s (n=100000)
# (warning: too few iterations for a reliable count)
Edit: I'd like to add that placing two "\n" causes the execution to take
twice as long, at least for wallclock seconds.
timethis 100000: 16 wallclock secs ( 0.15 usr + 0.52 sys = 0.67 CPU) # 149253.73/s (n=100000)
I don't think buffering has much to do with it. I'm guessing it's because the terminal needs to scroll when you print a newline to it (or print enough characters to fill a line). When I benchmark these functions writing to a file or to /dev/null, there is not much of a difference.
use Benchmark;
timethis(1000000, 'main');
timethis(1000000, 'main2');
select STDERR; $| = 0; select STDOUT; # enable buffering on STDERR
sub main { print STDERR "you are the bomb. \n" }
sub main2 { print STDERR "you are the bomb. " }
$ perl benchmark.pl 2> a_file
timethis 1000000: 21 wallclock secs ( 4.67 usr + 13.38 sys = 18.05 CPU) # 55410.87/s
timethis 1000000: 21 wallclock secs ( 4.91 usr + 13.34 sys = 18.25 CPU) # 54797.52/s
$ perl benchmark.pl 2> /dev/null
timethis 1000000: 26 wallclock secs ( 2.86 usr + 10.36 sys = 13.22 CPU) # 75648.69/s
timethis 1000000: 27 wallclock secs ( 2.86 usr + 10.30 sys = 13.16 CPU) # 76010.95/s
$ perl benchmark.pl 2> a_file (without buffering)
timethis 1000000: 29 wallclock secs ( 3.78 usr + 12.14 sys = 15.92 CPU) # 62806.18/s
timethis 1000000: 29 wallclock secs ( 3.27 usr + 12.51 sys = 15.78 CPU) # 63367.34/s
$ perl benchmark.pl 2> /dev/tty (window has 35 lines and buffers 10000, YMMV)
[ 200000 declarations of how you are a bomb deleted ]
timethis 100000: 53 wallclock secs ( 0.98 usr + 3.73 sys = 4.72 CPU) # 21190.93/s
timethis 100000: 9 wallclock secs ( 0.36 usr + 1.94 sys = 2.30 CPU) # 43535.05/s
Summary: extra flushing reduces performance by about 10%. Extra scrolling on the terminal reduces performance by about 50%.
It's not the \n per se that causes this problem. Rather, successive calls to print are buffered by the OS until the \n character is encountered or the buffer is full. At that point, the output buffer is flushed to the screen. Flushing the output to the screen is a (relatively) expensive operation, so the loop in which you flush the output buffer many times has much slower performance than the loop in which you only flush the buffer once at the end (which happens implicitly when your program exits).
Newline flushes output.
In most stdio implementations, buffering varies with the type of output device ... Serial devices, including terminals, modems, mice, and joysticks, are normally line-buffered; stdio sends the entire line out only when it gets the newline