On CPU cache access - cpu-cache

I decided to experiment with the code examples from the paper What Every Programmer Should Know About Memory (PDF), and saw a different result from section 6.2.
The programs are doing matrix multiplications, I have made a little change and put them at https://github.com/herberteuler/cpumemory.
In the paper, it is said that the optimization from matrix1.c to matrix2.c, by transposing the second matrix, could save about 76.6% of cpu cycles (section 6.2.1, page 50):
Original Transposed
Cycles 16,765,297,870 3,922,373,010
Relative 100% 23.4%
But on my node, the result is very different from the above list.
This is my kernel and CPU information:
$ uname -a
Linux herberteuler 3.9-1-amd64 #1 SMP Debian 3.9.8-1 x86_64 GNU/Linux
$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Core(TM)2 Duo CPU P8600 # 2.40GHz
stepping : 10
microcode : 0xa07
cpu MHz : 2401.000
cache size : 3072 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm ida dtherm tpr_shadow vnmi flexpriority
bogomips : 4788.33
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Core(TM)2 Duo CPU P8600 # 2.40GHz
stepping : 10
microcode : 0xa07
cpu MHz : 800.000
cache size : 3072 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm ida dtherm tpr_shadow vnmi flexpriority
bogomips : 4788.33
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
And this is the result of executing matrix1 and matrix2:
$ ./matrix1
cpu cycles: 18071621964
$ ./matrix2
cpu cycles: 15716582775
Why don't I see the huge reduction of CPU cycles in matrix2, as expected?
Thanks in advance.

Related

DB2 online backup not working and can't force the application

I've started full online backup of DB2 database (primary on HADR) on NFS. The db is about 3 TB.
From the beginnig it's 0% of work completed.
And I can't find the application responsible for backup:
db2 list applications show detail | grep -i backup
I hit Ctrl+C, but it's still listed in the db2 list utilities (even after 13hrs it's 0%).
Because of the backup the tablespaces are in the 0x0800 State.
What is the resolution of this riddle? How can I cancel this backup without stopping the instance and make the tablepaces back to 0x0000 state?
I've tried to takeover HADR with Standby server to Primary, but it didn't responded, possibly because of 0x0800 states of tablespaces.
Regards
db2pd -uti
Database Member 0 -- Active -- Up 59 days 21:12:12 -- Date 2023-02-04-13.11.23.647422
Utilities:
Address ID Type State Invoker Priority StartTime DBName NumPhases CurPhase Description
0x078000000800EB40 18236 BACKUP 0 0 0 Fri Feb 3 20:05:13 DB_NAME 1 1 online db
Progress:
Address ID PhaseNum CompletedWork TotalWork StartTime Description
0x078000000800F008 18236 1 0 bytes 3203172077568 bytes Fri Feb 3 20:05:13 n/a
And the output of the db2diag.log after executing the backup:
2023-02-03-20.05.13.038004+060 E1202639A533 LEVEL: Info
PID : 31982054 TID : 14137 PROC : db2sysc 0
INSTANCE: inst_name NODE : 000 DB : DBNAME
APPHDL : 0-39477 APPID: *LOCAL.inst_name.230203190514
AUTHID : inst_name HOSTNAME: primary
EDUID : 14137 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, database utilities, sqluxGetDegreeParallelism, probe:558
DATA #1 : <preformatted>
Autonomic backup/restore - using parallelism = 10.
2023-02-03-20.05.13.824937+060 E1203173A560 LEVEL: Info
PID : 31982054 TID : 14137 PROC : db2sysc 0
INSTANCE: inst_name NODE : 000 DB : DBNAME
APPHDL : 0-39477 APPID: *LOCAL.inst_name.230203190514
AUTHID : inst_name HOSTNAME: primary
EDUID : 14137 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, database utilities, sqluxGetAvailableHeapPages, probe:634
DATA #1 : <preformatted>
Autonomic BAR - heap consumption.
Targetting (50%) - 262144 of 524288 pages.
PID : 31982054 TID : 14137 PROC : db2sysc 0
INSTANCE: inst_name NODE : 000 DB : DBNAME
APPHDL : 0-39477 APPID: *LOCAL.inst_name.230203190514
AUTHID : inst_name HOSTNAME: primary
EDUID : 14137 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, database utilities, sqluxGetDegreeParallelism, probe:558
DATA #1 : <preformatted>
Autonomic backup/restore - using parallelism = 10.
2023-02-03-20.05.13.824937+060 E1203173A560 LEVEL: Info
PID : 31982054 TID : 14137 PROC : db2sysc 0
INSTANCE: inst_name NODE : 000 DB : DBNAME
APPHDL : 0-39477 APPID: *LOCAL.inst_name.230203190514
AUTHID : inst_name HOSTNAME: primary
EDUID : 14137 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, database utilities, sqluxGetAvailableHeapPages, probe:634
DATA #1 : <preformatted>
Autonomic BAR - heap consumption.
Targetting (50%) - 262144 of 524288 pages.
2023-02-03-20.05.13.825914+060 E1203734A547 LEVEL: Info
PID : 31982054 TID : 14137 PROC : db2sysc 0
INSTANCE: inst_name NODE : 000 DB : DBNAME
APPHDL : 0-39477 APPID: *LOCAL.inst_name.230203190514
AUTHID : inst_name HOSTNAME: primary
EDUID : 14137 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, database utilities, sqlubTuneBuffers, probe:889
DATA #1 : <preformatted>
Autonomic backup - tuning enabled.
Using buffer size = 4097, number = 20.
INSTANCE: inst_name NODE : 000 DB : DBNAME
APPHDL : 0-39477 APPID: *LOCAL.inst_name.230203190514
AUTHID : inst_name HOSTNAME: primary
EDUID : 14137 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, database utilities, sqluxGetAvailableHeapPages, probe:634
DATA #1 : <preformatted>
Autonomic BAR - heap consumption.
Targetting (50%) - 262144 of 524288 pages.
2023-02-03-20.05.13.825914+060 E1203734A547 LEVEL: Info
PID : 31982054 TID : 14137 PROC : db2sysc 0
INSTANCE: inst_name NODE : 000 DB : DBNAME
APPHDL : 0-39477 APPID: *LOCAL.inst_name.230203190514
AUTHID : inst_name HOSTNAME: primary
EDUID : 14137 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, database utilities, sqlubTuneBuffers, probe:889
DATA #1 : <preformatted>
Autonomic backup - tuning enabled.
Using buffer size = 4097, number = 20.
2023-02-03-20.05.13.868295+060 E1204282A493 LEVEL: Info
PID : 31982054 TID : 14137 PROC : db2sysc 0
INSTANCE: inst_name NODE : 000 DB : DBNAME
APPHDL : 0-39477 APPID: *LOCAL.inst_name.230203190514
AUTHID : inst_name HOSTNAME: primary
EDUID : 14137 EDUNAME: db2agent (DBNAME) 0
FUNCTION: DB2 UDB, database utilities, sqlubSetupJobControl, probe:2066
MESSAGE : Starting an online db backup.
2023-02-03-23.13.48.244675+060 E1204776A430 LEVEL: Info
PID : 31982054 TID : 12081 PROC : db2sysc 0
INSTANCE: inst_name NODE : 000 DB : DBNAME
HOSTNAME: primary
EDUID : 12081 EDUNAME: db2logmgr (DBNAME) 0
FUNCTION: DB2 UDB, data protection services, sqlpgArchiveLogFile, probe:3108
DATA #1 : <preformatted>
Started archive for log file S0048345.LOG.
db2pd -db DBNAME -apinfo 39477
Database Member 0 -- Database DBANME -- Active -- Up 60 days 00:21:25 -- Date 2023-02-04-16.20.38.252022
snapapp Time: 2023-02-04-16.20.38.253042
Application :
Address : 0x0A0001001470ECE0
AppHandl [nod-index] : 39477 [000-39477]
TranHdl : 692
Application PID : 11600524
Application Node Name : primary
IP Address: n/a
Connection Start Time : 2023-02-03-20.05.13.035704
Client User ID : instname
System Auth ID : instname
Coordinator EDU ID : 14137
Coordinator Member : 0
Registered Agents : 1
Active Agents : 1
Locks timeout value : NotSet
Locks Escalation : No
Workload ID : 1
Workload Occurrence ID : 1951323
Trusted Context : n/a
Connection Trust Type : non trusted
Role Inherited : n/a
Application Status : RequestInterrupted
Application Name : db2bp
Application ID : *LOCAL.instname.230203190514
ClientUserID : n/a
ClientWrkstnName : n/a
ClientApplName : n/a
ClientAccntng : n/a
CollectActData: N
CollectActPartition: C
SectionActuals: N
UOW start time : 2023-02-03-20.05.13.037338
UOW stop time :
--------------------------------------------------------------------------------
db2pd -db DBNAME -barstats 14137
Printing out Backup Runtime Statistics at 2023-02-04-16.23.20.523322:
Backup Related EDUs:
---------------------------------------------------------------------
Backup agent ID: 14137
MC 0 (EDU ID): db2med.14137.0 (70333)
Table Spaces to be Backed Up (appTblSpace):
---------------------------------------------------------------------
numEntry: 14
Table Spaces:
tblSpaceName: SYSCATSPACE
tblSpaceID: 0
tblSpaceType: 2
tblSpaceDataType: 0
tblSpaceSize: 28420 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 28668
pageSize: 32768
extSize: 4
actualSize: 931266560
scanPages: T
backupLSN: 0000000000000000
tblSpaceName: USERSPACE1
tblSpaceID: 2
tblSpaceType: 2
tblSpaceDataType: 32
tblSpaceSize: 29457664 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 29458016
pageSize: 32768
extSize: 32
actualSize: 965268733952
scanPages: T
backupLSN: 0000000000000000
tblSpaceName: SYSTOOLSPACE
tblSpaceID: 3
tblSpaceType: 2
tblSpaceDataType: 32
tblSpaceSize: 1652 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 2044
pageSize: 32768
extSize: 4
actualSize: 54132736
scanPages: T
backupLSN: 0000000000000000
tblSpaceName: SL
tblSpaceID: 4
tblSpaceType: 2
tblSpaceDataType: 32
tblSpaceSize: 4512 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 4832
pageSize: 32768
extSize: 32
actualSize: 147849216
scanPages: T
backupLSN: 0000000000000000
tblSpaceName: MONITOR
tblSpaceID: 5
tblSpaceType: 2
tblSpaceDataType: 32
tblSpaceSize: 3712 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 4128
pageSize: 32768
extSize: 32
actualSize: 121634816
scanPages: T
backupLSN: 0000000000000000
tblSpaceName: XMLSPACE1
tblSpaceID: 6
tblSpaceType: 2
tblSpaceDataType: 32
tblSpaceSize: 57837504 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 57837600
pageSize: 32768
extSize: 32
actualSize: 1895219331072
scanPages: T
backupLSN: 0000000000000000
tblSpaceName: INDSPACE1
tblSpaceID: 7
tblSpaceType: 2
tblSpaceDataType: 32
tblSpaceSize: 32384 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 32512
pageSize: 32768
extSize: 32
actualSize: 1061158912
scanPages: T
backupLSN: 0000000000000000
tblSpaceName: ARCHSPACE1
tblSpaceID: 8
tblSpaceType: 2
tblSpaceDataType: 32
tblSpaceSize: 9817120 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 9817760
pageSize: 32768
extSize: 32
actualSize: 321687388160
scanPages: T
backupLSN: 0000000000000000
tblSpaceName: DSMSPACE
tblSpaceID: 9
tblSpaceType: 2
tblSpaceDataType: 32
tblSpaceSize: 160 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 160
pageSize: 32768
extSize: 32
actualSize: 5242880
scanPages: T
backupLSN: 0000000000000000
tblSpaceName: DLSPACE
tblSpaceID: 11
tblSpaceType: 2
tblSpaceDataType: 32
tblSpaceSize: 301504 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 302048
pageSize: 32768
extSize: 32
actualSize: 9879683072
scanPages: T
backupLSN: 0000000000000000
tblSpaceName: DLSPACECOPY
tblSpaceID: 12
tblSpaceType: 2
tblSpaceDataType: 32
tblSpaceSize: 96 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 992
pageSize: 32768
extSize: 32
actualSize: 3145728
scanPages: T
backupLSN: 0000000000000000
tblSpaceName: REPORTS
tblSpaceID: 13
tblSpaceType: 2
tblSpaceDataType: 32
tblSpaceSize: 64352 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 64352
pageSize: 32768
extSize: 32
actualSize: 2108686336
scanPages: T
backupLSN: 0000000000000000
tblSpaceName: HISTORY
tblSpaceID: 14
tblSpaceType: 2
tblSpaceDataType: 32
tblSpaceSize: 3680 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 4064
pageSize: 32768
extSize: 32
actualSize: 120586240
scanPages: T
backupLSN: 0000000000000000
tblSpaceName: DLSPACECOPY2
tblSpaceID: 15
tblSpaceType: 2
tblSpaceDataType: 32
tblSpaceSize: 161600 (in 4K pages)
nContainers: 1
backupInProgressTurnedOn: T
backupActiveIsSet: T
autoResizeLockAcquired: T
extentMovementLockAcquired: T
olbLockAcquired: F
userSpecified: T
tblSpaceTotalPages: 161760
pageSize: 32768
extSize: 32
actualSize: 5291114496
scanPages: T
backupLSN: 0000000000000000
Table Space Queue:
[0]
tblSpaceName: XMLSPACE1
tblSpaceID: 6
backupStatus: pending
[1]
tblSpaceName: USERSPACE1
tblSpaceID: 2
backupStatus: pending
[2]
tblSpaceName: ARCHSPACE1
tblSpaceID: 8
backupStatus: pending
[3]
tblSpaceName: DLSPACE
tblSpaceID: 11
backupStatus: pending
[4]
tblSpaceName: DLSPACECOPY2
tblSpaceID: 15
backupStatus: pending
[5]
tblSpaceName: REPORTS
tblSpaceID: 13
backupStatus: pending
[6]
tblSpaceName: INDSPACE1
tblSpaceID: 7
backupStatus: pending
[7]
tblSpaceName: SYSCATSPACE
tblSpaceID: 0
backupStatus: pending
[8]
tblSpaceName: SL
tblSpaceID: 4
backupStatus: pending
[9]
tblSpaceName: MONITOR
tblSpaceID: 5
backupStatus: pending
[10]
tblSpaceName: HISTORY
tblSpaceID: 14
backupStatus: pending
[11]
tblSpaceName: SYSTOOLSPACE
tblSpaceID: 3
backupStatus: pending
[12]
tblSpaceName: DSMSPACE
tblSpaceID: 9
backupStatus: pending
[13]
tblSpaceName: DLSPACECOPY
tblSpaceID: 12
backupStatus: pending
The next table space to be backed up:
[0]
tblSpaceName: XMLSPACE1
tblSpaceID: 6
backupStatus: pending
Performance statistics:
---------------------------------------------------------------------
Parallelism = 10
Number of buffers = 20
Buffer size = 16781312 (4097 4kB pages)
BM# Total I/O MsgQ WaitQ Buffers Bytes
--- -------- -------- -------- -------- -------- --------
000 40811.03 0.00 0.00 40811.00 0 0
001 40811.03 0.00 0.00 40810.96 0 0
002 40811.03 0.00 0.00 40811.00 0 0
003 40811.03 0.00 0.00 40811.00 0 0
004 40811.03 0.00 0.00 40811.00 0 0
005 40811.03 0.00 0.00 40811.00 0 0
006 40811.03 0.00 0.00 40810.98 0 0
007 40811.03 0.00 0.00 40811.00 0 0
008 40811.03 0.00 0.00 40811.00 0 0
009 40811.04 0.00 0.00 40811.00 0 0
--- -------- -------- -------- -------- -------- --------
TOT - - - - 0 0
MC# Total I/O MsgQ WaitQ Buffers Bytes
--- -------- -------- -------- -------- -------- --------
000 73085.95 0.00 0.00 0.00 0 0
--- -------- -------- -------- -------- -------- --------
TOT - - - - 0 0
Size estimates:
---------------------------------------------------------------------
Total size estimate (bytes): 3203172077568
Pre-adjusted total size estimate (bytes): 3203172077568
Init data estimate (bytes): 4645780
User data estimate (bytes): 3201899954176
End data estimate (bytes): 1073735192
Size estimate for MC1 (bytes): 3203172077568
Size estimate for remaining MCs (bytes): 3203155300352
Progress Monitor:
---------------------------------------------------------------------
Phase #: 1 of 1
Total work units (bytes): 3203172077568
Completed work units (bytes): 0
Other Backup Statistics:
---------------------------------------------------------------------
Database bufferpool flushing time: 0.00 seconds
Table space bufferpool flushing time 1: 0.00 seconds
Table space bufferpool flushing time 2: Not Recorded
Table space bufferpool flushing time 3: Not Recorded
Database recovery history file (db2rhist.asc) elapsed processing time: Not Recorded
Table space change history file (db2tschg.his) elapsed processing time: Not Recorded
No logs are included in this image.

MongoDB slow - memory usage very high

I have a 5 note replicaset mongoDB - 1 primary, 3 secondaries and 1 arbiter.
I am using mong version 4.2.3
Sizes:
“dataSize” : 688.4161271536723,
“indexes” : 177,
“indexSize” : 108.41889953613281
My Primary is very slow - each command from the shell takes a long time to return.
Memory usage seems very high, and looks like mongodb is consuming more than 50% of the RAM:
# free -lh
total used free shared buff/cache available
Mem: 188Gi 187Gi 473Mi 56Mi 740Mi 868Mi
Low: 188Gi 188Gi 473Mi
High: 0B 0B 0B
Swap: 191Gi 117Gi 74Gi
------------------------------------------------------------------
Top Memory Consuming Process Using ps command
------------------------------------------------------------------
PID PPID %MEM %CPU CMD
311 49145 97.8 498 mongod --config /etc/mongod.conf
23818 23801 0.0 3.8 /bin/prometheus --config.file=/etc/prometheus/prometheus.yml
23162 23145 0.0 8.4 /usr/bin/cadvisor -logtostderr
25796 25793 0.0 0.4 postgres: checkpointer
23501 23484 0.0 1.0 /postgres_exporter
24490 24473 0.0 0.1 grafana-server --homepath=/usr/share/grafana --config=/etc/grafana/grafana.ini --packaging=docker cfg:default.log.mode=console
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
311 systemd+ 20 0 313.9g 184.6g 2432 S 151.7 97.9 26229:09 mongod
23818 nobody 20 0 11.3g 150084 17988 S 20.7 0.1 8523:47 prometheus
23162 root 20 0 12.7g 93948 5964 S 65.5 0.0 18702:22 cadvisor
serverStatus memeory shows this:
octopusrs0:PRIMARY> db.serverStatus().mem
{
"bits" : 64,
"resident" : 189097,
"virtual" : 321404,
"supported" : true
}
octopusrs0:PRIMARY> db.serverStatus().tcmalloc.tcmalloc.formattedString
------------------------------------------------
MALLOC: 218206510816 (208097.9 MiB) Bytes in use by application
MALLOC: + 96926863360 (92436.7 MiB) Bytes in page heap freelist
MALLOC: + 3944588576 ( 3761.9 MiB) Bytes in central cache freelist
MALLOC: + 134144 ( 0.1 MiB) Bytes in transfer cache freelist
MALLOC: + 713330688 ( 680.3 MiB) Bytes in thread cache freelists
MALLOC: + 1200750592 ( 1145.1 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 320992178176 (306122.0 MiB) Actual memory used (physical + swap)
MALLOC: + 13979086848 (13331.5 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 334971265024 (319453.5 MiB) Virtual address space used
MALLOC:
MALLOC: 9420092 Spans in use
MALLOC: 234 Thread heaps in use
MALLOC: 4096 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
How can I detrmine what is causing this high memory consumption, and what can I do to return to nornal memory consumption?
Thanks,
Tamar

Netbeans 12.3 Can't See Javadoc Despite URL Set?

I'm using Netbeans 12.3 (Manjaro Linux, system specs later in the post), but can't seem to get Javadoc to appear for core classes.
I have the URL set in platform manager, but a clean & build or generate javadoc does nothing.
Here is the Netbeans readout:
System Specs:
System:
Host: MidnightStarSign Kernel: 5.10.34-1-MANJARO x86_64 bits: 64 compiler: gcc
v: 10.2.0 Desktop: KDE Plasma 5.21.4 Distro: Manjaro Linux base: Arch Linux
Machine:
Type: Desktop Mobo: ASUSTeK model: PRIME X570-PRO v: Rev X.0x
serial: <superuser required> UEFI: American Megatrends v: 3001
date: 12/04/2020
CPU:
Info: 16-Core model: AMD Ryzen 9 5950X bits: 64 type: MT MCP arch: Zen 3
rev: 0 cache: L2: 8 MiB
flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
bogomips: 217607
Speed: 4161 MHz min/max: 2200/3400 MHz boost: enabled Core speeds (MHz):
1: 4161 2: 4738 3: 2880 4: 2911 5: 2906 6: 2949 7: 2880 8: 2910 9: 3931
10: 4610 11: 3874 12: 3866 13: 3876 14: 4432 15: 4829 16: 3877 17: 3870
18: 2880 19: 3621 20: 2959 21: 2879 22: 2880 23: 2880 24: 3874 25: 2879
26: 3148 27: 4143 28: 4838 29: 3864 30: 3868 31: 3021 32: 3717
Graphics:
Device-1: NVIDIA GA104 [GeForce RTX 3070] vendor: ASUSTeK driver: nvidia
v: 460.73.01 bus-ID: 0a:00.0
Device-2: Microdia USB 2.0 Camera type: USB driver: snd-usb-audio,uvcvideo
bus-ID: 7-1:2
Display: x11 server: X.Org 1.20.11 driver: loaded: nvidia resolution:
1: 1920x1080 2: 1920x1080 3: 1920x1080
OpenGL: renderer: GeForce RTX 3070/PCIe/SSE2 v: 4.6.0 NVIDIA 460.73.01
direct render: Yes
Audio:
Device-1: NVIDIA vendor: ASUSTeK driver: snd_hda_intel v: kernel
bus-ID: 0a:00.1
Device-2: AMD Starship/Matisse HD Audio vendor: ASUSTeK driver: snd_hda_intel
v: kernel bus-ID: 0c:00.4
Device-3: JMTek LLC. Plugable USB Audio Device type: USB
driver: hid-generic,snd-usb-audio,usbhid bus-ID: 3-1:2
Device-4: Schiit Audio Schiit Modi 3+ type: USB driver: snd-usb-audio
bus-ID: 3-2:3
Device-5: ASUSTek ASUS AI Noise-Cancelling Mic Adapter type: USB
driver: hid-generic,snd-usb-audio,usbhid bus-ID: 5-5:3
Device-6: Microdia USB 2.0 Camera type: USB driver: snd-usb-audio,uvcvideo
bus-ID: 7-1:2
Sound Server-1: ALSA v: k5.10.34-1-MANJARO running: yes
Sound Server-2: JACK v: 0.125.0 running: no
Sound Server-3: PulseAudio v: 14.2 running: yes
Sound Server-4: PipeWire v: 0.3.26 running: yes
Network:
Device-1: Intel I211 Gigabit Network vendor: ASUSTeK driver: igb v: kernel
port: f000 bus-ID: 06:00.0
IF: enp6s0 state: up speed: 1000 Mbps duplex: full mac: 24:4b:fe:5b:08:2a
Bluetooth:
Device-1: Cambridge Silicon Radio Bluetooth Dongle (HCI mode) type: USB
driver: btusb v: 0.8 bus-ID: 3-5.3:6
Report: rfkill ID: hci0 rfk-id: 0 state: up address: see --recommends
Drives:
Local Storage: total: 3.78 TiB used: 1.82 TiB (48.1%)
ID-1: /dev/nvme0n1 vendor: Western Digital model: WDS100T3X0C-00SJG0
size: 931.51 GiB temp: 48.9 C
ID-2: /dev/nvme1n1 vendor: Western Digital model: WDS100T2B0C-00PXH0
size: 931.51 GiB temp: 41.9 C
ID-3: /dev/sda vendor: Seagate model: ST2000LM015-2E8174 size: 1.82 TiB
ID-4: /dev/sdb type: USB vendor: Samsung model: Flash Drive FIT
size: 119.51 GiB
ID-5: /dev/sdc type: USB vendor: Toshiba model: TransMemory size: 14.92 GiB
ID-6: /dev/sdd type: USB vendor: SanDisk model: Gaming Xbox 360 size: 7.48 GiB
Partition:
ID-1: / size: 767 GiB used: 745.17 GiB (97.2%) fs: btrfs dev: /dev/dm-0
mapped: luks-466d5812-64c7-4a28-bcc4-a1a5adfa9450
ID-2: /boot/efi size: 511 MiB used: 26.1 MiB (5.1%) fs: vfat
dev: /dev/nvme0n1p1
ID-3: /home size: 767 GiB used: 745.17 GiB (97.2%) fs: btrfs dev: /dev/dm-0
mapped: luks-466d5812-64c7-4a28-bcc4-a1a5adfa9450
Swap:
ID-1: swap-1 type: partition size: 64 GiB used: 3.2 MiB (0.0%) dev: /dev/dm-1
mapped: luks-81b2dc57-06f5-4471-b484-77c3a516f307
Sensors:
System Temperatures: cpu: 55.5 C mobo: 0 C gpu: nvidia temp: 40 C
Fan Speeds (RPM): N/A gpu: nvidia fan: 0%
Info:
Processes: 818 Uptime: 2d 15h 11m Memory: 62.79 GiB used: 27.82 GiB (44.3%)
Init: systemd Compilers: gcc: 10.2.0 clang: 11.1.0 Packages: 1963 Shell: Bash
v: 5.1.4 inxi: 3.3.04
How can I get the Javadoc to display properly?

Pocketsphinx french does not work on my buildroot embedded device

I try to install Pocketsphinx in French on a Buildroot embedded device. So I dowloaded the default french language model: https://sourceforge.net/projects/cmusphinx/files/Acoustic%20and%20Language%20Models/French/
I tried several times to make pocketsphinx works using command line on the embedded device, but it failed each time. I learned that it could be caused by the huge size of my dictionary (~ 100 000 words, called fr.dict), so I created a much smaller dictionary of 100 words (called fr-test.dict). But it doesn't seem to have changed anything.
I use the pocketsphinx_continuous command line. I defined a hmm, a dictionary and a language model which are pulled out from this folder :
Three differents hmm:
cmusphinx-fr-5.2
cmusphinx-fr-ptm-5.2
cmusphinx-fr-ptm-8khz-5.2
Two dictionaries:
fr.dict
fr-test.dict
And three language models
fr.lm.dmp
fr-small.lm.bin
fr-phone.lm.dmp
The first two parameters doesn't seem to change anything. However, the thrid one does change the error messages I receive.
With the first lm (fr.lm.dmp) the complete log are:
# pocketsphinx_continuous -adcdev plug:pcm.mic -inmic yes -hmm /mnt/usb/sphinx-f
rench/cmusphinx-fr-5.2 -lm /mnt/usb/sphinx-french/fr.lm.dmp -dict /mnt/usb/sphin
x-french/
INFO: pocketsphinx.c(152): Parsed model-specific feature parameters from /mnt/usb/sphinx-french/cmusphinx-fr-5.2/feat.params
Current configuration:
[NAME] [DEFLT] [VALUE]
-agc none none
-agcthresh 2.0 2.000000e+00
-allphone
-allphone_ci no no
-alpha 0.97 9.700000e-01
-ascale 20.0 2.000000e+01
-aw 1 1
-backtrace no no
-beam 1e-48 1.000000e-48
-bestpath yes yes
-bestpathlw 9.5 9.500000e+00
-ceplen 13 13
-cmn current current
-cmninit 8.0 8.0
-compallsen no no
-debug 0
-dict /mnt/usb/sphinx-french/
-dictcase no no
-dither no no
-doublebw no no
-ds 1 1
-fdict
-feat 1s_c_d_dd 1s_c_d_dd
-featparams
-fillprob 1e-8 1.000000e-08
-frate 100 100
-fsg
-fsgusealtpron yes yes
-fsgusefiller yes yes
-fwdflat yes yes
-fwdflatbeam 1e-64 1.000000e-64
-fwdflatefwid 4 4
-fwdflatlw 8.5 8.500000e+00
-fwdflatsfwin 25 25
-fwdflatwbeam 7e-29 7.000000e-29
-fwdtree yes yes
-hmm /mnt/usb/sphinx-french/cmusphinx-fr-5.2
-input_endian little little
-jsgf
-keyphrase
-kws
-kws_delay 10 10
-kws_plp 1e-1 1.000000e-01
-kws_threshold 1 1.000000e+00
-latsize 5000 5000
-lda
-ldadim 0 0
-lifter 0 22
-lm /mnt/usb/sphinx-french/fr.lm.dmp
-lmctl
-lmname
-logbase 1.0001 1.000100e+00
-logfn
-logspec no no
-lowerf 133.33334 1.300000e+02
-lpbeam 1e-40 1.000000e-40
-lponlybeam 7e-29 7.000000e-29
-lw 6.5 6.500000e+00
-maxhmmpf 30000 30000
-maxwpf -1 -1
-mdef
-mean
-mfclogdir
-min_endfr 0 0
-mixw
-mixwfloor 0.0000001 1.000000e-07
-mllr
-mmap yes yes
-ncep 13 13
-nfft 512 512
-nfilt 40 25
-nwpen 1.0 1.000000e+00
-pbeam 1e-48 1.000000e-48
-pip 1.0 1.000000e+00
-pl_beam 1e-10 1.000000e-10
-pl_pbeam 1e-10 1.000000e-10
-pl_pip 1.0 1.000000e+00
-pl_weight 3.0 3.000000e+00
-pl_window 5 5
-rawlogdir
-remove_dc no no
-remove_noise yes yes
-remove_silence yes yes
-round_filters yes yes
-samprate 16000 1.600000e+04
-seed -1 -1
-sendump
-senlogdir
-senmgau
-silprob 0.005 5.000000e-03
-smoothspec no no
-svspec
-tmat
-tmatfloor 0.0001 1.000000e-04
-topn 4 4
-topn_beam 0 0
-toprule
-transform legacy dct
-unit_area yes yes
-upperf 6855.4976 6.800000e+03
-uw 1.0 1.000000e+00
-vad_postspeech 50 50
-vad_prespeech 20 20
-vad_startspeech 10 10
-vad_threshold 2.0 2.000000e+00
-var
-varfloor 0.0001 1.000000e-04
-varnorm no no
-verbose no no
-warp_params
-warp_type inverse_linear inverse_linear
-wbeam 7e-29 7.000000e-29
-wip 0.65 6.500000e-01
-wlen 0.025625 2.562500e-02
INFO: feat.c(715): Initializing feature stream to type: '1s_c_d_dd', ceplen=13, CMN='current', VARNORM='no', AGC='none'
INFO: cmn.c(143): mean[0]= 12.00, mean[1..12]= 0.0
INFO: acmod.c(152): Reading linear feature transformation from /mnt/usb/sphinx-french/cmusphinx-fr-5.2/feature_transform
INFO: mdef.c(518): Reading model definition: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/mdef
INFO: bin_mdef.c(181): Allocating 101051 * 8 bytes (789 KiB) for CD tree
INFO: tmat.c(149): Reading HMM transition probability matrices: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/transition_matrices
INFO: acmod.c(113): Attempting to use PTM computation module
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/means
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/variances
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(304): 0 variance values floored
INFO: ptm_mgau.c(804): Number of codebooks exceeds 256: 2108
INFO: acmod.c(115): Attempting to use semi-continuous computation module
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/means
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/variances
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(304): 0 variance values floored
INFO: acmod.c(117): Falling back to general multi-stream GMM computation
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/means
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/variances
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(304): 0 variance values floored
INFO: ms_senone.c(149): Reading senone mixture weights: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/mixture_weights
INFO: ms_senone.c(200): Truncating senone logs3(pdf) values by 10 bits
INFO: ms_senone.c(207): Not transposing mixture weights in memory
INFO: ms_senone.c(268): Read mixture weights for 2108 senones: 1 features x 8 codewords
INFO: ms_senone.c(320): Mapping senones to individual codebooks
INFO: ms_mgau.c(144): The value of topn: 4
INFO: phone_loop_search.c(114): State beam -225 Phone exit beam -225 Insertion penalty 0
INFO: dict.c(320): Allocating 4099 * 20 bytes (80 KiB) for word entries
INFO: dict.c(333): Reading main dictionary: /mnt/usb/sphinx-french/
INFO: dict.c(213): Dictionary size 0, allocated 0 KiB for strings, 0 KiB for phones
INFO: dict.c(336): 0 words read
INFO: dict.c(358): Reading filler dictionary: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/noisedict
INFO: dict.c(213): Dictionary size 3, allocated 0 KiB for strings, 0 KiB for phones
INFO: dict.c(361): 3 words read
INFO: dict2pid.c(396): Building PID tables for dictionary
INFO: dict2pid.c(406): Allocating 36^3 * 2 bytes (91 KiB) for word-initial triphones
INFO: dict2pid.c(132): Allocated 15696 bytes (15 KiB) for word-final triphones
INFO: dict2pid.c(196): Allocated 15696 bytes (15 KiB) for single-phone word triphones
INFO: ngram_model_trie.c(347): Trying to read LM in trie binary format
INFO: ngram_model_trie.c(358): Header doesn't match
INFO: ngram_model_trie.c(176): Trying to read LM in arpa format
INFO: ngram_model_trie.c(69): No \data\ mark in LM file
INFO: ngram_model_trie.c(438): Trying to read LM in DMP format
INFO: ngram_model_trie.c(520): ngrams 1=62304, 2=18541132, 3=23627127
calloc(23627127,16) failed from ngrams_raw.c(278)
With the two others, I receive these logs:
# pocketsphinx_continuous -adcdev plug:pcm.mic -inmic yes -hmm /mnt/usb/sphinx-f
rench/cmusphinx-fr-5.2 -lm /mnt/usb/sphinx-french/fr-phone.lm.dmp -dict /mnt/usb
/sphinx-french/fr.dict
INFO: pocketsphinx.c(152): Parsed model-specific feature parameters from /mnt/usb/sphinx-french/cmusphinx-fr-5.2/feat.params
Current configuration:
[NAME] [DEFLT] [VALUE]
-agc none none
-agcthresh 2.0 2.000000e+00
-allphone
-allphone_ci no no
-alpha 0.97 9.700000e-01
-ascale 20.0 2.000000e+01
-aw 1 1
-backtrace no no
-beam 1e-48 1.000000e-48
-bestpath yes yes
-bestpathlw 9.5 9.500000e+00
-ceplen 13 13
-cmn current current
-cmninit 8.0 8.0
-compallsen no no
-debug 0
-dict /mnt/usb/sphinx-french/fr.dict
-dictcase no no
-dither no no
-doublebw no no
-ds 1 1
-fdict
-feat 1s_c_d_dd 1s_c_d_dd
-featparams
-fillprob 1e-8 1.000000e-08
-frate 100 100
-fsg
-fsgusealtpron yes yes
-fsgusefiller yes yes
-fwdflat yes yes
-fwdflatbeam 1e-64 1.000000e-64
-fwdflatefwid 4 4
-fwdflatlw 8.5 8.500000e+00
-fwdflatsfwin 25 25
-fwdflatwbeam 7e-29 7.000000e-29
-fwdtree yes yes
-hmm /mnt/usb/sphinx-french/cmusphinx-fr-5.2
-input_endian little little
-jsgf
-keyphrase
-kws
-kws_delay 10 10
-kws_plp 1e-1 1.000000e-01
-kws_threshold 1 1.000000e+00
-latsize 5000 5000
-lda
-ldadim 0 0
-lifter 0 22
-lm /mnt/usb/sphinx-french/fr-phone.lm.dmp
-lmctl
-lmname
-logbase 1.0001 1.000100e+00
-logfn
-logspec no no
-lowerf 133.33334 1.300000e+02
-lpbeam 1e-40 1.000000e-40
-lponlybeam 7e-29 7.000000e-29
-lw 6.5 6.500000e+00
-maxhmmpf 30000 30000
-maxwpf -1 -1
-mdef
-mean
-mfclogdir
-min_endfr 0 0
-mixw
-mixwfloor 0.0000001 1.000000e-07
-mllr
-mmap yes yes
-ncep 13 13
-nfft 512 512
-nfilt 40 25
-nwpen 1.0 1.000000e+00
-pbeam 1e-48 1.000000e-48
-pip 1.0 1.000000e+00
-pl_beam 1e-10 1.000000e-10
-pl_pbeam 1e-10 1.000000e-10
-pl_pip 1.0 1.000000e+00
-pl_weight 3.0 3.000000e+00
-pl_window 5 5
-rawlogdir
-remove_dc no no
-remove_noise yes yes
-remove_silence yes yes
-round_filters yes yes
-samprate 16000 1.600000e+04
-seed -1 -1
-sendump
-senlogdir
-senmgau
-silprob 0.005 5.000000e-03
-smoothspec no no
-svspec
-tmat
-tmatfloor 0.0001 1.000000e-04
-topn 4 4
-topn_beam 0 0
-toprule
-transform legacy dct
-unit_area yes yes
-upperf 6855.4976 6.800000e+03
-uw 1.0 1.000000e+00
-vad_postspeech 50 50
-vad_prespeech 20 20
-vad_startspeech 10 10
-vad_threshold 2.0 2.000000e+00
-var
-varfloor 0.0001 1.000000e-04
-varnorm no no
-verbose no no
-warp_params
-warp_type inverse_linear inverse_linear
-wbeam 7e-29 7.000000e-29
-wip 0.65 6.500000e-01
-wlen 0.025625 2.562500e-02
INFO: feat.c(715): Initializing feature stream to type: '1s_c_d_dd', ceplen=13, CMN='current', VARNORM='no', AGC='none'
INFO: cmn.c(143): mean[0]= 12.00, mean[1..12]= 0.0
INFO: acmod.c(152): Reading linear feature transformation from /mnt/usb/sphinx-french/cmusphinx-fr-5.2/feature_transform
INFO: mdef.c(518): Reading model definition: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/mdef
INFO: bin_mdef.c(181): Allocating 101051 * 8 bytes (789 KiB) for CD tree
INFO: tmat.c(149): Reading HMM transition probability matrices: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/transition_matrices
INFO: acmod.c(113): Attempting to use PTM computation module
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/means
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/variances
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(304): 0 variance values floored
INFO: ptm_mgau.c(804): Number of codebooks exceeds 256: 2108
INFO: acmod.c(115): Attempting to use semi-continuous computation module
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/means
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/variances
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(304): 0 variance values floored
INFO: acmod.c(117): Falling back to general multi-stream GMM computation
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/means
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/variances
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(304): 0 variance values floored
INFO: ms_senone.c(149): Reading senone mixture weights: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/mixture_weights
INFO: ms_senone.c(200): Truncating senone logs3(pdf) values by 10 bits
INFO: ms_senone.c(207): Not transposing mixture weights in memory
INFO: ms_senone.c(268): Read mixture weights for 2108 senones: 1 features x 8 codewords
INFO: ms_senone.c(320): Mapping senones to individual codebooks
INFO: ms_mgau.c(144): The value of topn: 4
INFO: phone_loop_search.c(114): State beam -225 Phone exit beam -225 Insertion penalty 0
INFO: dict.c(320): Allocating 109102 * 20 bytes (2130 KiB) for word entries
INFO: dict.c(333): Reading main dictionary: /mnt/usb/sphinx-french/fr.dict
INFO: dict.c(213): Dictionary size 105003, allocated 1018 KiB for strings, 1375 KiB for phones
INFO: dict.c(336): 105003 words read
INFO: dict.c(358): Reading filler dictionary: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/noisedict
INFO: dict.c(213): Dictionary size 105006, allocated 0 KiB for strings, 0 KiB for phones
INFO: dict.c(361): 3 words read
INFO: dict2pid.c(396): Building PID tables for dictionary
INFO: dict2pid.c(406): Allocating 36^3 * 2 bytes (91 KiB) for word-initial triphones
INFO: dict2pid.c(132): Allocated 15696 bytes (15 KiB) for word-final triphones
INFO: dict2pid.c(196): Allocated 15696 bytes (15 KiB) for single-phone word triphones
INFO: ngram_model_trie.c(347): Trying to read LM in trie binary format
INFO: ngram_model_trie.c(358): Header doesn't match
INFO: ngram_model_trie.c(176): Trying to read LM in arpa format
INFO: ngram_model_trie.c(69): No \data\ mark in LM file
INFO: ngram_model_trie.c(438): Trying to read LM in DMP format
INFO: ngram_model_trie.c(520): ngrams 1=38, 2=1240, 3=23231
INFO: lm_trie.c(473): Training quantizer
INFO: lm_trie.c(481): Building LM trie
INFO: ngram_search_fwdtree.c(74): Initializing search tree
INFO: ngram_search_fwdtree.c(101): 742 unique initial diphones
INFO: ngram_search_fwdtree.c(186): Creating search channels
INFO: ngram_search_fwdtree.c(323): Max nonroot chan increased to 136
INFO: ngram_search_fwdtree.c(333): Created 12 root, 8 non-root channels, 14 single-phone words
INFO: ngram_search_fwdflat.c(157): fwdflat: min_ef_width = 4, max_sf_win = 25
INFO: continuous.c(307): pocketsphinx_continuous COMPILED ON: Apr 18 2019, AT: 18:26:09
INFO: continuous.c(252): Ready....
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
A new "Input overrun" log appear every second. And remember these results does not change with another hmm or a smaller dictionary.
Any idea what is happening and how I can solve the problem ?
I will give to more information here:
1) When I use the "top" bash command to see the consumption of Pocketsphinx while running it, it return the following:
Mem: 292420K used, 717876K free, 552K shrd, 46032K buff, 112032K cached
CPU: 0% usr 3% sys 0% nic 95% idle 0% io 0% irq 0% sirq
Load average: 0.20 0.64 0.69 1/166 2377
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
2061 1 root S 16520 2% 3% sndnrj -d plug:aec
2130 2099 root R 76496 8% 0% /usr/bin/ws-proxyd -i lo -p 7894 -D /u
2371 2100 root S 33428 3% 0% pocketsphinx_continuous -adcdev plug:p
2271 2099 root S 19068 2% 0% /usr/bin/ws-proxyd -i lo -p 7895 -D /u
2276 2099 root S 99m 10% 0% /usr/bin/node /opt/app/index.js --no-u
1868 1 root S 1992 0% 0% /sbin/klogd -n
1912 2 root SW 0 0% 0% [ksdioirqd/sdio]
2104 2099 root S 22140 2% 0% /usr/bin/storaged -v 0
2073 1 root S 11796 1% 0% sc-am /etc/sc-am.cfg
2269 2099 root S 9952 1% 0% /usr/bin/sc-directive -c /flash/etc/di
2103 2099 root S 8028 1% 0% /usr/bin/sc_net
2099 1 root S 7448 1% 0% nsm
2114 2099 root S 7128 1% 0% /usr/bin/sc-led-matrix -f /dev/i2c-2
2102 2099 root S 5144 1% 0% /usr/bin/sysinfod -c /etc/sysinfod/sys
2119 1 root S 5096 1% 0% /usr/sbin/wpa_supplicant -u
2092 2091 www-data S 5064 0% 0% nginx: worker process
2065 1 root S 4972 0% 0% pupd -d /dev/spidev32766.0 -f /usr/sha
2091 1 root S 4932 0% 0% nginx: master process /usr/sbin/nginx
2253 1 root S 3860 0% 0% /usr/libexec/bluetooth/bluetoothd
So I don't think my problem is the CPU.
2) When I use the lm fr-small.lm.bin with the small dictionary of 100 words, it returned the following:
pocketsphinx_continuous -adcdev plug:pcm.mic -inmic yes -hmm /mnt/usb/sphinx-f
rench/cmusphinx-fr-5.2 -lm /mnt/usb/sphinx-french./fr-small.lm.bin -dict /mnt/us
b/sphinx-french/fr-test.dict
INFO: pocketsphinx.c(152): Parsed model-specific feature parameters from /mnt/usb/sphinx-french/cmusphinx-fr-5.2/feat.params
Current configuration:
[NAME] [DEFLT] [VALUE]
-agc none none
-agcthresh 2.0 2.000000e+00
-allphone
-allphone_ci no no
-alpha 0.97 9.700000e-01
-ascale 20.0 2.000000e+01
-aw 1 1
-backtrace no no
-beam 1e-48 1.000000e-48
-bestpath yes yes
-bestpathlw 9.5 9.500000e+00
-ceplen 13 13
-cmn current current
-cmninit 8.0 8.0
-compallsen no no
-debug 0
-dict /mnt/usb/sphinx-french/fr-test.dict
-dictcase no no
-dither no no
-doublebw no no
-ds 1 1
-fdict
-feat 1s_c_d_dd 1s_c_d_dd
-featparams
-fillprob 1e-8 1.000000e-08
-frate 100 100
-fsg
-fsgusealtpron yes yes
-fsgusefiller yes yes
-fwdflat yes yes
-fwdflatbeam 1e-64 1.000000e-64
-fwdflatefwid 4 4
-fwdflatlw 8.5 8.500000e+00
-fwdflatsfwin 25 25
-fwdflatwbeam 7e-29 7.000000e-29
-fwdtree yes yes
-hmm /mnt/usb/sphinx-french/cmusphinx-fr-5.2
-input_endian little little
-jsgf
-keyphrase
-kws
-kws_delay 10 10
-kws_plp 1e-1 1.000000e-01
-kws_threshold 1 1.000000e+00
-latsize 5000 5000
-lda
-ldadim 0 0
-lifter 0 22
-lm /mnt/usb/sphinx-french./fr-small.lm.bin
-lmctl
-lmname
-logbase 1.0001 1.000100e+00
-logfn
-logspec no no
-lowerf 133.33334 1.300000e+02
-lpbeam 1e-40 1.000000e-40
-lponlybeam 7e-29 7.000000e-29
-lw 6.5 6.500000e+00
-maxhmmpf 30000 30000
-maxwpf -1 -1
-mdef
-mean
-mfclogdir
-min_endfr 0 0
-mixw
-mixwfloor 0.0000001 1.000000e-07
-mllr
-mmap yes yes
-ncep 13 13
-nfft 512 512
-nfilt 40 25
-nwpen 1.0 1.000000e+00
-pbeam 1e-48 1.000000e-48
-pip 1.0 1.000000e+00
-pl_beam 1e-10 1.000000e-10
-pl_pbeam 1e-10 1.000000e-10
-pl_pip 1.0 1.000000e+00
-pl_weight 3.0 3.000000e+00
-pl_window 5 5
-rawlogdir
-remove_dc no no
-remove_noise yes yes
-remove_silence yes yes
-round_filters yes yes
-samprate 16000 1.600000e+04
-seed -1 -1
-sendump
-senlogdir
-senmgau
-silprob 0.005 5.000000e-03
-smoothspec no no
-svspec
-tmat
-tmatfloor 0.0001 1.000000e-04
-topn 4 4
-topn_beam 0 0
-toprule
-transform legacy dct
-unit_area yes yes
-upperf 6855.4976 6.800000e+03
-uw 1.0 1.000000e+00
-vad_postspeech 50 50
-vad_prespeech 20 20
-vad_startspeech 10 10
-vad_threshold 2.0 2.000000e+00
-var
-varfloor 0.0001 1.000000e-04
-varnorm no no
-verbose no no
-warp_params
-warp_type inverse_linear inverse_linear
-wbeam 7e-29 7.000000e-29
-wip 0.65 6.500000e-01
-wlen 0.025625 2.562500e-02
INFO: feat.c(715): Initializing feature stream to type: '1s_c_d_dd', ceplen=13, CMN='current', VARNORM='no', AGC='none'
INFO: cmn.c(143): mean[0]= 12.00, mean[1..12]= 0.0
INFO: acmod.c(152): Reading linear feature transformation from /mnt/usb/sphinx-french/cmusphinx-fr-5.2/feature_transform
INFO: mdef.c(518): Reading model definition: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/mdef
INFO: bin_mdef.c(181): Allocating 101051 * 8 bytes (789 KiB) for CD tree
INFO: tmat.c(149): Reading HMM transition probability matrices: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/transition_matrices
INFO: acmod.c(113): Attempting to use PTM computation module
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/means
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/variances
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(304): 0 variance values floored
INFO: ptm_mgau.c(804): Number of codebooks exceeds 256: 2108
INFO: acmod.c(115): Attempting to use semi-continuous computation module
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/means
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/variances
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(304): 0 variance values floored
INFO: acmod.c(117): Falling back to general multi-stream GMM computation
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/means
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(127): Reading mixture gaussian parameter: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/variances
INFO: ms_gauden.c(242): 2108 codebook, 1 feature, size:
INFO: ms_gauden.c(244): 8x32
INFO: ms_gauden.c(304): 0 variance values floored
INFO: ms_senone.c(149): Reading senone mixture weights: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/mixture_weights
INFO: ms_senone.c(200): Truncating senone logs3(pdf) values by 10 bits
INFO: ms_senone.c(207): Not transposing mixture weights in memory
INFO: ms_senone.c(268): Read mixture weights for 2108 senones: 1 features x 8 codewords
INFO: ms_senone.c(320): Mapping senones to individual codebooks
INFO: ms_mgau.c(144): The value of topn: 4
INFO: phone_loop_search.c(114): State beam -225 Phone exit beam -225 Insertion penalty 0
INFO: dict.c(320): Allocating 4199 * 20 bytes (82 KiB) for word entries
INFO: dict.c(333): Reading main dictionary: /mnt/usb/sphinx-french/fr-test.dict
INFO: dict.c(213): Dictionary size 100, allocated 0 KiB for strings, 0 KiB for phones
INFO: dict.c(336): 100 words read
INFO: dict.c(358): Reading filler dictionary: /mnt/usb/sphinx-french/cmusphinx-fr-5.2/noisedict
INFO: dict.c(213): Dictionary size 103, allocated 0 KiB for strings, 0 KiB for phones
INFO: dict.c(361): 3 words read
INFO: dict2pid.c(396): Building PID tables for dictionary
INFO: dict2pid.c(406): Allocating 36^3 * 2 bytes (91 KiB) for word-initial triphones
INFO: dict2pid.c(132): Allocated 15696 bytes (15 KiB) for word-final triphones
INFO: dict2pid.c(196): Allocated 15696 bytes (15 KiB) for single-phone word triphones
INFO: ngram_model_trie.c(347): Trying to read LM in trie binary format
INFO: ngram_search_fwdtree.c(74): Initializing search tree
INFO: ngram_search_fwdtree.c(101): 35 unique initial diphones
INFO: ngram_search_fwdtree.c(186): Creating search channels
INFO: ngram_search_fwdtree.c(323): Max nonroot chan increased to 157
INFO: ngram_search_fwdtree.c(333): Created 2 root, 29 non-root channels, 4 single-phone words
INFO: ngram_search_fwdflat.c(157): fwdflat: min_ef_width = 4, max_sf_win = 25
INFO: continuous.c(307): pocketsphinx_continuous COMPILED ON: Apr 18 2019, AT: 18:26:09
INFO: continuous.c(252): Ready....
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)
Input overrun, read calls are too rare (non-fatal)

MT7621 Soc Crypto Engine - IRQ not mapped

I am using the latest Openwrt trunk firmware (kernel 4.3) and have successfully compiled the driver for its CryptoEngine, an internal ipsec accelerator of MT7621 Soc (which as far as I understood is on an internal bus called AMBA / APB).
The driver seems to work (CryptoEngine gets configured successfully by sending and receiving packets, so it does "react" on memory mapped registers).
However the driver is unable to hook up the irq (so it works only in polling mode), because request_irq always fails with error code 89 (EDESTADDRREQ Destination address required) on IRQ 19 (driver log does not give any error before) and I can't understand why (I am not a linux device driver expert, just a basic understanding, let's say that I managed to recompile the driver).
Any idea why this happens ? Could it be a problem with the board dts file ? Here follows the dmesg log and a link to the dts files used (Dmesg Log too long to include, Soc dtsi (dev.openwrt.org/browser/trunk/target/linux/ramips/dts/mt7621.dtsi) and Board dts))
CryptoEngine memory map 0x1E004000-0x1E004FFF
root#OpenWrt:/# lspci
00:00.0 PCI bridge: Device 0e8d:0801 (rev 01)
00:01.0 PCI bridge: Device 0e8d:0801 (rev 01)
00:02.0 PCI bridge: Device 0e8d:0801 (rev 01)
01:00.0 Network controller: MEDIATEK Corp. MT7662E 802.11ac PCI Express Wireless Network Adapter
02:00.0 Network controller: MEDIATEK Corp. MT7662E 802.11ac PCI Express Wireless Network Adapter
03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
root#OpenWrt:/# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
8: 63126 63253 63143 63987 MIPS GIC Local 1 timer
10: 1513 0 0 0 MIPS GIC 10 1e100000.ethernet
11: 2 0 0 0 MIPS GIC 11 mt76pci
29: 0 0 0 0 MIPS GIC 29 xhci-hcd:usb1
31: 2 0 0 0 MIPS GIC 31 mt76pci
33: 7976 0 0 0 MIPS GIC 33 serial
63: 1178 0 0 0 MIPS GIC 63 IPI call
64: 0 1305 0 0 MIPS GIC 64 IPI call
65: 0 0 1257 0 MIPS GIC 65 IPI call
66: 0 0 0 1087 MIPS GIC 66 IPI call
67: 144709 0 0 0 MIPS GIC 67 IPI resched
68: 0 25822 0 0 MIPS GIC 68 IPI resched
69: 0 0 97134 0 MIPS GIC 69 IPI resched
70: 0 0 0 43037 MIPS GIC 70 IPI resched
root#OpenWrt:/# cat /proc/ioports
ffffffff-ffffffff : /pcie#1e140000