Process Explorer has columns for CPU time (down to milliseconds) and CPU Cycles. For WinDbg I am aware of the !runaway command, also !runaway 7 for more details, but it shows CPU time only.
Are the CPU cycles also available somehow in a user mode crash dump?
What I have tried:
I looked at dt nt!_KTHREAD and I see it has a CycleTime property
ntdll!_KTHREAD
+0x000 Header : _DISPATCHER_HEADER
+0x018 CycleTime : Uint8B
I tried to query that property in a !for_each_thread, but WinDbg responds that it's available in kernel mode only.
Why do I want those CPU cycles?
I am working on a training for JetBrains dotTrace. It has an option to count CPU cycles and I'd like to explain where this cycles come from. Above kernel structure and Process Explorer is probably enough, but it would be awesome to see it live or post mortem in a user mode dump. I explain a lot of basics with WinDbg.
Following the implementation of GetProcessTimes() in ReactOS, you can see that the information is copied from the process' KPROCESS. So, indeed, it's only physically present in a dump that includes kernel memory.
C:\tw>ls -l
total 0
C:\tw>cdb -c ".dump /ma .\tw.dmp;q" calc.exe | grep writ
Dump successfully written
C:\tw>cdb -c "lm;!peb;.dump /ma .\tw1.dmp;q" calc.exe | grep writ
Dump successfully written
C:\tw>cdb -c ".ttime;q" -z tw.dmp | grep -B 3 quit
Created: Wed Apr 5 20:03:55.919 2017 ()
Kernel: 0 days 0:00:00.046
User: 0 days 0:00:00.000
quit:
C:\tw>cdb -c ".ttime;q" -z tw1.dmp | grep -B 3 quit
Created: Wed Apr 5 20:04:28.682 2017 ()
Kernel: 0 days 0:00:00.031
User: 0 days 0:00:00.000
quit:
C:\tw>
Related
I want to run rdiff-backup and then switch of the raspberrypi it was running on.
I use the following script:
#!/bin/sh
date > /home/mik/rdiff-backup.log
echo "rsync start" >> /home/mik/rdiff-backup.log
rdiff-backup -v5 --print-statistics offlinebackup#server::/srv/backup /srv/datenserverBackup/backup >> /home/mik/rdiff-backup.log 2>&1
sync
date >> /home/mik/rdiff-backup.log
echo "rdiff-backup end" >> /home/mik/rdiff-backup.log
df -h >> /home/mik/rdiff-backup.log
sync
halt
The log file looks good (for the rdiff-backup part):
Sat 12 Aug 08:20:59 UTC 2017
rsync start
Unable to import win32security module. Windows ACLs
not supported by filesystem at /srv/backup
escape_dos_devices not required by filesystem at /srv/backup
Warning: name offlinebackup not found on system, dropping ACL entry.
Further ACL entries dropped with this name will not trigger further warnings
Using rdiff-backup version 1.2.8
Executing ssh -C offlinebackup#server rdiff-backup --server
-----------------------------------------------------------------
Detected abilities for source (read only) file system:
Access control lists On
Extended attributes On
Windows access control lists Off
Case sensitivity On
Escape DOS devices Off
Escape trailing spaces Off
Mac OS X style resource forks Off
Mac OS X Finder information Off
-----------------------------------------------------------------
Unable to import win32security module. Windows ACLs
not supported by filesystem at /srv/datenserverBackup/backup/rdiff-backup-data/rdiff-backup.tmp.0
escape_dos_devices not required by filesystem at /srv/datenserverBackup/backup/rdiff-backup-data/rdiff-backup.tmp.0
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
Ownership changing On
Hard linking On
fsync() directories On
Directory inc permissions On
High-bit permissions On
Symlink permissions Off
Extended filenames On
Windows reserved filenames Off
Access control lists On
Extended attributes On
Windows access control lists Off
Case sensitivity On
Escape DOS devices Off
Escape trailing spaces Off
Mac OS X style resource forks Off
Mac OS X Finder information Off
-----------------------------------------------------------------
Backup: must_escape_dos_devices = 0
Starting increment operation /srv/backup to /srv/datenserverBackup/backup
Processing changed file .
Incrementing mirror file /srv/datenserverBackup/backup
Processing changed file abc
Incrementing mirror file /srv/datenserverBackup/backup/abc
Processing changed file abc/def
Incrementing mirror file /srv/datenserverBackup/backup/abc/def
Processing changed file abc/def/testfile.dxf
Incrementing mirror file /srv/datenserverBackup/backup/abc/def/testfile.dxf
--------------[ Session statistics ]--------------
StartTime 1502526061.00 (Sat Aug 12 08:21:01 2017)
EndTime 1502527913.72 (Sat Aug 12 08:51:53 2017)
ElapsedTime 1852.72 (30 minutes 52.72 seconds)
SourceFiles 151099
SourceFileSize 386321558216 (360 GB)
MirrorFiles 151097
MirrorFileSize 386321447731 (360 GB)
NewFiles 2
NewFileSize 110485 (108 KB)
DeletedFiles 0
DeletedFileSize 0 (0 bytes)
ChangedFiles 1
ChangedSourceSize 0 (0 bytes)
ChangedMirrorSize 0 (0 bytes)
IncrementFiles 4
IncrementFileSize 0 (0 bytes)
TotalDestinationSizeChange 110485 (108 KB)
Errors 0
--------------------------------------------------
The backup is working, but then the script ends right there.
rdiff-backup.log contains the full report of rdiff-backup. But neither the line "rdiff-backup end", nor the output of "df -h".
How can I make it ran to the end?
Thanks for your answers
I finally found a workaround, that solves my problem.
My sciprt which is called after booting from /etc/init.d is calling the other script which does the actual work (i.e. backup my data, and write the log file) as a background task.
/etc/init.d/CallAfterBoot.sh
#!/bin/sh
sleep 30
/home/me/DoBackup.sh & # '&' starts the script in background
/home/me/DoBackup.sh is the script I posted above which is now runing correctly.
Same script running as the same user now behaves differently. There's got to be some bug somewhere, however, it works for me now.
I am looking a dump file collected from production environment for high cpu usage. I ran !threadpool and !runaway command as follows
0:000> !ThreadPool
CPU utilization: 100%
Worker Thread: Total: 6 Running: 2 Idle: 4 MaxLimit: 32767 MinLimit: 4
Work Request in Queue: 0
--------------------------------------
Number of Timers: 8
--------------------------------------
Completion Port Thread:Total: 8 Free: 3 MaxFree: 8 CurrentLimit: 8 MaxLimit: 1000 MinLimit: 4
0:000> !runaway
ERROR: !runaway: extension exception 0x80004002.
"Unable to get thread times - dumps may not have time information"
I want to know what threads are consuming most cpu time but I cannot run !runaway command. Are there any other commands in sos, sosex or any other extension that could be helpful in this case?
You need a tool that adds the necessary information to the dump.
In WinDbg, the .dump command has the /mt MiniOption, which
Adds additional thread information to the minidump. This includes thread times, which can be displayed by using the !runaway extension or the .ttime (Display Thread Times) command when debugging the minidump.
(Emphasis: links in WinDbg)
The t option is included in the a option as well, so .dump /ma is fine as well.
To find out whether or not your dump has that information, use the undocumented command .dumpdebug like this:
.shell -ci ".dumpdebug" findstr "MiniDump"
If there is a line
1000 MiniDumpWithThreadInfo
the information is contained and you have a different issue. If it's not there, the time info is not available.
Most other tools I know do not provide such detailed settings, so it's more or less luck, whether this info is included or not.
On my pi after start there is no free memory, but i can not found, waht uses it:
pi#node1 ~ $ cat /proc/cpuinfo
processor : 0
model name : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS : 2.00
Features : half thumb fastmult vfp edsp java tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xb76
CPU revision : 7
Hardware : BCM2708
Revision : 0013
Serial : 00000000bf2e5e5c
pi#node1 ~ $ uname -a
Linux node1 4.0.7+ #801 PREEMPT Tue Jun 30 18:15:24 BST 2015 armv6l GNU/Linux
pi#node1 ~ $ head -n1 /etc/issue
Raspbian GNU/Linux 7 \n \l
pi#node1 ~ $ grep MemTotal /proc/meminfo
MemTotal: 493868 kB
pi#node1 ~ $ grep "model name" /proc/cpuinfo
model name : ARMv6-compatible processor rev 7 (v6l)
pi#node1 ~ $ ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -5
0.6 0.2 6244 2377 -bash
0.3 0.0 6748 2458 sort -k 1 -nr
0.3 0.0 4140 2457 ps -eo pmem,pcpu,vsize,pid,cmd
0.2 0.1 9484 2376 sshd: pi#pts/0
0.2 0.1 5600 2236 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 104:107
pi#node1 ~ $ free
total used free shared buffers cached
Mem: 493868 478364 15504 0 500 4956
-/+ buffers/cache: 472908 20960
Swap: 102396 116 102280
I am not a linux expert, but if I understand it right, there is just 15Mb free memory, but no task uses more than 0.6%. Than why is not there more free?
Memory is not exclusively allocated by Processes.
The bootloader and the init ram filesystem is stored in memory.
The kernel (could be very big) is loaded into memory.
The kernel reserve memory for it's processes. ps shows 0.0% for these system processes.
Driver allocate buffer memory
The graphics card needs memory
If you have not configured your swap space on a harddrive or SD card, it uses memory.
The network system allocates memory for unix sockets and shared memory.
100 processes with 0.1 % are 10%.
And, if you start a process and stop it not all of it memory will be released.
Try it. Show the memory usage with free. Start a process that need some memory. Stop the process and use free again. I would bet that there is more memory usage than before.
Edit
Here is an example of a pi with less memory usage. I have no problems running java on it. I have a WLAN Dongle and a original NOIR CAM installed.
I installed Raspbian Wheezy. I used a kernel that I compiled from sources:
> uname -a
Linux raspberrypi 3.18.14+ #2 PREEMPT Sun May 31 20:19:04 UTC 2015 armv6l GNU/Linux
> head -n1 /etc/issue
Raspbian GNU/Linux 7 \n \l
On this pi I can run java -version in an acceptable period of time.
time java -version
java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) Client VM (build 25.0-b70, mixed mode)
real 0m1.012s
user 0m0.800s
sys 0m0.190s
Here is my memory footprint
> free
total used free shared buffers cached
Mem: 380816 138304 242512 0 8916 96728
-/+ buffers/cache: 32660 348156
Swap: 102396 0 102396
I learnt that .tlist command in windbg dumps all the processes running in the system at the time of creating crash dump.
I would like to see the Memory Information of each process. So that it will help me to see if the system is over loaded by any specific process.
!process 0 1 will list all the processes and show memory related info for each. I issued this command using livekd and got all the processes. And here's my chrome process (which I picked out from the output):
PROCESS fffffa8007cb4200
SessionId: 1 Cid: 1158 Peb: 7efdf000 ParentCid: 0ff8
DirBase: 1b7962000 ObjectTable: fffff8a00addb010 HandleCount: 135.
Image: chrome.exe
VadRoot fffffa80090a6f80 Vads 169 Clone 0 Private 4037. Modified 3702. Locked 0.
DeviceMap 0000000000000000
Token fffff8a0091f9120
ElapsedTime 00:05:49.161
UserTime 00:00:00.000
KernelTime 00:00:00.000
QuotaPoolUsage[PagedPool] 0
QuotaPoolUsage[NonPagedPool] 0
Working Set Sizes (now,min,max) (8020, 50, 345) (32080KB, 200KB, 1380KB)
PeakWorkingSetSize 10137
VirtualSize 144 Mb
PeakVirtualSize 151 Mb
PageFaultCount 66631
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 5784
Job fffffa8009822e30
Note memory related properties such as "Working Set Sizes", "Virtual Size", etc.
ps. Works with livekd and with system memory dumps (which I believe is what livekd does).
Marc
This information is not contained in process dump. .tlist queries your current system, not the state when the dump was taken. If you can take a system dump, than you can check out processes and their memory usage, as Marc Sherman already answered.
Is there a command which can tell me whats in the Solaris run queue?
I can get a count using vmstat, but I need to know what processes/threads are in there.
The run-queue is always changing, so it's almost impossible to get the set of processes in the current run-queue.
That said, you can get an approximation by looking at the STAT (state) field of the process list from ps. When running the command below:
$ ps aux
...the if the STAT field begins with R, then the process is marked RUNNABLE by the kernel, which on most operating systems means that it is in the run-queue. Here's what a runnable process looks like on my machine:
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 78179 0.0 0.0 599828 480 s003 R+ 7:51AM 0:00.00 ps aux
On solaris, you can also use the prstat command and look at the STATE column. The value run indicates that the process is on the run-queue. (Also note that the value cpuN indicates that the process is currently running on processor N.
For example:
$ prstat -s cpu -n 5
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
13974 kincaid 888K 432K run 40 0 36:14.51 67% cpuhog/1
27354 kincaid 2216K 1928K run 31 0 314:48.51 27% server/5
14690 root 136M 46M sleep 59 0 0:00.59 2.3% Xsun/1
14797 kincaid 9192K 7496K sleep 59 0 0:00.10 0.9% dtwm/8
14851 kincaid 24M 14M sleep 48 0 0:00.03 0.3% netscape/1
Total: 97 processes, 190 lwps, load averages: 2.18, 2.15, 2.11
I was about to correct 0xfe answer when I saw you already did it. The run queue is containing theads not processes so the -L option is mandatory with the prstat command if you want to have the number of "state run" lines more or less matching the run queue. Beware that sampling artifacts will probably prevent to get accurate matches.
In any case, if you want to precisely know what processes/threads are sitting in the run queue you'd rather go the dtrace way assuming you are running Solaris 10 or newer.
The whoqueue.d script which might already been in /usr/demo/dtrace directory on your machine will be a good start:
# dtrace -s /usr/demo/dtrace/whoqueue.d
Run queue of length 1:
24349/1 (dtrace)
Run queue of length 3:
0/0 (sched)
0/0 (sched)
0/0 (sched)
Run queue of length 4:
22468/30 (java)
22468/17 (java)
22468/23 (java)
22468/10 (java)
Have a look at this page for details.