Looking at a crash dump in windbg, I can see that all current threads are stalled at
> ~2k
ChildEBP RetAddr
00d2fcc8 7d4e27dc ntdll_7d600000!ZwWaitForMultipleObjects+0x15
or SingleObject variations of the same.
As a handle to the object to wait on is passed to ZwWaitForMultipleObjects, I assumed that I could work out which object it was using a variation of
!do <address>
with the right address -- but I don't know how to construct the right address. I'm assuming I need some offset from the ChildEBP?
Try these steps:
Use the "~2s" command to switch context to thread #2 (this step is arguably redundant but I find it easier to operate in the right thread context)
Use the "kb" command to display the thread's call stack including each function's first three arguments. You will get something like:
ChildEBP RetAddr Args to Child
0dc7fa30 768b0962 00000004 0dc7fa80 00000001 ntdll!ZwWaitForMultipleObjects+0x15
0dc7facc 73c61339 0dc7fa80 0dc7fb14 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100
Taking the above call stack as an example, you can see that the number of handles passed to ZwWaitForMultipleObjects is 4 (the value of the first argument). The address of the handle array is the second argument. In the above example the address is 0dc7fa80
Use the "dd" command to display the contents of the handle array. In the case of the above call stack, use "dd 0dc7fa80" which will give something like:
0dc7fa80 000001f0 000001f8 0000020c 000001ec
0dc7fa90 73a53c1b 00000000 0d462f70 00000001
0dc7faa0 0cf7afe0 00000003 0dc7fac8 00000004
Assuming this is a 32bit process, the handles are the first four individual DWORDs: "1f0", "1f8", "20c" and "1ec".
You can see the details of each handle using the "!handle" WinDbg extension like so:
!handle 1f0 F
The F flag will display more details about the handle, including its count and name (if it has one associated with it)
If you suspect the handles were passed in from managed code, then you'll need to load SOS or PSSCOR and use the !ClrStack command to display the managed call stack details.
Related
I implemented a lua debugger using the vscode extension. However, I encountered a problem while displaying variables.
The variables shown at the breakpoint are switched according to the stack frameid. When show variables, I need to get the correct frameid.
Switching to a certain stack level for the first time, I can get the stack frameid through the args of scopesRequest(), but when I switch to another stack level, the system will not give me a notify when switching back. because args of variablesRequest() have not current frameid, which will cause debugger use wrong stack level and get wrong variable.
eg.
Stack:
Function1 luafile:line 26 frameid = 0
Function2 luafile:line 30 frameid = 1
When I switch to function2, scopesRequest() will prompt me for frameid=1.
When I switch back to function1, I can't get a notification of the stack switch. At this time, expand UpValue, the variable read by debugger is wrong.
enter image description here
I know this may sound nit-picky, but I want to know if there is a reason for these semantics.
Child to me means those are the things it will pass to the function it is about to call, however if you break on function name, then a given record will already exist and will not change. That record shows what that function received from its parent.
Lets say that I break on wmain and step in a few lines.
windbg will report:
# ChildEBP RetAddr Args to Child
00 005dfeb8 009e138b 00000001 00ac9bc0 00acfac8 ASMTests!wmain+0x19
Now those values wont change.
When I enter the next function DoBuf0, they don't change so they are not actually the args to the child, They are the args received from the parent.
Eg:
My wmain passes "buf0" as second argument to DoBuf0
# ChildEBP RetAddr Args to Child
00 005dfe0c 009e1131 005dfe34 009e218c 005dfe74 ASMTests!DoBuf0
01 005dfeb8 009e138b 00000001 00ac9bc0 00acfac8 ASMTests!wmain+0x91
0:000:x86> da 009e218c
009e218c "buf0"
They are the args the parent (wmain) passed to the child (DoBuf0), But they are NOT displayed on the Parents line, they are displayed on the childs. Wouldn't it make more sense to just call them Args from parent? In the case above args to Child implicates that DoBuf0 will be passing them to ITs child.
Thanks
Child to me means those are the things it will pass to the function it
is about to call
How could the window show things that have not happened yet? Do you expect some sort of speculative execution simulation? A "child" is something that has been called, and the call has left evidence data on the stack.
Quote from the WinDbg help
Raw args displays the first three parameters that are passed to the
function. On an x86-based processor, this display includes the first
three parameters that are passed to the function ("Args to Child").
Addrs displays various frame-related addresses. On an x86-based
processor, this display includes the base pointer for the stack frame
("ChildEBP") and the return address ("RetAddr").
I'm trying to track down who is making an allocation of a certain size, and I tried using the user mode stack trace db (gflags +ust), but due to FPO I can't see the entire stack. So instead I wanted to set a breakpoint on RtlAllocateHeap when it makes the allocation size I'm looking for. The only problem is I can't seem to find out a way to get this to work.
I initially tried using #esi since it looked like the third parameter was being passed using this register, but it doesn't appear that's always the case. So then I tried #ebp-c to give me the third parameter, but that doesn't appear to always work, so I tried #esp+14 and that didn't work either.
No matter what I do I can't seem to find a way to get this to actually fire when I want it to. It seems like this should work, but I'm guessing it is using leaf function optimizations in certain cases which is preventing me from being able to do this.
Anyone have any ideas on how to get this to work.
Heap stack trace:
0:013> !heap -p -a 0c060710
address 0c060710 found in
_HEAP # 1420000
HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
0c0606f8 09c3 0000 [00] 0c060710 04e00 - (busy)
77abb234 ntdll!RtlAllocateHeap+0x00000274
75ee404b ole32!CRetailMalloc_Alloc+0x00000016
76454557 OLEAUT32!APP_DATA::AllocCachedMem+0x00000060
7645476a OLEAUT32!SysAllocStringByteLen+0x0000003d
764547bf OLEAUT32!ErrStringCopyNoNull+0x00000016
764547e3 OLEAUT32!VariantCopy+0x0000007f
Break points tried:
bp ntdll!RtlAllocateHeap "j #esi == 0x4e00 ''; 'gc'"
bp ntdll!RtlAllocateHeap "j poi(#ebp-c) == 0x4e00 ''; 'gc'"
bp ntdll!RtlAllocateHeap "j poi(#esp+14) == 0x4e00 ''; 'gc'"
I think I have tested this:
Break on allocation size == 1303
bp ntdll!RtlAllocateHeap "j(poi(#esp+c) = 0x1303) 'k';'gc'"
Use poi(#esp+c)
At the first instruction of the function, you do not have EBP. So the structure on the stack for cdecl and stdcall is
<return address>
<First arg>
<second arg>
etc.
I set unhandled exception filter by doing:
SetUnhandledExceptionFilter(UnhandledException)
and in my UnhandledException function, I write out a minidump using MiniDumpWriteDump:
MiniDumpWriteDump(GetCurrentProcess(),
GetCurrentProcessId(), hFile,
MiniDumpNormal, excpInfo? &eInfo :
NULL, NULL, NULL);
When I look at the stack trace in WinDbg, I don't see much:
0:023> kb
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
15e7e7dc 08f00150 08f00a78 08f00000 08f00150 ntdll!ZwGetContextThread+0x12
15e7e7fc 777d301e 6f00016e 00000044 627e056a 0x8f00150
15e7e890 777d2ffa 777d2bf2 00000000 00650000 ntdll!RtlInterlockedFlushSList+0x889
15e7e894 777d2bf2 00000000 00650000 077e1b88 ntdll!RtlInterlockedFlushSList+0x865
15e7e8b4 772b14d1 08f00000 00000000 077e0c48 ntdll!RtlInterlockedFlushSList+0x45d
15e7e8d0 777ce023 077e1b88 08f00138 00000001 kernel32!HeapFree+0x14
15e7e8e8 777d48c6 777bfafa 777d419a ffffffff ntdll!RtlFreeHeap+0x7e
15e7e914 777e9ed7 ffffffff 15e7e938 15e7e944 ntdll!RtlImageNtHeader+0xe2
15e7e93c 777e9e49 00010000 00000000 08c70000 ntdll!RtlDestroyHeap+0x139
15e7e958 75d5458e 08f00000 00000000 08ccfe46 ntdll!RtlDestroyHeap+0xab
15e7e9c4 777ce023 104d9250 104d91f0 104d9250 KERNELBASE!HeapDestroy+0xe
00000000 00000000 00000000 00000000 00000000 ntdll!RtlFreeHeap+0x7e
The MiniDumpWriteDump documentation mentions that the strack trace may not be good, but I don't really understand what I should do about it:
http://msdn.microsoft.com/en-us/library/ms680360%28v=vs.85%29.aspx
Any help appreciated!
You cannot get reliable call stack from thread which is currently executing. To get call stack from the dump file, WinDbg extracts thread context record from the dump file (CONTEXT structure, which is basically snapshot of all registers for the thread). Based on the registers (specifically RIP and RSP) and symbols it can walk the stack and extract the call stack. For running thread, there is no way to get consistent CONTEXT structure, because it changes every instruction.
The link you looked at on MSDN mentioned easy way to get consistent CONTEXT for the currently executing thread. The code looks like this:
__try
{
RaiseException(0, 0, 0, 0);
}
__except (
MyStackTraceFilter(GetExceptionInformation()->ContextRecord)))
{
// do nothing in the handler
}
Here the work is done in MyStackTraceFilter -- you have to provide this function. Input parameter would be CONTEXT record you can rely on - a snapshot for current thread at specific time when exceptin is raised. You can actually write code to walk the stack inside the MyStackTraceFilter and you get nice call stack of the running thread. This might be workaround for you if you only interested in call stack.
In most cases it is possible to get a 'better' call stack from inconsistend CONTEXT structure. If you can rely that RSP/esp is more or less correct, then what you do is
dump the stack ("dds esp" for x86 or "dqs rsp" for x64)
try to guess where was the last stack frame before exception
use command 'k BasePtr StackPtr InstructionPtr' to dump the call stack
If you're calling MinidumpWriteDump from an unhandled exception filter then you should have a valid EXCEPTION_POINTERS to pass in the MINIDUMP_EXCEPTION_INFORMATION parameter. If you've done that then the minidump will contain an exception stream that contains a special CONTEXT which represents the state of the crashing thread. When you load a dump like this in WinDBG it will display the message:
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
You need to enter the .ecxr command to load that context so you can display the stack trace starting from that point.
How do I find out which thread is the owner of my Event handle in windbg:
I'm running
!handle 00003aec f
and get
Handle 00003aec
Type Event
Attributes 0
GrantedAccess 0x1f0003:
Delete,ReadControl,WriteDac,WriteOwner,Synch
QueryState,ModifyState
HandleCount 2
PointerCount 4
Name <none>
No object specific information available
back, and as there is no Name I haven't figured out how to get the owner out to prove which thread my thread is waiting on
[Edit] I must work against a dump as the original process needs to be restarted on the users machine, so can't debug a live session
The best discussion on the subject I've found so far is on this blog, but unfortunately we end up using different lock methods (I end up using WaitForMultipleObjectsEx and the description is for WaitForSingleObject), and he seems to have access to a live process
the stacktrace of my thread (the one that is blocked on something and where I'm looking for the current owner) is:
0:045> k9
ChildEBP RetAddr
1130e050 7c90e9ab ntdll!KiFastSystemCallRet
1130e054 7c8094e2 ntdll!ZwWaitForMultipleObjects+0xc
1130e0f0 79ed98fd kernel32!WaitForMultipleObjectsEx+0x12c
1130e158 79ed9889 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0x6f
1130e178 79ed9808 mscorwks!Thread::DoAppropriateAptStateWait+0x3c
1130e1fc 79ed96c4 mscorwks!Thread::DoAppropriateWaitWorker+0x13c
1130e24c 79ed9a62 mscorwks!Thread::DoAppropriateWait+0x40
1130e2a8 79e78944 mscorwks!CLREvent::WaitEx+0xf7
1130e2bc 7a162d84 mscorwks!CLREvent::Wait+0x17
1130e33c 7a02fd94 mscorwks!CRWLock::RWWaitForSingleObject+0x6d
1130e364 79ebd3af mscorwks!CRWLock::StaticAcquireWriterLock+0x12e
1130e410 00f24557 mscorwks!CRWLock::StaticAcquireWriterLockPublic+0xc9
Looking at the callstack it appears that the stack in question is using a ReaderWriterLock locking mechanism.
1130e410 00f24557 mscorwks!CRWLock::StaticAcquireWriterLockPublic+0xc9
Change to thread 9 and using sos.dll run !dso to dump out the managed ReaderWriterLock object. Then run !do on that the ReaderWriterLock object. I believe that there is an owning thread field that you can query. I will test it and see.
The old school way to determine this is to run ~*e !clrstack and examine all of the managed threads that are waiting on a readerwriter lock and then see if you can find the thread that has entered the same function but passed through the lock (ie. different offset)
Thanks,
Aaron
Note: Not sure if there is a way to link posts but this one is very similar to
How do I find the lockholder (reader) of my ReaderWriterLock in windbg
Use the !htrace command to get the thread ID. You must first, possibly at the start of the program, enable the collection of traces with !htrace -enable.
0:001> !htrace 00003aec
--------------------------------------
Handle = 0x00003aec - OPEN
Thread ID = 0x00000b48, Process ID = 0x000011e8
...
The above output is fictional, it will be different for your system. But it will give you the piece of information you need - the thread ID (0x00000b48 in my example).
I must work against a dump as the
original process needs to be restarted
on the users machine, so can't debug a
live session.
I am not 100% sure but I think this will work:
Attach to the process and run !htrace -enable
Detach from the process with qd. The executable will continue.
You can now take a dump file and use the above command - I think you will have the described results.
You can dig that out of a kernel dump.
Now, as far as kernel debugging goes, livekd from sysinternals should be sufficient but unfortunately it is only usable on a running system.
There's also a kernel mode memory acquisition tool which might be of use to take a dump with (in windbg's stead) for later inspection.
Otherwise, enabling handle tracing (!htrace -enable) and (if code unique to particular thread), the handle ownership could be concluded from a stack trace.
Here's the definitive answer I found. Never tried myself. You'll need live debugging to determine the owner though. But it's pretty quick.
http://weblogs.thinktecture.com/ingo/2006/08/who-is-blocking-that-mutex---fun-with-windbg-cdb-and-kd.html