Suppose I'm broken into Kernel Debugger, during a system call or an IOCTL that started from user mode. I want to see the full stack - starting from user mode and switching to kernel mode.
I know some of the pieces:
I can list all the processes with !process 0 0, and see the threads inside them. Or, drowse using !dml_proc.
If I know the current usermode thread, I can switch to it and see the whole start: !thread <usermode_thread> 15.
But how do I tie them together? Specifically:
How do I know the current usermode thread?
Can I retrieve it programatically, so I can have a one-liner !thread <some_magic> 15?
WinDbg keeps track of the current thread in both user mode as well as kernel mode in the pseudo register #$thread, and current process in the pseudo register #$proc.
To see current thread stack: !thread #$thread 0x1f.
This is the full stack, starting from _RtlUserThreadStart up to the last api which usually would be SwapContext() if the thread ceded execution.
To see current process with all its threads' stacks: !process #$proc 0x1f
here is a complete current threads stack spat out by both commands
!proc #$proc 1f as well as !thread #$thread 1f
the first will print the callstack of all threads of a specific process
the second will print the call stack of a specific currently running thread
you can notice the wait status as RUNNING on the first line of Threads output
:\>head -2 sample.txt
kd> !process #$proc 0x17
PROCESS 84efc748 SessionId: 0 Cid: 0470 Peb: 7ffdc000 ParentCid: 01e8
:\>wc -l sample.txt
517 sample.txt
:\>grep THREAD sample.txt
THREAD 84efcb60 Cid 0470.0474 Teb: 7ffdf000 Win32Thread: ff9f1c00 WAIT: (UserRequest) UserMode Non-Alertable
THREAD 84f166b8 Cid 0470.0478 Teb: 7ffde000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Alertable
THREAD 85303820 Cid 0470.0490 Teb: 7ffd9000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
THREAD 85307bb0 Cid 0470.0494 Teb: 7ffd8000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
THREAD 853064e8 Cid 0470.0498 Teb: 7ffd7000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
THREAD 84e71d48 Cid 0470.049c Teb: 7ffd6000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
THREAD 8530eb80 Cid 0470.04a8 Teb: 7ffda000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
THREAD 85371030 Cid 0470.0538 Teb: 7ffd4000 Win32Thread: 00000000 WAIT: (WrQueue) UserMode Alertable
THREAD 8538ea08 Cid 0470.0588 Teb: 7ffad000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
THREAD 853d6310 Cid 0470.05a4 Teb: 7ffaa000 Win32Thread: 00000000 WAIT: (WrQueue) UserMode Alertable
THREAD 8540fa78 Cid 0470.06bc Teb: 7ffaf000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
THREAD 84173030 Cid 0470.0740 Teb: 7ffa6000 Win32Thread: 00000000 WAIT: (WrQueue) UserMode Non-Alertable
THREAD 84f77990 Cid 0470.0174 Teb: 7ffdd000 Win32Thread: 00000000 RUNNING on processor 0
THREAD 84f77990 Cid 0470.0174 Teb: 7ffdd000 Win32Thread: 00000000 RUNNING on processor 0
:\>grep !thread sample.txt
kd> !thread #$thread 1f
:\>grep -A 2 !thread sample.txt
kd> !thread #$thread 1f
THREAD 84f77990 Cid 0470.0174 Teb: 7ffdd000 Win32Thread: 00000000 RUNNING on processor 0
Not impersonating
:\>
the full call stack of a currently running thread broken on NtDeviceIoControlFile
kd> !thread #$thread 1f
THREAD 84f77990 Cid 0470.0174 Teb: 7ffdd000 Win32Thread: 00000000 RUNNING on processor 0
Not impersonating
DeviceMap 919b0008
Owning Process 84efc748 Image: svchost.exe
Attached Process N/A Image: N/A
Wait Start TickCount 49461 Ticks: 1 (0:00:00:00.010)
Context Switch Count 659 IdealProcessor: 0
UserTime 00:00:00.040
KernelTime 00:00:03.314
Win32 Start Address ntdll!TppWorkerThread (0x770403e7)
Stack Init 8c0e3fd0 Current 8c0e3bf8 Base 8c0e4000 Limit 8c0e1000 Call 00000000
Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
ChildEBP RetAddr
8c0e3d04 8283787a nt!NtDeviceIoControlFile
8c0e3d04 770570b4 nt!KiFastCallEntry+0x12a (FPO: [0,3] TrapFrame # 8c0e3d34)
0292f518 77055864 ntdll!KiFastSystemCallRet (FPO: [0,0,0])
0292f51c 76dd14c9 ntdll!ZwDeviceIoControlFile+0xc (FPO: [10,0,0])
0292f560 76dd15f9 NSI!NsiIoctl+0x5d (FPO: [Non-Fpo])
0292f584 76dd15c7 NSI!NsiEnumerateObjectsAllParametersEx+0x23 (FPO: [Non-Fpo])
0292f5d0 718718e2 NSI!NsiEnumerateObjectsAllParameters+0x79 (FPO: [Non-Fpo])
0292f620 71871858 ncsi!CNcsiNsiTable::AllocateAndGetTable+0x51 (FPO: [Non-Fpo])
0292f644 718717c6 ncsi!UpdateInterfaceStatsByFamily+0xb0 (FPO: [Non-Fpo])
0292f664 71871e49 ncsi!UpdateInterfaceStats+0x26 (FPO: [Non-Fpo])
0292f694 71871d4e ncsi!PassiveProbe+0x6f (FPO: [Non-Fpo])
0292f6f4 718715d5 ncsi!NcsiUpdateConnectivityStatusList+0x4a1 (FPO: [Non-Fpo])
0292f6f8 7704112c ncsi!NcsiUpdateConnectivityStatusListTimer+0x5 (FPO: [3,0,0])
0292f71c 77040842 ntdll!TppTimerpExecuteCallback+0x10f (FPO: [Non-Fpo])
0292f87c 756f3c45 ntdll!TppWorkerThread+0x572 (FPO: [Non-Fpo])
0292f888 770737f5 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
0292f8c8 770737c8 ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
0292f8e0 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])
Related
We have an old VB6 application that keeps freezing and/or crashing. We setup an easy shortcut for users to create a full dump when the program freezes.
We now have a week's worth of dumps, and most of them show a cryptic stack from the hung thread, which doesn't even include any of our functions, and the last call is to user32!NtUserSetFocus.
We're really stuck here... can anyone help?
0:000> kb
ChildEBP RetAddr Args to Child
0018d788 755eee3e 002609ac 00000001 755eed34 user32!NtUserSetFocus+0x15
0018d7a0 755962fa 00260f22 00000110 002609ac user32!MB_DlgProc+0x10a
0018d7cc 755bf9df 755eed34 00260f22 00000110 user32!InternalCallWinProc+0x23
0018d848 755bf784 00000000 755eed34 00260f22 user32!UserCallDlgProcCheckWow+0xd7
0018d898 755bf889 0160cf70 00000000 00000110 user32!DefDlgProcWorker+0xb7
0018d8b8 755962fa 00260f22 00000110 002609ac user32!DefDlgProcW+0x29
0018d8e4 75596d3a 755bf860 00260f22 00000110 user32!InternalCallWinProc+0x23
0018d95c 7559965e 00000000 76ee3d54 00260f22 user32!UserCallWinProcCheckWow+0x109
0018d9a0 755c206f 0160cf70 00000000 76ee3d54 user32!SendMessageWorker+0x581
0018da74 755bcf4b 75580000 00000008 00000000 user32!InternalCreateDialog+0xb9f
0018daac 755ef73c 75580000 1736f6a8 0010193a user32!InternalDialogBox+0xc1
0018db60 755efa18 00000030 ffffffff ffff0000 user32!SoftModalMessageBox+0x757
0018dcb8 755efc65 0018dcd0 00000000 755efbd1 user32!MessageBoxWorker+0x269
0018dd38 729af829 0018dd50 0010193a 0010193a user32!MessageBoxIndirectA+0x94
0018dd78 729af6a5 0018ddbc 00000000 00000000 msvbvm60!VBMessageBox2+0x92
0018dda0 729af9a0 729af7ce 0018ddbc 0018ddbc msvbvm60!MessageBoxPVoid+0x4b
0018ddd0 729a3d68 00000000 12aa8ef8 0018ddf8 msvbvm60!DlgEnableModeless+0x5e
0018de34 729a3db6 032807d4 002308e4 00000030 msvbvm60!_Scanint+0x13
0018de54 72a0c411 174be0e4 00000000 00000030 msvbvm60!RefMemberIDFromHxmod+0x39
0018de70 72a0c6f3 174be0e4 00000000 00000030 msvbvm60!EbShowError+0x3
0018de94 72a2497c 010831b8 00000000 00000000 msvbvm60!GetErrMsg+0x90
0018deb0 770fb6ad 0018df9c 00000000 0018dfec msvbvm60!SehUpdateStack+0x29
0018ded4 770fb67f 0018df9c 0018f6f8 0018dfec ntdll!ExecuteHandler2+0x26
0018def8 770fb620 0018df9c 0018f6f8 0018dfec ntdll!ExecuteHandler+0x24
0018df84 770b0163 0018df9c 0018dfec 0018df9c ntdll!RtlDispatchException+0x127
0018df84 74f3c42d 0018df9c 0018dfec 0018df9c ntdll!KiUserExceptionDispatcher+0xf
0018e4a0 72a10dcf c000008f 00000001 00000002 KERNELBASE!RaiseException+0x58
0018e4c0 72a0e228 010831b8 800a0061 0018e584 msvbvm60!CEnumConPnts::QueryInterface+0x34
0018e4d4 72a0e28c 010831b8 00000000 00000000 msvbvm60!BasicExcepDeferredFillIn+0x65
0018e4e4 72a0be99 00000061 11035a85 00000000 msvbvm60!BasicExcepDeferredFillIn+0xd2
00000000 00000000 00000000 00000000 00000000 msvbvm60!ValidateArray+0xb4
!runaway proves that this is indeed the hung thread.
0:000> !runaway
User Mode Time
Thread Time
0:ca8 0 days 0:01:00.325
8:13a4 0 days 0:00:00.171
6:10b4 0 days 0:00:00.062
10:1554 0 days 0:00:00.031
19:1598 0 days 0:00:00.000
18:a88 0 days 0:00:00.000
17:7b0 0 days 0:00:00.000
16:ba0 0 days 0:00:00.000
15:770 0 days 0:00:00.000
14:103c 0 days 0:00:00.000
13:16bc 0 days 0:00:00.000
12:17e4 0 days 0:00:00.000
11:1160 0 days 0:00:00.000
9:1194 0 days 0:00:00.000
7:dc8 0 days 0:00:00.000
5:1510 0 days 0:00:00.000
4:af0 0 days 0:00:00.000
3:e5c 0 days 0:00:00.000
2:10f4 0 days 0:00:00.000
1:ff8 0 days 0:00:00.000
!analyze -hang -v produces the following:
0:000> !analyze -hang -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
*** WARNING: Unable to verify checksum for mscorlib.ni.dll
GetUrlPageData2 (WinHttp) failed: 12152.
FAULTING_IP:
+0
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
CONTEXT: 00000000 -- (.cxr 0x0;r)
eax=00000001 ebx=00000000 ecx=00000000 edx=00000000 esi=0018dcd0 edi=00260f22
eip=755a218a esp=0018d788 ebp=0018d7a0 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00200246
user32!NtUserSetFocus+0x15:
755a218a 83c404 add esp,4
FAULTING_THREAD: 00000000
BUGCHECK_STR: HANG
DEFAULT_BUCKET_ID: APPLICATION_HANG
PROCESS_NAME: OurProcess.exe
ERROR_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>
EXCEPTION_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: OurProcess.exe
ANALYSIS_VERSION: 6.3.9600.16384 (debuggers(dbg).130821-1623) x86fre
MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0xca8 (0)
Current frame:
ChildEBP RetAddr Caller, Callee
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
-- --- ------- --------------------------
0 758.ca8 Unknown
WAIT_CHAIN_COMMAND: ~0s;k;;
BLOCKING_THREAD: 00000ca8
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG
LAST_CONTROL_TRANSFER: from 755eee3e to 755a218a
STACK_TEXT:
0018d788 755eee3e 002609ac 00000001 755eed34 user32!NtUserSetFocus+0x15
0018d7a0 755962fa 00260f22 00000110 002609ac user32!MB_DlgProc+0x10a
0018d7cc 755bf9df 755eed34 00260f22 00000110 user32!InternalCallWinProc+0x23
0018d848 755bf784 00000000 755eed34 00260f22 user32!UserCallDlgProcCheckWow+0xd7
0018d898 755bf889 0160cf70 00000000 00000110 user32!DefDlgProcWorker+0xb7
0018d8b8 755962fa 00260f22 00000110 002609ac user32!DefDlgProcW+0x29
0018d8e4 75596d3a 755bf860 00260f22 00000110 user32!InternalCallWinProc+0x23
0018d95c 7559965e 00000000 76ee3d54 00260f22 user32!UserCallWinProcCheckWow+0x109
0018d9a0 755c206f 0160cf70 00000000 76ee3d54 user32!SendMessageWorker+0x581
0018da74 755bcf4b 75580000 00000008 00000000 user32!InternalCreateDialog+0xb9f
0018daac 755ef73c 75580000 1736f6a8 0010193a user32!InternalDialogBox+0xc1
0018db60 755efa18 00000030 ffffffff ffff0000 user32!SoftModalMessageBox+0x757
0018dcb8 755efc65 0018dcd0 00000000 755efbd1 user32!MessageBoxWorker+0x269
0018dd38 729af829 0018dd50 0010193a 0010193a user32!MessageBoxIndirectA+0x94
0018dd78 729af6a5 0018ddbc 00000000 00000000 msvbvm60!VBMessageBox2+0x92
0018dda0 729af9a0 729af7ce 0018ddbc 0018ddbc msvbvm60!MessageBoxPVoid+0x4b
0018ddd0 729a3d68 00000000 12aa8ef8 0018ddf8 msvbvm60!DlgEnableModeless+0x5e
0018de34 729a3db6 032807d4 002308e4 00000030 msvbvm60!_Scanint+0x13
0018de54 72a0c411 174be0e4 00000000 00000030 msvbvm60!RefMemberIDFromHxmod+0x39
0018de70 72a0c6f3 174be0e4 00000000 00000030 msvbvm60!EbShowError+0x3
0018de94 72a2497c 010831b8 00000000 00000000 msvbvm60!GetErrMsg+0x90
0018deb0 770fb6ad 0018df9c 00000000 0018dfec msvbvm60!SehUpdateStack+0x29
0018ded4 770fb67f 0018df9c 0018f6f8 0018dfec ntdll!ExecuteHandler2+0x26
0018def8 770fb620 0018df9c 0018f6f8 0018dfec ntdll!ExecuteHandler+0x24
0018df84 770b0163 0018df9c 0018dfec 0018df9c ntdll!RtlDispatchException+0x127
0018df84 74f3c42d 0018df9c 0018dfec 0018df9c ntdll!KiUserExceptionDispatcher+0xf
0018e4a0 72a10dcf c000008f 00000001 00000002 KERNELBASE!RaiseException+0x58
0018e4c0 72a0e228 010831b8 800a0061 0018e584 msvbvm60!CEnumConPnts::QueryInterface+0x34
0018e4d4 72a0e28c 010831b8 00000000 00000000 msvbvm60!BasicExcepDeferredFillIn+0x65
0018e4e4 72a0be99 00000061 11035a85 00000000 msvbvm60!BasicExcepDeferredFillIn+0xd2
00000000 00000000 00000000 00000000 00000000 msvbvm60!ValidateArray+0xb4
FOLLOWUP_IP:
msvbvm60!VBMessageBox2+92
729af829 8bd8 mov ebx,eax
SYMBOL_STACK_INDEX: e
SYMBOL_NAME: msvbvm60!VBMessageBox2+92
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: msvbvm60
IMAGE_NAME: msvbvm60.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4a5bda6c
STACK_COMMAND: ~0s ; kb
BUCKET_ID: HANG_msvbvm60!VBMessageBox2+92
FAILURE_BUCKET_ID: APPLICATION_HANG_cfffffff_msvbvm60.dll!VBMessageBox2
ANALYSIS_SOURCE: UM
FAILURE_ID_HASH_STRING: um:application_hang_cfffffff_msvbvm60.dll!vbmessagebox2
FAILURE_ID_HASH: {e6dc63dc-251f-a6d5-c66e-f5e07e418955}
Followup: MachineOwner
---------
Firstly I'd find out what it's doing. When it hangs get a couple of crash dumps and compare the stacks to find out where exactly it's spinning. Also some of the symbols have large offsets (user32!InternalCreateDialog+0xb9f), are you using a symbol server?
I'm debugging a process in WinDbg, and the process exited:
0:009> g
(bunch of regs...)
ntdll!NtTerminateProcess+0xc:
770ad43c c20800 ret 8
0:009> g
^ No runnable debuggees error in 'g'
At this point, how do I get the process' exit code?
You could find it as the second argument of ZwTerminateProcess. NtTerminateProcess is just the kernel version of it, right?
0:000> kb
ChildEBP RetAddr Args to Child
003ff414 7774d5ac ffffffff 1234abcd 00000000 ntdll!ZwTerminateProcess+0x12
003ff430 759c79ec 00000000 77e8f3b0 ffffffff ntdll!RtlExitUserProcess+0x85
...
Or the fourth parameter of RtlExitUserProcess
0:000> kn
# ChildEBP RetAddr
00 003ff414 7774d5ac ntdll!ZwTerminateProcess+0x12
01 003ff430 759c79ec ntdll!RtlExitUserProcess+0x85
...
0:000> .frame 01
01 003ff430 759c79ec ntdll!RtlExitUserProcess+0x85
0:000> dd esp L4
003ff414 7771fcc2 7774d5ac ffffffff 1234abcd
After running the Crash/Hang Analyzer in the DebugDiag tool it list that thread 17 is locking.
Running a !locks command in windbg also points to thread 17.
I am up to this point...
0:000> ~17 kb
ChildEBP RetAddr Args to Child
0261fcc4 7c827b69 77e6202c 00000002 0261fd14 ntdll!KiFastSystemCallRet
0261fcc8 77e6202c 00000002 0261fd14 00000001 ntdll!ZwWaitForMultipleObjects+0xc
0261fd70 77e62fbe 00000002 7a3d0468 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
0261fd8c 79f9a9d2 00000002 7a3d0468 00000000 kernel32!WaitForMultipleObjects+0x18
0261fdac 79f9ab07 000f2ab0 0261feb0 000f32b0 mscorwks!SVR::WaitForFinalizerEvent+0x77
0261fdc0 79eac697 0261feb0 00000000 00000000 mscorwks!SVR::GCHeap::FinalizerThreadWorker+0x49
0261fdd4 79eac633 0261feb0 0261fe5c 79fb685f mscorwks!Thread::DoADCallBack+0x32a
0261fe68 79eac553 0261feb0 3aa1ce47 00000000 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0261fea4 79f4431c 0261feb0 00000000 0010f080 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
0261fecc 79f4432d 79f9aabc 00000008 0261ff14 mscorwks!ManagedThreadBase_NoADTransition+0x32
0261fedc 79f45ecb 79f9aabc 3aa1cff7 00000000 mscorwks!ManagedThreadBase::FinalizerBase+0xd
0261ff14 79fd2733 00000000 00000003 ffffffff mscorwks!SVR::GCHeap::FinalizerThreadStart+0xbb
0261ffb8 77e6482f 000f32b0 00000000 00000000 mscorwks!Thread::intermediateThreadProc+0x49
0261ffec 00000000 79fd26ed 000f32b0 00000000 kernel32!BaseThreadStart+0x34
From my understanding that the stack in thread 17 was trying to run this last
0261fcc4 7c827b69 77e6202c 00000002 0261fd14 ntdll!KiFastSystemCallRet
What do I do from here to go further on my analysis?
It’s a large 32 bits mixed mode MFC 7.0 app on XP, the user tells that he was using a feature which is implemented in managed code.
The crach is in a thread that has acquired the LoaderLock, and seems to orgin from .NET workerthread pool.
0:016> !cs -o -l
-----------------------------------------
DebugInfo = 0x7c97e1a0
Critical section = 0x7c97e174 (ntdll!LdrpLoaderLock+0x0)
LOCKED
LockCount = 0x4
OwningThread = 0x00000260
RecursionCount = 0x1
LockSemaphore = 0x7BC
SpinCount = 0x00000000
OwningThread DbgId = ~16s
OwningThread Stack =
ChildEBP RetAddr Args to Child
0f66e400 7c90df4a 7c8648a2 00000002 0f66e57c ntdll!KiFastSystemCallRet (FPO: [0,0,0])
0f66e404 7c8648a2 00000002 0f66e57c 00000001 ntdll!ZwWaitForMultipleObjects+0xc (FPO: [5,0,0])
0f66e74c 7c83ab50 0f66e774 7c839b39 0f66e77c kernel32!UnhandledExceptionFilter+0x8b9 (FPO: [Non-Fpo])
0f66e754 7c839b39 0f66e77c 00000000 0f66e77c kernel32!BaseThreadStart+0x4d (FPO: [Non-Fpo])
0f66e77c 7c9032a8 0f66e868 0f66ffdc 0f66e884 kernel32!_except_handler3+0x61 (FPO: [Uses EBP] [3,0,7])
0f66e7a0 7c90327a 0f66e868 0f66ffdc 0f66e884 ntdll!ExecuteHandler2+0x26
0f66e850 7c90e48a 00000000 0f66e884 0f66e868 ntdll!ExecuteHandler+0x24
0f66e850 79247eb4 00000000 0f66e884 0f66e868 ntdll!KiUserExceptionDispatcher+0xe (FPO: [2,0,0]) (CONTEXT # 0f66e884)
0f66eb4c 7929a46e 0e715d80 792483ef 0e715d80 mscorwks!Thread::UnhijackThread+0xb (FPO: [0,0,0])
0f66eb54 792483ef 0e715d80 00000000 00000000 mscorwks!Thread::RareEnablePreemptiveGC+0x36 (FPO: [0,0,0])
0f66eb64 792a6ff9 06ee0000 00000000 00000000 mscorwks!Thread::RareDisablePreemptiveGC+0x5f (FPO: [0,0,0])
0f66ec10 79247e14 06ee0000 00000003 00000000 mscorwks!SystemDomain::RunDllMain+0x7d (FPO: [Non-Fpo])
0f66ee98 603d6a2c 00000001 00000003 00000000 mscorwks!ExecuteDLL+0x3c0 (FPO: [Non-Fpo])
0f66eed8 603d70a3 06ee0000 0f66eebc 00000000 mscoreei!CorDllMainWorker+0x153 (FPO: [Non-Fpo])
0f66ef14 79015012 00000000 00000003 00000000 mscoreei!_CorDllMain+0x111 (FPO: [Non-Fpo])
0f66ef30 7c90118a 06ee0000 00000003 00000000 mscoree!ShellShim__CorDllMain+0xad (FPO: [Non-Fpo])
0f66ef50 7c91397b 06ef841e 06ee0000 00000003 ntdll!LdrpCallInitRoutine+0x14
0f66efc8 7c80c136 00000000 793fa180 7c80934a ntdll!LdrShutdownThread+0xd7 (FPO: [Non-Fpo])
0f66f000 792ee8ad 00000000 00000000 792ee78a kernel32!ExitThread+0x3e (FPO: [Non-Fpo])
0f66f020 792edfcb 00000000 00000000 00000000 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x123 (FPO: [Non-Fpo])
Some interesting vales on the stack might be the 06ee0000 and 0f66eebc.
The first is the base address for myMixedModeDll, and the second:
0:016> ln 06ef841e
(06ef841e) myMixedModeDll!CorDllMain | (06ef8424) myMixedModeDll!CDialog::CDialog
Exact matches:
The actual exception should be here:
0:000> .cxr 0f66e884;kb
eax=000000df ebx=00000000 ecx=0e715d80 edx=000003a4 esi=0e715d80 edi=00010000
eip=79247eb4 esp=0f66eb50 ebp=0f66ec10 iopl=0 nv up ei ng nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010282
mscorwks!Thread::UnhijackThread+0xb:
79247eb4 8910 mov dword ptr [eax],edx ds:0023:000000df=????????
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr Args to Child
0f66eb4c 7929a46e 0e715d80 792483ef 0e715d80 mscorwks!Thread::UnhijackThread+0xb
0f66eb54 792483ef 0e715d80 00000000 00000000 mscorwks!Thread::RareEnablePreemptiveGC+0x36
And yes the eax is not good:
0:000> u 79247eae
mscorwks!Thread::UnhijackThread+0x5:
79247eae 8b5178 mov edx,dword ptr [ecx+78h]
79247eb1 8b417c mov eax,dword ptr [ecx+7Ch]
79247eb4 8910 mov dword ptr [eax],edx
Yes, ECX has been restored properly
0:016> dd #ecx+0x78 L1
0e715df8 000003a4
0:016> dd #ecx+0x7c L1
0e715dfc 000000df
0:016> dd #ecx L0x20
0e715d80 0e6f4798 00000000 ffffffff 00000000
0e715d90 00000000 00000020 00000000 0e715da0
0e715da0 0e715da0 0e715da0 00000000 00000000
0e715db0 00000000 000000df 00000000 00000000
0e715dc0 00000000 00000000 00000000 00000000
0e715dd0 00000000 00000000 00000000 00000000
0e715de0 00000000 00000000 00000000 00000000
0e715df0 0e7093e8 00002733 000003a4 000000df
The last error value
0:016> !gle
LastErrorValue: (Win32) 0 (0) - The operation completed successfully.
LastStatusValue: (NTSTATUS) 0xc0000034 - Object Name not found.
This .NET is version 1.1.4322 , and the sos! Claims that thread #16 is not a managed thread.
0:016> !t
ThreadCount: 10
UnstartedThread: 0
BackgroundThread: 10
PendingThread: 0
DeadThread: 0
PreEmptive GC Alloc Lock
ID ThreadOBJ State GC Context Domain Count APT Exception
0 0xc8c 0x001ae598 0x4220 Enabled 0x1b7df804:0x1b7df8d8 0x001fda98 0 STA
5 0xcd4 0x001caea0 0xb220 Enabled 0x00000000:0x00000000 0x001fda98 0 MTA (Finalizer)
8 0xe28 0x0c56ac40 0x220 Enabled 0x00000000:0x00000000 0x001fda98 0 Ukn
10 0x8a8 0x0e5f4b48 0x800220 Enabled 0x1b822518:0x1b824458 0x001fda98 0 MTA (Threadpool Completion Port)
11 0xc18 0x0e6d6a60 0x800220 Enabled 0x1b8651cc:0x1b867008 0x001fda98 0 MTA (Threadpool Completion Port)
12 0xa54 0x00190c28 0x220 Enabled 0x1b5247f0:0x1b52650c 0x001fda98 0 Ukn
13 0xe9c 0x0e6627f8 0x220 Enabled 0x1b5307f0:0x1b53250c 0x001fda98 0 Ukn
14 0xe58 0x0e6b11a0 0x1800220 Enabled 0x00000000:0x00000000 0x001fda98 0 MTA (Threadpool Worker)
15 0x8dc 0x0e6d68a8 0x220 Enabled 0x00000000:0x00000000 0x001fda98 0 Ukn
17 0xbcc 0x0e709378 0x220 Enabled 0x1b52c7f0:0x1b52e50c 0x001fda98 0 Ukn
0:016> !ClrStack
Thread 16
Not a managed thread.
How can I find out more to reveal the cause to this crash ?
Admins told us that there is some prolem in production. They noticed big jump in memory usage and in requests waiting.
I received one crash dump. I need help analysing it.
Using Debug Diagnostic Tool I found this:
Detected possible blocking or leaked critical section at 0x1e5bd320 owned by thread 141 in dllhst3g.exe__Metastorm Process Engine__PID__7444__Date__10_25_2011__Time_01_19_15PM__686__Manual Dump.dmp
Impact of this lock
11,59% of threads blocked
(Threads 97 137 142 143 144 147 207 208 211 212 213 214 215 216 217 218 219 221 222 223 224 225 226 227 228 229 230)
The following functions are trying to enter this critical section
ACECORE+20eb
The following module(s) are involved with this critical section
C:\Program Files\Common Files\Microsoft Shared\OFFICE12\ACECORE.DLL from Microsoft Corporation
From the listed threads only thread 142 is waiting for thread 141. All others are waiting for thread 142.
~141
141 Id: 1d14.1b38 Suspend: 0 Teb: 7fee4000 Unfrozen
Start: msvcrt!_endthreadex+0x2f (77bcb4bc)
Priority: 0 Priority class: 32 Affinity: ff
~141s : Edit: new stacktrace after fixing symbols
kb
2a2efdcc 7c827b89 77e6202c 00000003 2a2efe1c ntdll!KiFastSystemCallRet
2a2efdd0 77e6202c 00000003 2a2efe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
2a2efe78 7739bbd1 00000003 2a2efea0 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
2a2efed4 7739ce36 00000002 2a2eff74 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141
2a2efef0 4a77cb28 00000002 2a2eff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
2a2eff84 77bcb530 33c23fe8 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
2a2effb8 77e6482f 37e3b7e8 00000000 00000000 msvcrt!_endthreadex+0xa3
2a2effec 00000000 77bcb4bc 37e3b7e8 00000000 kernel32!BaseThreadStart+0x34
~142
142 Id: 1d14.1128 Suspend: 0 Teb: 7feb9000 Unfrozen
Start: msvcrt!_endthreadex+0x2f (77bcb4bc)
Priority: 0 Priority class: 32 Affinity: ff
~142s : Edit: new stacktrace after fixing symbols
kb
31b5aa88 7c827b99 7c83d09c 00006c44 00000000 ntdll!KiFastSystemCallRet
31b5aa8c 7c83d09c 00006c44 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
31b5aac8 7c83d0e7 00006c44 00000004 00000000 ntdll!RtlpWaitOnCriticalSection+0x1a3
*** ERROR: Module load completed but symbols could not be loaded for ACECORE.DLL
31b5aae8 3c9e20eb 1e5bd320 31b5ab3c 2ae13a61 ntdll!RtlEnterCriticalSection+0xa8
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 00000000 00000000 00000000 ACECORE+0x20eb
!clrstack
Shows call to System.Data.OleDb.OleDbConnection.Open()
I found it connects to xls
Provider=Microsoft.ACE.OLEDB.12.0;Data Source=c:\temp\somefile.xls;Extended Properties="Excel 8.0;HDR=Yes;"
~97
97 Id: 1d14.730 Suspend: 0 Teb: 7fee5000 Unfrozen
Start: mscorwks!CorExitProcess+0x21ef9 (79f756cf)
Priority: 0 Priority class: 32 Affinity: ff
Edit: new stacktrace after fixing symbols
~97s
kb
27e1ca10 7c827b99 77e61d1e 000018e4 00000000 ntdll!KiFastSystemCallRet
27e1ca14 77e61d1e 000018e4 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
27e1ca84 77e61c8d 000018e4 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
27e1ca98 7769c7ad 000018e4 ffffffff 2386d3a8 kernel32!WaitForSingleObject+0x12
27e1cab4 7778b5cb 24a1c758 2386d3a8 00000000 ole32!GetToSTA+0x7c
27e1cad4 7778c38b 27e1cb9c 27e1cc9c 22d3e674 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xcb
27e1cbb4 776c0585 22d3e674 27e1ccac 27e1cc9c ole32!CRpcChannelBuffer::SendReceive2+0xd3
27e1cc20 776c051a 22d3e674 27e1ccac 27e1cc9c ole32!CAptRpcChnl::SendReceive+0xab
27e1cc74 77ce347f 22d3e674 27e1ccac 27e1cc9c ole32!CCtxComChnl::SendReceive+0x1a9
27e1cc90 77ce352f 45e02be4 27e1ccd8 0600016e rpcrt4!NdrProxySendReceive+0x43
27e1d080 77ce35a6 776762b8 776794ba 27e1d0b8 rpcrt4!NdrClientCall2+0x206
27e1d0a0 77c65037 00000014 00000004 27e1d0d0 rpcrt4!ObjectStublessClient+0x8b
27e1d0b0 776ad951 45e02be4 00000000 27e1d89c rpcrt4!ObjectStubless+0xf
27e1d0d0 776acb4b 77794960 00000001 00000000 ole32!CProcessActivator::CCICallback+0x6d
27e1d0f0 776acafc 77794960 27e1d6f8 00000000 ole32!CProcessActivator::AttemptActivation+0x2c
27e1d12c 776ada3b 77794960 27e1d6f8 00000000 ole32!CProcessActivator::ActivateByContext+0x4f
27e1d154 776aaf9e 77794960 00000000 27e1d89c ole32!CProcessActivator::CreateInstance+0x49
27e1d194 4a777108 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
27e1d210 776aaf9e 443dd8f8 00000000 27e1d89c comsvcs!CSTAPoolActivator::CreateInstance+0x5a9
27e1d250 4a766303 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
27e1d2a0 776aaf9e 341d3168 00000000 27e1d89c comsvcs!CStdContextActivator::CreateInstance+0x221
27e1d2e0 77727f8a 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
27e1d310 776f5c55 0009a0f8 00000000 27e1d89c ole32!CSurrogateProcessActivator::CreateInstance+0xf7
27e1d344 776aaf9e 7779487c 00000000 27e1d89c ole32!CClientContextActivator::CreateInstance+0xc9
27e1d384 776ab12f 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
27e1d68c 776a67ba 361f8c54 00000000 00000015 ole32!ICoCreateInstanceEx+0x3f8
27e1d6c0 7769b9b3 361f8c54 00000000 00000000 ole32!CComActivator::DoCreateInstance+0x6a
27e1ddc8 4a7516d8 45bfeb58 361f8c54 00000000 ole32!CComActivator::StandardCreateInstance+0x7c
27e1de3c 4a751fc6 00000000 361f8c40 00000000 comsvcs!CClassFactoryWrapper::ActivateOnMachine+0xaf
27e1de74 7a078d9b 361f8c40 00000000 79edda70 comsvcs!CClassFactoryWrapper::CreateInstance+0x80
27e1e118 7a07a1e6 361f8c40 00000000 00000000 mscorwks!ComClassFactory::CreateInstanceFromClassFactory+0x102
27e1e174 7a0bf10a 1e772994 00000000 27e1e1b4 mscorwks!ComClassFactory::CreateInstance+0x91
27e1e184 7a0c870b 1e772994 1e772994 79faa672 mscorwks!AllocateComObject_ForManaged+0x1e
27e1e1b4 79e9c82b 1e772994 234b20e8 79faa737 mscorwks!AllocateObject+0x38
27e1e1c0 79faa737 05ff2627 00000001 072336fc mscorwks!MethodTable::Allocate+0x35
27e1e260 792c25c3 1e772994 27e1e278 792c257c mscorwks!CRemotingServices::AllocateUninitializedObject+0xdf
27e1e278 792c1951 0f5b1b50 27e1e2d0 03100120 mscorlib_ni+0x2025c3
27e1e290 792c143e 44194464 00000000 00000016 mscorlib_ni+0x201951
27e1e2ac 79e71e04 00000001 27e1e334 79faa569 mscorlib_ni+0x20143e
27e1e2b8 79faa569 792c1400 072336fc 00000000 mscorwks!CTPMethodTable__CallTargetHelper3+0xf
27e1e334 79faa5d8 072336fc 00000000 00000001 mscorwks!CTPMethodTable::CallTarget+0xdd
27e1e348 79faa614 790fd65c 072336fc 00000000 mscorwks!CTPMethodTable::CallTarget+0x14
27e1e368 7a04b578 1e772994 00000000 00000001 mscorwks!CRemotingServices::CreateProxyOrObject+0x38
27e1e414 1e923c99 0f5b1a58 0f5b1b1c 27e1e464 mscorwks!JIT_NewCrossContextHelper+0xa9
WARNING: Frame IP not in any known module. Following frames may be wrong.
27e1e430 776e2fae 00000000 00000064 00000001 0x1e923c99
27e1e464 79e71b4c 27e1ef70 00000064 00000001 ole32!CoGetContextToken+0x29
27e1e494 79e821b9 27e1ee70 00000008 27e1ee20 mscorwks!CallDescrWorker+0x33
27e1e514 7a0f851b 27e1ee70 00000008 27e1ee20 mscorwks!CallDescrWorkerWithHandler+0xa3
27e1e5e4 79270454 00000001 00000000 00000000 mscorwks!CMessage::Dispatch+0x162
27e1e63c 7977c16e 00000001 00000000 075d788c mscorlib_ni+0x1b0454
27e1e658 6744d146 0752309c 0f5b19d8 0752304c mscorlib_ni+0x6bc16e
27e1e690 197cb7c7 0d0e3640 79e7a6b8 27e1ea80 System_EnterpriseServices_ni+0x5d146
27e1e70c 197f3d04 27e1eae4 0071f13b 36c6b460 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2b7c7)
27e1e74c 77720df0 27e1eae4 35ccbed0 00000000 System_EnterpriseServices_Wrapper_197f0000!System::EnterpriseServices::Thunk::FilteringCallbackFunction+0x44
27e1e798 7772189c 00000000 35ccbed0 197f3cc0 ole32!EnterForCallback+0xc4
27e1e8f8 776f0418 27e1e7d0 197f3cc0 27e1eae4 ole32!SwitchForCallback+0x1a3
27e1e924 7769c194 35ccbed0 197f3cc0 27e1eae4 ole32!PerformCallback+0x54
27e1e9bc 776e316c 36c6b460 197f3cc0 27e1eae4 ole32!CObjectContext::InternalContextCallback+0x159
27e1ea0c 79e71d8b 36c6b470 197f3cc0 27e1eae4 ole32!CObjectContext::ContextCallback+0x85
27e1ea68 197cbc82 776e30e7 0f5b1a28 072323f0 mscorwks!PInvokeCalliReturnFromCall
27e1ea88 197cbc82 0010c8f0 00000000 00000008 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2bc82)
27e1eab4 197cbb51 07231a74 27e1eb2c 79282f85 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2bc82)
27e1eac0 79282f85 27e1eb44 00000000 00000000 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2bb51)
27e1eb2c 797f5a0d 03100788 072323f0 075d7830 mscorlib_ni+0x1c2f85
00000000 00000000 00000000 00000000 00000000 mscorlib_ni+0x735a0d
And all other threads waiting for thread 142 have same stack trace.
I don't know what thread 141 is doing. Thread 142 cannot open connection to xls file. And not sure why other threads wait for thread 142.
Maybe you have any ideas?
EDIT:
analyze -v -hang
GetPageUrlData failed, server returned HTTP status 404
URL requested: http://watson.microsoft.com/StageOne/dllhst3g_exe/5_2_3790_3959/unknown/0_0_0_0/00000000.htm?Retriage=1
FAULTING_IP:
+1e02faf0157df58
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 0000008d
BUGCHECK_STR: HANG
PROCESS_NAME: dllhst3g.exe
ERROR_CODE: (NTSTATUS) 0xcfffffff -
EXCEPTION_CODE: (NTSTATUS) 0xcfffffff -
MOD_LIST:
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0xc2c (17)
Current frame:
ChildEBP RetAddr Caller,Callee
1974f450 655f1110 (MethodDesc 0x651b7efc +0x30 System.Data.ProviderBase.WrappedIUnknown.ReleaseHandle())
1974f93c 792e5e4f (MethodDesc 0x79107064 +0xf System.Runtime.InteropServices.SafeHandle.Dispose(Boolean))
1974f944 792e5d6b (MethodDesc 0x79107030 +0x1b System.Runtime.InteropServices.SafeHandle.Finalize())
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
0 1d14.71c Speculated (Triage) -->
17 1d14.c2c Critical Section -->
141 1d14.1b38 Event
WAIT_CHAIN_COMMAND: ~0s;k;;~17s;k;;~141s;k;;
BLOCKING_THREAD: 00001b38
DEFAULT_BUCKET_ID: APPLICATION_HANG_WRONG_SYMBOLS
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_WRONG_SYMBOLS
LAST_CONTROL_TRANSFER: from 7c827b89 to 7c82847c
STACK_TEXT:
2a2efdcc 7c827b89 77e6202c 00000003 2a2efe1c ntdll!KiFastSystemCallRet
2a2efdd0 77e6202c 00000003 2a2efe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
2a2efe78 7739bbd1 00000003 2a2efea0 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
2a2efed4 7739ce36 00000002 2a2eff74 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141
2a2efef0 4a77cb28 00000002 2a2eff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
2a2eff84 77bcb530 33c23fe8 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
2a2effb8 77e6482f 37e3b7e8 00000000 00000000 msvcrt!_endthreadex+0xa3
2a2effec 00000000 77bcb4bc 37e3b7e8 00000000 kernel32!BaseThreadStart+0x34
FOLLOWUP_IP:
ntdll!KiFastSystemCallRet+0
7c82847c c3 ret
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: ntdll!KiFastSystemCallRet+0
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: ntdll
IMAGE_NAME: ntdll.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4cc18322
STACK_COMMAND: ~141s ; kb
BUCKET_ID: HANG_ntdll!KiFastSystemCallRet+0
FAILURE_BUCKET_ID: APPLICATION_HANG_WRONG_SYMBOLS_cfffffff_ntdll.dll!KiFastSystemCallRet
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/dllhst3g_exe/5_2_3790_3959/45d69678/unknown/0_0_0_0/bbbbbbb4/cfffffff/00000000.htm?Retriage=1
Followup: MachineOwner
Not sure I understand it. But thread 17 is finalizer thread. So looks like it is blocked? And I thinkg it as again some kind of access to xls?
~17s
kb
ChildEBP RetAddr Args to Child
1974f08c 7c827b99 7c83d09c 00003a98 00000000 ntdll!KiFastSystemCallRet
1974f090 7c83d09c 00003a98 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
1974f0cc 7c83d0e7 00003a98 00000004 00000001 ntdll!RtlpWaitOnCriticalSection+0x1a3
1974f0ec 2ae1fa94 29202124 32787727 00000000 ntdll!RtlEnterCriticalSection+0xa8
WARNING: Stack unwind information not available. Following frames may be wrong.
1974f160 4c856487 2263eff4 00000000 00000005 ACEOLEDB!DllGetClassObject+0xdc2d
1974f174 4c862d06 2263eff4 4c8633c4 2263efe8 oledb32!CACMDynamic::CmFinalRelease+0x50
1974f17c 4c8633c4 2263efe8 2263efe8 1974f1e4 oledb32!CDCM::FinalRelease+0x1b
1974f198 4c8633f6 00000000 2263efe8 1974f1b8 oledb32!ATL::CComPolyObject::~CComPolyObject+0x2a
1974f1a8 4c88d5ad 2263efe8 2370a39c 1974f1f4 oledb32!ATL::CComPolyObject::Release+0x25
1974f1b8 4a757681 1a271fa0 2263efe8 2370a39c oledb32!CDCMCreator::DestroyResource+0xf
1974f1f4 4a75793e 2370a39c 2263efe8 00000000 comsvcs!CHolder::SafeDispenserDriver::DestroyResource+0x20
1974f20c 4a758a35 23151cd0 00000000 2fa44808 comsvcs!CHolder::ProcessDestroyList+0x2e
1974f238 4c88d30e 23151cd0 2263efe8 2263efe8 comsvcs!CHolder::FreeResource+0x7f
1974f268 4c878a3a 2263efe8 00000000 271aa824 oledb32!CDCMCreator::ReleaseResource+0x31
1974f288 4c8545e7 271aa824 00000000 4c85456a oledb32!CDPO::ReturnDCMToPool+0x89
1974f294 4c85456a 271aa818 271aa818 1974f320 oledb32!CDPO::FinalRelease+0xb
1974f2b0 4c88e78a 271aa818 1974f2d0 4c85218c oledb32!ATL::CComPolyObject::~CComPolyObject+0x2a
1974f2bc 4c85218c 00000001 00000000 000eaaf8 oledb32!ATL::CComPolyObject::`scalar deleting destructor'+0xd
1974f2d0 4c8521aa 271aa818 1974f330 79f63ff9 oledb32!ATL::CComPolyObject::Release+0x27
1974f2dc 79f63ff9 271aa824 3b6a3777 000eaaf8 oledb32!ATL::CComContainedObject::Release+0x11
1974f330 79f640ac 271aa828 3b6a3733 000eaaf8 mscorwks!ReleaseTransitionHelper+0x5f
1974f374 79f64110 271aa828 00000000 3b6a37ef mscorwks!SafeReleaseHelper+0x8c
1974f3a8 79f53b5d 271aa828 00000000 3b6a3017 mscorwks!SafeRelease+0x2f
1974f450 655f1110 000eaaf8 00000000 1974f470 mscorwks!MarshalNative::Release+0xb0
1974f460 79e71b4c 1974f4dc 000c5958 1974f4f0 System_Data_ni!_bidW103 (System_Data_ni+0x4a1110)
1974f470 79e821b9 1974f540 00000000 1974f510 mscorwks!CallDescrWorker+0x33
1974f4f0 79e96531 1974f540 00000000 1974f510 mscorwks!CallDescrWorkerWithHandler+0xa3
1974f634 79e96564 655f10e0 1974f6bc 1974f6a4 mscorwks!MethodDesc::CallDescr+0x19c
1974f650 79e96c4c 655f10e0 1974f6bc 1974f6a4 mscorwks!MethodDesc::CallTargetWorker+0x1f
1974f670 79eccd2a 1974f6a4 0b402780 00000004 mscorwks!MethodDescCallSite::Call_RetObjPtr+0x1c
1974f770 79eccd5e 0b402780 000eaaf8 000eaaf8 mscorwks!SafeHandle::RunReleaseMethod+0x89
1974f870 79eccbf4 00000001 0d0e3640 79e7a1c8 mscorwks!SafeHandle::Release+0x11b
1974f89c 79f83999 3b6a3d7b 00000000 000eaaf8 mscorwks!SafeHandle::Dispose+0x23
1974f93c 792e5e4f 1974f970 792e5d6b 0b3ac100 mscorwks!SafeHandle::Finalize+0xab
1974f944 792e5d6b 0b3ac100 ffffffff 00000000 mscorlib_ni+0x225e4f
1974f970 79f7169a 1974f9c4 79ef465c 1974fc1c mscorlib_ni+0x225d6b
1974f9d4 79f7175b 0b402780 792e5d50 08000000 mscorwks!MethodTable::GetObjCreateDelegate+0xaf
1974f9f4 79f71609 0b402780 0b402780 00000000 mscorwks!MethodTable::CallFinalizer+0xa0
1974fa08 79fd46a6 0b402780 3b6a3e1f 00000000 mscorwks!SVR::CallFinalizer+0xa6
1974fa58 79fd45d7 000eaaf8 00000415 1974fadf mscorwks!SVR::GCHeap::TraceGCSegments+0x1b0
1974fae0 79f5832f 0740eeb4 00000000 1974feb0 mscorwks!SVR::GCHeap::TraceGCSegments+0x2f6
1974faf4 79e9848f 1974fd04 000eaaf8 00000000 mscorwks!SVR::ProfScanRootsHelper+0x69
1974fb08 79e9842b 1974feb0 1974fb90 79fa6a6b mscorwks!Thread::DoADCallBack+0x32a
1974fb9c 79e98351 1974feb0 3b6a3f9f 000eaaf8 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
1974fbd8 79ec4322 1974feb0 00000000 1974fc98 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
1974fbe8 79f581f3 1974feb0 1974fc8c 79fa6a6b mscorwks!Thread::RaiseCrossContextException+0x434
1974fc98 79f58279 1a9cb678 79ec430e 1974feb0 mscorwks!Thread::DoADCallBack+0xcd
1974fcb8 79f58265 1974feb0 1974fd20 79f582e4 mscorwks!Thread::DoADCallBack+0x322
1974fcc4 79f582e4 1a9cb678 79f58316 1974fd04 mscorwks!ManagedThreadBase::FinalizerAppDomain+0x25
1974fd20 79fd45d7 000eaaf8 00000000 1974fda7 mscorwks!SVR::GCHeap::TraceGCSegments+0x251
1974fda8 79fd48a8 00000000 00000000 1974feb0 mscorwks!SVR::GCHeap::TraceGCSegments+0x2f6
1974fdc0 79e9848f 1974feb0 00000000 00000000 mscorwks!SVR::GCHeap::FinalizerThreadWorker+0xb7
1974fdd4 79e9842b 1974feb0 1974fe5c 79fa6a6b mscorwks!Thread::DoADCallBack+0x32a
1974fe68 79e98351 1974feb0 3b6a3ae3 00000000 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
1974fea4 79f074d4 1974feb0 00000000 1a9cb678 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
1974fecc 79f074e5 79fd4809 00000008 1974ff14 mscorwks!ManagedThreadBase_NoADTransition+0x32
1974fedc 79f090b3 79fd4809 3b6a3b53 00000000 mscorwks!ManagedThreadBase::FinalizerBase+0xd
1974ff14 79f75715 00000000 00000007 ffffffff mscorwks!SVR::GCHeap::FinalizerThreadStart+0xbb
1974ffb8 77e6482f 000eb528 00000000 00000000 mscorwks!Thread::intermediateThreadProc+0x49
1974ffec 00000000 79f756cf 000eb528 00000000 kernel32!BaseThreadStart+0x34
You need to find who owns the following critical section that your thread is waiting on:
1974f0ec 2ae1fa94 29202124 32787727 00000000 ntdll!RtlEnterCriticalSection+0xa8
You can do an automated critical section analysis with call stacks:
!locks -v
This will dump all critical section locks that are in a locked state and the call stacks of the threads, you then need to scan each call stack for each lock to see if say Thread A is waiting on lock 1 which is owned by Thread B, Thread B is waiting on lock 2 which is owned by Thread A.
Hope this helps