w3wp process hosting my .NET application is crashing at random times. I have collected a dump file by setting up a second chance exception rule using DebugDiag. Here are the steps I have performed.
The lastevent command shows a .NET exception.
0:027> .lastevent
Last event: 1ae4.2e98: CLR exception - code e0434352 (first/second chance not available)
The stack trace of this thread looks as follows,
0:027> !CLRStack
OS Thread Id: 0x2e98 (68)
Child SP IP Call Site
000000266c1bab18 00007fff6e5d95fc [HelperMethodFrame: 000000266c1bab18]
000000266c1bac00 00007fff5cb38afb System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)
000000266c1bcb48 00007fff6657120d [HelperMethodFrame: 000000266c1bcb48]
000000266c1bcc30 00007fff5cb359b5 System.Runtime.AsyncResult.Complete(Boolean)
000000266c1bea00 00007fff6657120d [FaultingExceptionFrame: 000000266c1bea00]
000000266c1bef00 00007fff5bca63a5 System.Web.HttpApplication+AsyncEventExecutionStep.OnAsyncEventCompletion(System.IAsyncResult)
000000266c1bef60 00007fff5cb3586c System.Runtime.AsyncResult.Complete(Boolean)
000000266c1befd0 00007fff570da192 System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(System.Object)
000000266c1bf020 00007fff5cb38b63 System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
000000266c1bf090 00007fff5cb38ac7 System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)
000000266c1bf0f0 00007fff650a045c System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*) [f:\dd\ndp\clr\src\BCL\system\threading\overlapped.cs # 135]
000000266c1bf2a0 00007fff66416793 [GCFrame: 000000266c1bf2a0]
000000266c1bf498 00007fff66416793 [DebuggerU2MCatchHandlerFrame: 000000266c1bf498]
000000266c1bf628 00007fff66416793 [ContextTransitionFrame: 000000266c1bf628]
000000266c1bf858 00007fff66416793 [DebuggerU2MCatchHandlerFrame: 000000266c1bf858]
pe command shows a callback exception
0:027> !pe
Exception object: 0000002552364d38
Exception type: System.Runtime.CallbackException
Message: Async Callback threw an exception.
InnerException: System.NullReferenceException, Use !PrintException 0000002552364ae8 to see more.
StackTrace (generated):
SP IP Function
000000266C1BCC30 00007FFF5CB359B5 System_ServiceModel_Internals_ni!System.Runtime.AsyncResult.Complete(Boolean)+0x235
000000266C1BEFD0 00007FFF570DA192 System_ServiceModel_Activation_ni!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(System.Object)+0x92
000000266C1BF020 00007FFF5CB38B63 System_ServiceModel_Internals_ni!System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x53
000000266C1BF090 00007FFF5CB38AFB System_ServiceModel_Internals_ni!System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x6b
000000266C1BF0F0 00007FFF650A045C mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x7c
StackTraceString: <none>
HResult: 80131501
There are nested exceptions on this thread. Run with -nested for details
Per output of previous command, I look for nested exceptions
0:027> !PrintException -nested /d 0000002552364ae8
Exception object: 0000002552364ae8
Exception type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
InnerException: <none>
StackTrace (generated):
SP IP Function
000000266C1BEF00 00007FFF5BCA63A5 System_Web_ni!System.Web.HttpApplication+AsyncEventExecutionStep.OnAsyncEventCompletion(System.IAsyncResult)+0x55
000000266C1BEF60 00007FFF5CB3586C System_ServiceModel_Internals_ni!System.Runtime.AsyncResult.Complete(Boolean)+0xec
StackTraceString: <none>
HResult: 80004003
Nested exception -------------------------------------------------------------
Exception object: 0000002552364d38
Exception type: System.Runtime.CallbackException
Message: Async Callback threw an exception.
InnerException: System.NullReferenceException, Use !PrintException 0000002552364ae8 to see more.
StackTrace (generated):
SP IP Function
000000266C1BCC30 00007FFF5CB359B5 System_ServiceModel_Internals_ni!System.Runtime.AsyncResult.Complete(Boolean)+0x235
000000266C1BEFD0 00007FFF570DA192 System_ServiceModel_Activation_ni!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(System.Object)+0x92
000000266C1BF020 00007FFF5CB38B63 System_ServiceModel_Internals_ni!System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x53
000000266C1BF090 00007FFF5CB38AFB System_ServiceModel_Internals_ni!System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x6b
000000266C1BF0F0 00007FFF650A045C mscorlib_ni!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)+0x7c
StackTraceString: <none>
HResult: 80131501
My next step is to find the stack trace for that NullReferenceException
0:027> !do 0000002552364ae8
Name: System.NullReferenceException
MethodTable: 00007fff652865a0
EEClass: 00007fff64c3f180
Size: 160(0xa0) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007fff65276948 400028e 8 System.String 0 instance 000000244f54d350 _className
00007fff6528f6b8 400028f 10 ...ection.MethodBase 0 instance 0000002552366f58 _exceptionMethod
00007fff65276948 4000290 18 System.String 0 instance 0000000000000000 _exceptionMethodString
00007fff65276948 4000291 20 System.String 0 instance 000000244bf27668 _message
00007fff65286788 4000292 28 ...tions.IDictionary 0 instance 00000025523751a8 _data
00007fff65276b78 4000293 30 System.Exception 0 instance 0000000000000000 _innerException
00007fff65276948 4000294 38 System.String 0 instance 0000000000000000 _helpURL
00007fff65276f28 4000295 40 System.Object 0 instance 0000002552364cc0 _stackTrace
00007fff65276f28 4000296 48 System.Object 0 instance 0000000000000000 _watsonBuckets
00007fff65276948 4000297 50 System.String 0 instance 0000000000000000 _stackTraceString
00007fff65276948 4000298 58 System.String 0 instance 0000000000000000 _remoteStackTraceString
00007fff65279288 4000299 88 System.Int32 1 instance 0 _remoteStackIndex
00007fff65276f28 400029a 60 System.Object 0 instance 0000000000000000 _dynamicMethods
00007fff65279288 400029b 8c System.Int32 1 instance -2147467261 _HResult
00007fff65276948 400029c 68 System.String 0 instance 000000255242ce60 _source
00007fff6528fbc0 400029d 78 System.IntPtr 1 instance 0 _xptrs
00007fff65279288 400029e 90 System.Int32 1 instance -532462766 _xcode
00007fff65250340 400029f 80 System.UIntPtr 1 instance 7fff5bca63a4 _ipForWatsonBuckets
00007fff65265538 40002a0 70 ...ializationManager 0 instance 0000002552364c40 _safeSerializationManager
00007fff65276f28 400028d b8 System.Object 0 shared static s_EDILock
>> Domain:Value 000000244acba390:NotInit 000000267072bd80:NotInit <<
Here I attempt to get the stack trace of NullReferenceException. This looks like a SByte array.
0:027> !do 0000002552364cc0
Name: System.SByte[]
MethodTable: 00007fff65202b20
EEClass: 00007fff64c34f60
Size: 120(0x78) bytes
Array: Rank 1, Number of elements 96, Type SByte (Print Array)
Content: ........0Qkv&....c.[.......l&...x..[............kX.\....`..l&...`..\............................
Fields:
None
My expectation is to get a stacktrace/details of what method/line of code/object is responsible for causing this null reference. I also attempt to dump the contents of SByte array but that doesn't provide me any useful information. Any suggestions on how can I get more information about this NullReferenceException?
First of all, I think that !pe -nested already shows the call stack that you're looking for:
0:027> !PrintException -nested /d 0000002552364ae8
[...]
StackTrace (generated):
SP IP Function
000000266C1BEF00 00007FFF5BCA63A5 System_Web_ni!System.Web.HttpApplication+AsyncEventExecutionStep.OnAsyncEventCompletion(System.IAsyncResult)+0x55
000000266C1BEF60 00007FFF5CB3586C System_ServiceModel_Internals_ni!System.Runtime.AsyncResult.Complete(Boolean)+0xec
[...]
Next, a stack trace is exactly what you see in that SByte[]: it's just a bunch of numbers.
Those number do get their meaning only together with PDB files. The PDB file contains the information to turn numbers into names.
Have a look at the values in the SP column and IP column. They are:
000000266C1BEF00
000000266C1BEF60
00007FFF5BCA63A5
00007FFF5CB3586C
And now, use a converter that converts those values into text:
You'll find that the printable characters resssemble those of the output shown in SByte[] by WinDbg:
0:027> !do 0000002552364cc0
[...]
Content: ........0Qkv&....c.[.......l&...x..[............kX.\....`..l&...`..\............................
The order may be different due to little/big-endianness.
Related
I can use poi in windbg scripting to traverse the fields and then print the field i am interested in.
For e.g. if i have all objects of type X which has field X.y.z where z is an array, where y is at offset 0x10 and z is at offset 0x8. I can write
.foreach ( dSM { !dumpheap -short -type X})
{
.printf "%p\n", poi(poi(${dSM}+0x10)+0x8);
!dumparray poi(poi(${dSM}+0x10)+0x8)
}
Now i want to print first/second element of all these arrays, how can i do that ?
using !do poi(poi(poi(${dSM}+0x10)+0x8)) does not work.
0:045> !DumpArray /d 000001d3b96787a8
Name: ABC[]
MethodTable: 00007ffc951e76e0
EEClass: 00007ffcf22f4480
Size: 56(0x38) bytes
Array: Rank 1, Number of elements 4, Type CLASS
Element Methodtable: 00007ffc951e6cc0
[0] 000001d3b9678788
[1] null
[2] null
[3] null
0:045> !dumpobj /d poi(000001d3b96787a8)
<Note: this object has an invalid CLASS field>
Invalid object
Array class is:
:045> !DumpClass /d 00007ffcf22f4480
Class Name: System.Object[]
mdToken: 0000000002000000
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Parent Class: 00007ffcf22f5018
Module: 00007ffcf2221000
Method Table: 00007ffcf2949e80
Vtable Slots: 18
Total Method Slots: 1c
Class Attributes: 2101
Transparency: Transparent
NumInstanceFields: 0
NumStaticFields: 0
Given this program:
using System;
namespace WinDbgArrayAccess
{
class Program
{
static void Main()
{
Program[] justAnArray = new Program[20];
for (int i =0; i<justAnArray.Length;i++) justAnArray[i] = new Program();
Console.WriteLine("Access the elements of the array in WinDbg now!");
Console.ReadLine();
}
}
}
You can see
0:006> !DumpArray /d 0336243c
Name: WinDbgArrayAccess.Program[]
MethodTable: 01914db0
EEClass: 71967820
Size: 92(0x5c) bytes
Array: Rank 1, Number of elements 20, Type CLASS
Element Methodtable: 01914d60
[0] 03362498
[1] 033624a4
[2] 033624b0
[3] 033624bc
[4] 033624c8
[5] 033624d4
...
Now you need to find those numbers somewhere in the memory. Since we hardly have a different starting point, let's start at the array's address:
0:006> dp 0336243c L10
0336243c 01914db0 00000014 03362498 033624a4
0336244c 033624b0 033624bc 033624c8 033624d4
0336245c 033624e0 033624ec 033624f8 03362504
0336246c 03362510 0336251c 03362528 03362534
Now, 01914db0 is the type of the object (called Method Table, MT). 0x14 is 0n20, which is the size of the array. And after that, it seems we have the elements, 03362498, 033624a4, 033624b0 etc.
How do we access that programmatically? Well, it's easy now:
0:006> .printf "%p",poi(0336243c+(2+3)*$ptrsize)
033624bc
Where 2 is to skip MT and Length and 3 is the index of the array, giving you the 4th element.
I was able to get first 2 elements by going forward poi(poi(array+10)+8) and poi(poi(array+10)+10) offsets from array
0:298> .foreach (obj { !DumpHeap -short -live -mt 00007ffc951e7ab8}) { .printf "%p\n", ${obj}; !mex.do2 poi(poi(poi(poi(${obj}+0x10)+0x8)+0x10)+0x8); !mex.do2 poi(poi(poi(poi(${obj}+0x10)+0x8)+0x10)+0x10); }
000001d379ac94a0
[raw] 000001d379ac8bc8 "XYZ_String"
[raw] 000001d379ac8c88 "XYZ_String_2"
000001d379e87168
[raw] 000001d379e86888 "ABCD_String"
[raw] 000001d379e86948 "ABCD_String_2"
000001d3b972a218
I'm trying to debug a large sketch, I'm occasionally getting a crash related to the web interface, and am trying to find out where exactly this is going wrong.
The stack trace ends with:
last failed alloc call: 4022D552(1480)
This particular address is not present in the decode stack trace itself, but may hold the key to finding the source of the problem. Any suggestions on how to track this one down?
Decoding 36 results
0x402222c5: BearSSL::WiFiClientSecure::_installClientX509Validator() at /home/wouter/.arduino15/packages/esp8266/tools/xtensa-lx106-elf-gcc/2.5.0-3-20ed2b9/xtensa-lx106-elf/include/c++/4.8.2/bits/shared_ptr_base.h line 986
: (inlined by) ?? at /home/wouter/.arduino15/packages/esp8266/tools/xtensa-lx106-elf-gcc/2.5.0-3-20ed2b9/xtensa-lx106-elf/include/c++/4.8.2/bits/shared_ptr.h line 316
: (inlined by) ?? at /home/wouter/.arduino15/packages/esp8266/tools/xtensa-lx106-elf-gcc/2.5.0-3-20ed2b9/xtensa-lx106-elf/include/c++/4.8.2/bits/shared_ptr.h line 598
: (inlined by) ?? at /home/wouter/.arduino15/packages/esp8266/tools/xtensa-lx106-elf-gcc/2.5.0-3-20ed2b9/xtensa-lx106-elf/include/c++/4.8.2/bits/shared_ptr.h line 614
: (inlined by) BearSSL::WiFiClientSecure::_installClientX509Validator() at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp line 877
0x40222cfc: BearSSL::WiFiClientSecure::_connectSSL(char const*) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp line 962
0x40219c70: esp_yield at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/core_esp8266_main.cpp line 91
0x4021a8c3: delay at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/core_esp8266_wiring.cpp line 54
0x40206d6d: WiFiClient::connect(IPAddress, unsigned short) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/libraries/ESP8266WiFi/src/include/ClientContext.h line 136
: (inlined by) WiFiClient::connect(IPAddress, unsigned short) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/libraries/ESP8266WiFi/src/WiFiClient.cpp line 170
0x40222eed: BearSSL::WiFiClientSecure::connect(char const*, unsigned short) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp line 231
0x40225230: BearSSL::PrivateKey::getEC() const at ?? line ?
0x40225230: BearSSL::PrivateKey::getEC() const at ?? line ?
0x40216b5c: HTTPClient::connect() at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp line 1165
0x4021c16e: uart_write at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/uart.cpp line 498
0x40217820: HTTPClient::sendRequest(char const*, unsigned char*, unsigned int) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp line 655
0x40217d98: HardwareSerial::write(unsigned char const*, unsigned int) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/HardwareSerial.h line 158
0x40273eba: sleep_reset_analog_rtcreg_8266 at ?? line ?
0x402180a5: Print::write(char const*) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/Print.h line 60
0x40217d83: HardwareSerial::write(unsigned char) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/HardwareSerial.h line 154
0x402179ba: HTTPClient::GET() at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp line 575
0x402118e8: HydroMonitorLogging::sendPostData(char*) at /home/wouter/Arduino/libraries/HydroMonitor/src/HydroMonitorLogging.cpp line 730
0x40225470: BearSSL::PrivateKey::getEC() const at ?? line ?
0x4022c4c0: _vsprintf_r at /home/earle/src/esp-quick-toolchain/repo/newlib/newlib/libc/stdio/vsprintf.c line 65
0x40217d98: HardwareSerial::write(unsigned char const*, unsigned int) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/HardwareSerial.h line 158
0x4022000a: spiffs_object_truncate at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/spiffs/spiffs_nucleus.cpp line 1727
0x40211d41: HydroMonitorLogging::transmitMessages() at /home/wouter/Arduino/libraries/HydroMonitor/src/HydroMonitorLogging.cpp line 337
0x40211caa: HydroMonitorLogging::transmitMessages() at /home/wouter/Arduino/libraries/HydroMonitor/src/HydroMonitorLogging.cpp line 326
0x40204dab: handleAPI() at /home/wouter/Arduino/Williams_fridge/Fridge_control/webAPI.ino line 149
0x402251c8: BearSSL::PrivateKey::getEC() const at ?? line ?
0x40217d98: HardwareSerial::write(unsigned char const*, unsigned int) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/HardwareSerial.h line 158
0x40218130: Print::println() at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/Print.cpp line 178
0x402121b5: HydroMonitorLogging::logData() at /home/wouter/Arduino/libraries/HydroMonitor/src/HydroMonitorLogging.cpp line 250
0x40217d83: HardwareSerial::write(unsigned char) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/HardwareSerial.h line 154
0x40224948: Print::write(char) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/Print.h line 73
0x40218114: Print::print(char) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/Print.cpp line 126
0x402106f4: HydroMonitorIsolatedSensorBoard::readSensor(bool) at /home/wouter/Arduino/libraries/HydroMonitor/src/HydroMonitorIsolatedSensorBoard.cpp line 36
0x40207a07: TwoWire::requestFrom(int, int) at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/libraries/Wire/Wire.cpp line 134
0x402042f9: loop at /home/wouter/Arduino/Williams_fridge/Fridge_control/loop.ino line 36
0x40219d20: loop_wrapper() at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/core_esp8266_main.cpp line 125
0x401015d5: cont_wrapper at /home/wouter/.arduino15/packages/esp8266/hardware/esp8266/2.5.2/cores/esp8266/cont.S line 81
Note: I also asked on the esp8266 forum, and opened an issue on github about the same.
Intermittent ESP crashes are most often due to running out of RAM. A "failed alloc call" likely refers to a failure to allocate memory for an operation (encryption takes a good bit, as does every open request in the queue). The stack trace will not point to a specific spot in your code because client libraries run "in the background" (triggered by interrupts or once-per-loop calls to a function that does everything) and no specific event in your firmware caused it. On a memory-starved ESP with a long response time, sending several AJAX requests at once from the frontend can cause the overload.
To see whether this is the case, try printing the result of ESP.getFreeHeap() at various points in your loop. If it falls much below 10kb, there is a risk that another request will drive it over the edge.
I tested the free heap and got around 32 kB. So didn't expect any memory issues.
Turns out, 32 kB is less than 17 kB + 5 kB - for clientSecure and server respectively. And that's what caused the crashes. Dropping SSL encryption completely solved the problem. The development team mentioned it must be contiguous memory - which for whatever reason that 32 kB of mine apparently isn't.
Simple conclusion: you can not run the server (just http - no encryption) AND have a secure client in the same program.
No AJAX, by the way. Haven't gotten around to that level yet.
When i use windbg !do to view a address,it output:
0:000> !do 01ef30f4
Name: System.Collections.Generic.List`1[[System.Byte[], mscorlib]]
MethodTable: 0021285c
EEClass: 6313a530
Size: 24(0x18) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
MT Field Offset Type VT Attr Value Name
633b70a0 4000ca6 4 System.Object[] 0 instance 01eed1b8 _items
63402df0 4000ca7 c System.Int32 1 instance 0 _size
63402df0 4000ca8 10 System.Int32 1 instance 0 _version
633ffa60 4000ca9 8 System.Object 0 instance 00000000 _syncRoot
633b70a0 4000caa 0 System.Object[] 0 shared static _emptyArray
**Domain:Value dynamic statics NYI 002a8428:NotInit**
What's the meaning of Domain:Value dynamic statics NYI 002a8428:NotInit?
For static fields, !do displays the static field's value in each AppDomain you have. The first value, 002a8428, is the AppDomain address, and the second value is the static field's value in that AppDomain.
I have a crash dump from a production server that shows an OutOfMemoryException. The exception itself is not relevant here.
I happened to run a !dso to view the stack objects:
0:042> !dso
OS Thread Id: 0x1014 (42)
ESP/REG Object Name
246eeb24 109a21bc System.UnhandledExceptionEventHandler
246eeb2c 39083998 System.Runtime.Remoting.Proxies.__TransparentProxy
246eeb34 39083b5c System.UnhandledExceptionEventArgs
246eeb48 39073280 System.Byte[]
246eec10 2e720050 System.OutOfMemoryException
[snip]
246ef250 0ac1c4d0 System.IO.MemoryStream <-- interesting
I thought the MemoryStream might have something to do with the error, so I dumped it:
0:042> !do 0ac1c4d0
Name: System.IO.MemoryStream
MethodTable: 7932d5e4
EEClass: 790ec318
Size: 52(0x34) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
MT Field Offset Type VT Attr Value Name
7933061c 400018a 4 System.Object 0 instance 00000000 __identity
7992cbcc 4001b6c 8 ...ream+ReadDelegate 0 instance 00000000 _readDelegate
7992cc58 4001b6d c ...eam+WriteDelegate 0 instance 00000000 _writeDelegate
7931bd9c 4001b6e 10 ...ng.AutoResetEvent 0 instance 00000000 _asyncActiveEvent
79332c4c 4001b6f 14 System.Int32 1 instance 1 _asyncActiveCount
7932e6fc 4001b6b 574 System.IO.Stream 0 shared static Null
>> Domain:Value 000dc0f0:NotInit 00109d58:109b6abc <<
79333470 4001c16 18 System.Byte[] 0 instance 50710038 _buffer
79332c4c 4001c17 1c System.Int32 1 instance 0 _origin
79332c4c 4001c18 20 System.Int32 1 instance 56071048 _position
79332c4c 4001c19 24 System.Int32 1 instance 56071048 _length
79332c4c 4001c1a 28 System.Int32 1 instance 67108864 _capacity
793044cc 4001c1b 2c System.Boolean 1 instance 1 _expandable
793044cc 4001c1c 2d System.Boolean 1 instance 1 _writable
793044cc 4001c1d 2e System.Boolean 1 instance 1 _exposable
793044cc 4001c1e 2f System.Boolean 1 instance 1 _isOpen
Wow, a 56,071,048 byte buffer seems a bit large. I'd like to see the contents of this buffer:
0:042> !do 50710038
Name: System.Byte[]
MethodTable: 79333470
EEClass: 790eeb6c
Size: 67108876(0x400000c) bytes
Array: Rank 1, Number of elements 67108864, Type Byte
Element Type: System.Byte
Fields:
None
The first 10 elements of the array are below:
0:042> !dumparray -start 0 -length 10 50710038
Name: System.Byte[]
MethodTable: 79333470
EEClass: 790eeb6c
Size: 67108876(0x400000c) bytes
Array: Rank 1, Number of elements 67108864, Type Byte
Element Methodtable: 79333520
[0] 50710040
[1] 50710041
[2] 50710042
[3] 50710043
[4] 50710044
[5] 50710045
[6] 50710046
[7] 50710047
[8] 50710048
[9] 50710049
This is a huge array. I'd rather not !dumparray the whole thing. I'd like to view the output in a file.
Question
Is it possible to dump the contents of this Byte[] to a file?
I am familiar with the .writemem command, but I can't seem to get this to work. I've tried writing the entire length, but WinDbg didn't like that:
0:042> .writemem C:\LargeBuffer.bin 50710040 L56071048
^ Range error in '.writemem C:\LargeBuffer.bin 50710040 l56071048'
Did I format that .writemem command incorrectly?
The L modifier for ranges is limited in size. If you want to get around the limit use the L? range modifier. The following command worked for me:
0:000> !do 0x04cc1000
Name: System.Byte[]
MethodTable: 68374944
EEClass: 680aaf1c
Size: 67108876(0x400000c) bytes
Array: Rank 1, Number of elements 67108864, Type Byte
Element Type:System.Byte
Content: ................................................................................................................................
Fields:
None
0:000> .writemem c:\temp\array.bin 0x04cc1000 L?0x400000c
Writing 400000c bytes
This is what worked for me:
.foreach($str {!DumpHeap /d -mt 00007ff890e96948 -min 0n126500 -short}){r#$t0= dwo(${$str}+8)*2;.writemem e:\temp\str\${$str}.txt ${$str}+c L? #$t0}
I have a simple test program causing an infinite wait on lock.
public class SyncBlock
{
}
class Program
{
public static SyncBlock sync = new SyncBlock();
private static void ThreadProc()
{
try
{
Monitor.Enter(sync);
}
catch (Exception)
{
//Monitor.Exit(sync);
Console.WriteLine("3rd party code threw an exception");
}
}
static void Main(string[] args)
{
Thread newThread = new Thread(ThreadProc);
newThread.Start();
Console.WriteLine("Acquiring lock");
Monitor.Enter(sync);
Console.WriteLine("Releasing lock");
Monitor.Exit(sync);
}
}
So the main thread is basically get locked when it tries to do Monitor.Enter(sync). If I looked at !clrStack on main thread, its output basically show it which make sense but when I try to see native side of stack, I am expecting to see some Wait on single/multiple object type of call but I don't see it. Can anyone explain it. Thanks
0:000> !CLRStack
PDB symbol for mscorwks.dll not loaded
OS Thread Id: 0x1e8 (0)
ESP EIP
0012f0a8 77455e74 [GCFrame: 0012f0a8]
0012f178 77455e74 [HelperMethodFrame_1OBJ: 0012f178] System.Threading.Monitor.Enter (System.Object)
0012f1d0 00a40177 ConsoleApplication1.Program.Main(System.String[])
0012f400 70fc1b4c [GCFrame: 0012f400]
0:000> kb
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be wrong.
0012eeb4 710afb92 0012ee68 002d6280 00000000 ntdll!KiFastSystemCallRet
0012ef1c 710af7c3 00000001 002d6280 00000000 mscorwks!StrongNameFreeBuffer+0x1b1f2
0012ef3c 710af8cc 00000001 002d6280 00000000 mscorwks!StrongNameFreeBuffer+0x1ae23
0012efc0 710af961 00000001 002d6280 00000000 mscorwks!StrongNameFreeBuffer+0x1af2c
0012f010 710afae1 00000001 002d6280 00000000 mscorwks!StrongNameFreeBuffer+0x1afc1
0012f06c 70fdc5ae ffffffff 00000001 00000000 mscorwks!StrongNameFreeBuffer+0x1b141
0012f080 710df68a ffffffff 00000001 00000000 mscorwks!LogHelp_NoGuiOnAssert+0x10562
0012f10c 710b1154 002aad90 ffffffff 002aad90 mscorwks!StrongNameFreeBuffer+0x4acea
0012f128 710b10d8 42b8b47d 00000000 002aad90 mscorwks!StrongNameFreeBuffer+0x1c7b4
0012f1e0 70fc1b4c 0012f1f0 0012f230 0012f270 mscorwks!StrongNameFreeBuffer+0x1c738
0012f1f0 70fd2219 0012f2c0 00000000 0012f290 mscorwks+0x1b4c
0012f270 70fe6591 0012f2c0 00000000 0012f290 mscorwks!LogHelp_NoGuiOnAssert+0x61cd
0012f3ac 70fe65c4 0023c038 0012f478 0012f444 mscorwks!CoUninitializeEE+0x2ead
0012f3c8 70fe65e2 0023c038 0012f478 0012f444 mscorwks!CoUninitializeEE+0x2ee0
0012f3e0 7103389d 0012f444 42b8b0f1 00000000 mscorwks!CoUninitializeEE+0x2efe
0012f544 710337bd 002332e0 00000001 0012f580 mscorwks!GetPrivateContextsPerfCounters+0xf546
0012f7ac 71033d0d 00000000 42b8b9c9 00000001 mscorwks!GetPrivateContextsPerfCounters+0xf466
0012fc7c 71033ef7 00ce0000 00000000 42b8979 mscorwks!GetPrivateContextsPerfCounters+0xf9b6
0012fccc 71033e27 00ce0000 42b8b8a1 00000000 mscorwks!CorExeMain+0x168
* ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscoreei.dll -
0012fd14 71cf55ab 71033d8f 0012fd30 71f37f16 mscorwks!CorExeMain+0x98
* ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Windows\system32\mscoree.dll -
0012fd20 71f37f16 00000000 71cf0000 0012fd44 mscoreei!CorExeMain+0x38
0012fd30 71f34de3 00000000 7723d0e9 7ffd8000 mscoree!CreateConfigStream+0x13f
0012fd44 774319bb 7ffd8000 084952f9 00000000 mscoree!CorExeMain+0x8
0012fd84 7743198e 71f34ddb 7ffd8000 00000000 ntdll!RtlInitializeExceptionChain+0x63
0012fd9c 00000000 71f34ddb 7ffd8000 00000000 ntdll!RtlInitializeExceptionChain+0x36
You have to point windbg to the microsoft windows symbols server to get a good stack trace.
type in the following in your windbg command window:
.sympath srv*c:\websymbols*http://msdl.microsoft.com/download/symbols
Also see this:
Using microsoft symbol server to get symbols
Also, to answer your original question about how to debug this, here is the cookbook:
0:000> !clrstack
OS Thread Id: 0x1358 (0)
ESP EIP
0012f328 7c90e514 [GCFrame: 0012f328]
0012f3f8 7c90e514 [HelperMethodFrame_1OBJ: 0012f3f8] System.Threading.Monitor.Enter(System.Object)
0012f450 00d10177 Program.Main(System.String[])
0012f688 79e71b4c [GCFrame: 0012f688]
In your original program, the background thread was started first. So, it acquired the lock. However it exited without releasing the lock. After that your main thread tried to acquire the lock and it is stuck because the lock is already owned.
How do you find out who owns it? First do a !threads followed by !syncblk.
0:000> !threads
ThreadCount: 3
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 1
Hosted Runtime: no
PreEmptive GC Alloc Lock
ID OSID ThreadOBJ State GC Context Domain Count APT Exception
0 1 1358 0014bb00 200a020 Enabled 00000000:00000000 001540d0 0 MTA
2 2 1360 0015e320 b220 Enabled 00000000:00000000 001540d0 0 MTA (Finalizer)
XXXX 3 0 00175a98 9820 Enabled 00000000:00000000 001540d0 1 Ukn
0:000> !syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
2 0017903c 3 1 00175a98 0 XXX 013503cc SyncBlock
-----------------------------
Total 2
CCW 0
RCW 0
ComClassFactory 0
Free 0
As you can see, !syncblk says that the owining thread object is 00175a98. From the !threads output, you can see that thread object 00175a98 is the dead thread that exited while owning the lock.
Hope this helps.