I'm trying to get uv4l working with logitech camera 310. I go to the address XXX:8090/stream and it works for a few seconds then stops. To get it back i need to restart the pi. Also, it sometimes outputs these:
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491012] Internal error: Oops: 80d [#1] SMP ARM
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491317] Process uv4l (pid: 575, stack limit = 0xb29d4210)
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491328] Stack: (0xb29d5d74 to 0xb29d6000)
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491341] 5d60: 80611830 b29d5d87 000346db
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491359] 5d80: 80c05dcc 00c02d00 00000004 000005a8 000005a8 00000000 b292d140 b29d5ec0
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491376] 5da0: b292d140 00000b50 b29d5ddc b29d5db8 8067a204 80611750 b29d5ddc b29d5dc8
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491392] 5dc0: 000005a8 000005a8 b292d220 b6172180 b29d5e74 b29d5de0 8067ad5c 8067a1b4
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491408] 5de0: b29d5dfc b29d5df0 00000000 76149afc b29d5e24 80c06984 80d04818 80c0495c
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491424] 5e00: b29d4020 00000000 00000000 00000000 b29d5e0c b3610000 00000001 b29d5ec8
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491440] 5e20: b292d250 00000000 00000000 000005a8 00000000 000005a8 7fffffff a65d0f74
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491456] 5e40: b3782800 80280000 b56b2f00 b292d140 00000000 9888e540 00000121 80108244
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491472] 5e60: b29d4000 00000000 b29d5e94 b29d5e78 806a513c 8067a978 8028fa98 8028fa04
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491488] 5e80: 00000000 00000000 b29d5ea4 b29d5e98 8060a878 806a50ac b29d5f8c b29d5ea8
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491504] 5ea0: 8060b914 8060a860 b29d5ec8 b29d5eb8 fffffff7 00000001 7613c388 00001000
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491520] 5ec0: 00000000 00000000 00000001 00000b50 000004b0 b29d5eb8 00000001 b3610000
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491535] 5ee0: 00000000 00000000 00000000 00000000 b349d6c0 b5991d80 00000000 80b7e940
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491551] 5f00: b29d5f74 b29d5f10 807146b4 8014472c 0000000a 80c7b980 80c71eb8 80c0312c
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491567] 5f20: 80b793b8 00000000 00000001 00000081 76149ae0 80108244 b29d4000 00000001
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491583] 5f40: b29d5fa4 b29d5f50 8019d6a0 8019c92c 00000000 00000000 ffffffff 00000000
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491598] 5f60: b29d4000 00000002 b29d4010 00000000 b29d5fb0 73a00480 7613c388 00000000
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491614] 5f80: b29d5fa4 b29d5f90 8060b964 8060b858 00000000 00000000 00000000 b29d5fa8
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491630] 5fa0: 801080c0 8060b948 73a00480 7613c388 0000000e 7613c388 00001000 00000000
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491646] 5fc0: 73a00480 7613c388 00000000 00000121 7613c388 71bac3b0 758ec150 00000001
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491662] 5fe0: 00000000 7533b790 76ec7504 76ec82f4 80000010 0000000e 00000000 00000000
Message from syslogd#raspberrypi at Mar 7 18:31:37 ...
kernel:[ 320.491831] Code: a8a0500c cafffff9 08bd8000 e3110020 (18a0500c)
I followed this guide http://www.linux-projects.org/uv4l/installation/. I've watched countless videos of people doing the exact same but it's just not working for me. Let me know if anything else is needed to help figure it out.
You are getting a kernel oops. This means something bad happened inside the kernel. Most likely, there is a kernel bug related to userspace USB I/O, as that is what uv4l's UVC driver is using. Try upgrading the kernel on your Raspberry Pi and check whether the issue is still there.
The bug might be triggered by normal I/O from the camera, but it might also be that there is something abnormal that causes unexpected data from the camera, which is then badly handled by the kernel. Possible reasons for that could be a bad USB cable or connector, or not entirely unlikely with the Pi, an inadequate power supply that cannot keep the voltage high enough for the hardware to run reliably.
Related
I'm using the latest Raspbian Jessie (2016-05-27) and Raspberry Pi 1 Model B (512 Mb). I'm trying to stream video through Logitec C270 (usb web cam) and mjpg-streamer.
I followed instructions here, but it does not work for me.
I'm using the following command to start streaming server and tried different combinations with resolution and frames per second:
/usr/local/bin/mjpg_streamer -i "/usr/local/lib/input_uvc.so -n" -o "/usr/local/lib/output_http.so -p 8888 -w /usr/local/www" &
After that the server is not starting at all. Soon I realised that new kernel requires a patch:
nano input_uvc_patch
--- plugins/input_uvc/input_uvc.c (revision 174)
+++ plugins/input_uvc/input_uvc.c (working copy)
## -405,9 +405,13 ##
if(pcontext->videoIn->formatIn == V4L2_PIX_FMT_YUYV) {
DBG("compressing frame from input: %d\n", (int)pcontext->id);
pglobal->in[pcontext->id].size = compress_yuyv_to_jpeg(pcontext->videoIn, pglobal->in[pcontext->id].buf, pcontext->videoIn->framesizeIn, gquality);
+ /* copy this frame's timestamp to user space */
+ pglobal->in[pcontext->id].timestamp = pcontext->videoIn->buf.timestamp;
} else {
DBG("copying frame from input: %d\n", (int)pcontext->id);
- pglobal->in[pcontext->id].size = memcpy_picture(pglobal->in[pcontext->id].buf, pcontext->videoIn->tmpbuffer, pcontext->videoIn->buf.bytesused);
+ pglobal->in[pcontext->id].size = memcpy_picture(pglobal->in[pcontext->id].buf, pcontext->videoIn->tmpbuffer, pcontext->videoIn->tmpbytesused);
+ /* copy this frame's timestamp to user space */
+ pglobal->in[pcontext->id].timestamp = pcontext->videoIn->tmptimestamp;
}
#if 0
## -418,8 +422,6 ##
prev_size = global->size;
#endif
- /* copy this frame's timestamp to user space */
- pglobal->in[pcontext->id].timestamp = pcontext->videoIn->buf.timestamp;
/* signal fresh_frame */
pthread_cond_broadcast(&pglobal->in[pcontext->id].db_update);
Index: plugins/input_uvc/v4l2uvc.c
===================================================================
--- plugins/input_uvc/v4l2uvc.c (revision 174)
+++ plugins/input_uvc/v4l2uvc.c (working copy)
## -450,6 +450,8 ##
*/
memcpy(vd->tmpbuffer, vd->mem[vd->buf.index], vd->buf.bytesused);
+ vd->tmpbytesused = vd->buf.bytesused;
+ vd->tmptimestamp = vd->buf.timestamp;
if(debug)
fprintf(stderr, "bytes in used %d \n", vd->buf.bytesused);
Index: plugins/input_uvc/v4l2uvc.h
===================================================================
--- plugins/input_uvc/v4l2uvc.h (revision 174)
+++ plugins/input_uvc/v4l2uvc.h (working copy)
## -28,6 +28,7 ##
#include <stdio.h>
+#include <stdint.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
## -105,6 +106,8 ##
int framecount;
int recordstart;
int recordtime;
+ uint32_t tmpbytesused;
+ struct timeval tmptimestamp;
};
patch -p0 < input_uvc_patch
make USE_LIBV4L2=true clean all
sudo make install
After that the server was up and the LED on camera blinked for a few seconds. Then I've got the following errors:
Full trace:
Ignoring empty buffer ...
Ignoring empty buffer ...
Ignoring empty buffer ...
Ignoring empty buffer ...
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.624402] Internal error: Oops: 5 [#1] ARM
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.758195] Process systemd-udevd (pid: 116, stack limit = 0xda5ae188)
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.766538] Stack: (0xda5afd58 to 0xda5b0000)
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.772716] fd40: 00000142 00000000
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.784357] fd60: da539000 da5afd8c da5afd84 da5afd78 d6956800 da539000 da5afdb4 da5afd88
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.795970] fd80: c04c5000 c04ab53c 00000008 7fffffff da539000 da5aff4c da539000 00000000
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.807600] fda0: 00000142 dbb5c6a4 da5afe14 da5afdb8 c04c5598 c04c4f84 da5afe50 da5afdc4
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.819330] fdc0: 0000000c da5afe50 da5aff4c 00000008 00000000 da480d00 00000000 00000074
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.831231] fde0: 00000000 00000000 c047b83c da5aff4c 00000000 00000000 daa11000 00000000
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.843342] fe00: da5afe30 00000000 da5afe24 da5afe18 c047b3b8 c04c5288 da5aff34 da5afe28
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.855608] fe20: c047ba70 c047b3a0 c01057dc 00000000 da5f6020 da47b934 192253cf 00000000
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.868069] fe40: da5afe6c da5afe50 c006634c 00000000 beeeaed4 00000028 80840fb8 0000011a
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.880651] fe60: da5afe8c 00000000 daa11840 c0107e24 80841fc0 da5b3680 da5afea4 da5affb0
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.893334] fe80: da5b3680 0000081f 80841fc0 da47b900 00000010 dbb5c6a4 00000000 da5afea8
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.906203] fea0: c0573cdc c0053218 00000000 fffffff5 00000000 beeeaef4 00000008 00000000
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.919120] fec0: 00000000 00000000 00000800 00000008 da5afeb4 c0823ae4 0000081f c0573b44
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.932037] fee0: 80841fc0 da5affb0 00001000 00003038 da5affac da5aff00 c0009204 c0146dfc
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.944971] ff00: da5aff1c da5aff10 c0146dfc c0146d68 daa11000 00000000 beeeaeb8 00000128
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.957905] ff20: c000f9e8 da5ae000 da5aff94 da5aff38 c047c7cc c047b8a0 00000000 00000001
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.970831] ff40: 7f6428c4 00000000 fffffff7 da5afe90 0000000c 00000001 00000000 00000000
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.983741] ff60: da5afe60 00000000 00000000 00000000 00000000 00000000 beeeaeb8 beeeaeb8
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 150.996677] ff80: b6fddf10 00000000 da5affa4 da5aff98 c047c814 c047c78c 00000000 da5affa8
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 151.009618] ffa0: c000f820 c047c808 beeeaeb8 b6fddf10 00000004 beeeaeb8 00000000 0000000c
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 151.022541] ffc0: beeeaeb8 b6fddf10 00000000 00000128 8083cb34 0000011a 808330c0 80840fb8
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 151.035446] ffe0: 00000000 beeeae94 7f61a388 b6f6d37c 60000010 00000004 00000000 00000000
Message from syslogd#raspberrypi at Jul 29 19:16:24 ...
kernel:[ 151.136256] Code: 1a000025 e59030b0 e3530000 0a00001b (e593400c)
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 151.810011] Internal error: Oops: 1 [#2] ARM
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 151.948402] Process systemd-udevd (pid: 748, stack limit = 0xd894a188)
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 151.956749] Stack: (0xd894be78 to 0xd894c000)
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 151.962924] be60: 00000000 d69568d4
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 151.974593] be80: c0876ce8 d6956800 d894beac d894be98 c04827e4 c0481bb0 00000000 fffffffe
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 151.986212] bea0: d894bef4 d894beb0 c04c49f8 c04827c8 c0107dd0 d782d160 d88a2a80 00000000
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 151.997855] bec0: 00000000 da619f20 da619f20 d782d160 00000000 d782d180 dad11010 d7871b28
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 152.009585] bee0: d782d180 00000008 d894bf0c d894bef8 c047a6ec c04c47e4 d6966f00 00000000
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 152.021489] bf00: d894bf1c d894bf10 c047a790 c047a6c8 d894bf5c d894bf20 c012dcdc c047a780
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 152.033629] bf20: 00000000 00000000 c0109c64 d6966f08 d6966f00 d8869f48 d8869b40 c0889f90
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 152.045938] bf40: 00000000 c000f9e8 d894a000 00000000 d894bf6c d894bf60 c012de88 c012dc50
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 152.058412] bf60: d894bf8c d894bf70 c003b2b0 c012de7c 00000004 d894a000 c000f9e8 d894bfb0
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 152.070976] bf80: d894bfac d894bf90 c0013524 c003b244 8083cb28 00000000 7f660004 00000006
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 152.083616] bfa0: 00000000 d894bfb0 c000f848 c0013450 00000000 00000000 00000000 00000000
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 152.096453] bfc0: 8083cb28 00000000 7f660004 00000006 beeeaaf8 7f66000c 8083cb28 80833008
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 152.109361] bfe0: 00000000 beeea93c 7f62de58 b6f6cd3c 60000010 0000000c 00000000 00000000
Message from syslogd#raspberrypi at Jul 29 19:16:25 ...
kernel:[ 152.225862] Code: e5963008 e3a01000 e2433001 e5863008 (e894000c)
Can anybody explain me what I'm doing wrong? Thank you.
The problem was that there was too low power for Raspberry Pi. So, it's not problem with Raspberry Pi or mjpg_streamer.
We have an old VB6 application that keeps freezing and/or crashing. We setup an easy shortcut for users to create a full dump when the program freezes.
We now have a week's worth of dumps, and most of them show a cryptic stack from the hung thread, which doesn't even include any of our functions, and the last call is to user32!NtUserSetFocus.
We're really stuck here... can anyone help?
0:000> kb
ChildEBP RetAddr Args to Child
0018d788 755eee3e 002609ac 00000001 755eed34 user32!NtUserSetFocus+0x15
0018d7a0 755962fa 00260f22 00000110 002609ac user32!MB_DlgProc+0x10a
0018d7cc 755bf9df 755eed34 00260f22 00000110 user32!InternalCallWinProc+0x23
0018d848 755bf784 00000000 755eed34 00260f22 user32!UserCallDlgProcCheckWow+0xd7
0018d898 755bf889 0160cf70 00000000 00000110 user32!DefDlgProcWorker+0xb7
0018d8b8 755962fa 00260f22 00000110 002609ac user32!DefDlgProcW+0x29
0018d8e4 75596d3a 755bf860 00260f22 00000110 user32!InternalCallWinProc+0x23
0018d95c 7559965e 00000000 76ee3d54 00260f22 user32!UserCallWinProcCheckWow+0x109
0018d9a0 755c206f 0160cf70 00000000 76ee3d54 user32!SendMessageWorker+0x581
0018da74 755bcf4b 75580000 00000008 00000000 user32!InternalCreateDialog+0xb9f
0018daac 755ef73c 75580000 1736f6a8 0010193a user32!InternalDialogBox+0xc1
0018db60 755efa18 00000030 ffffffff ffff0000 user32!SoftModalMessageBox+0x757
0018dcb8 755efc65 0018dcd0 00000000 755efbd1 user32!MessageBoxWorker+0x269
0018dd38 729af829 0018dd50 0010193a 0010193a user32!MessageBoxIndirectA+0x94
0018dd78 729af6a5 0018ddbc 00000000 00000000 msvbvm60!VBMessageBox2+0x92
0018dda0 729af9a0 729af7ce 0018ddbc 0018ddbc msvbvm60!MessageBoxPVoid+0x4b
0018ddd0 729a3d68 00000000 12aa8ef8 0018ddf8 msvbvm60!DlgEnableModeless+0x5e
0018de34 729a3db6 032807d4 002308e4 00000030 msvbvm60!_Scanint+0x13
0018de54 72a0c411 174be0e4 00000000 00000030 msvbvm60!RefMemberIDFromHxmod+0x39
0018de70 72a0c6f3 174be0e4 00000000 00000030 msvbvm60!EbShowError+0x3
0018de94 72a2497c 010831b8 00000000 00000000 msvbvm60!GetErrMsg+0x90
0018deb0 770fb6ad 0018df9c 00000000 0018dfec msvbvm60!SehUpdateStack+0x29
0018ded4 770fb67f 0018df9c 0018f6f8 0018dfec ntdll!ExecuteHandler2+0x26
0018def8 770fb620 0018df9c 0018f6f8 0018dfec ntdll!ExecuteHandler+0x24
0018df84 770b0163 0018df9c 0018dfec 0018df9c ntdll!RtlDispatchException+0x127
0018df84 74f3c42d 0018df9c 0018dfec 0018df9c ntdll!KiUserExceptionDispatcher+0xf
0018e4a0 72a10dcf c000008f 00000001 00000002 KERNELBASE!RaiseException+0x58
0018e4c0 72a0e228 010831b8 800a0061 0018e584 msvbvm60!CEnumConPnts::QueryInterface+0x34
0018e4d4 72a0e28c 010831b8 00000000 00000000 msvbvm60!BasicExcepDeferredFillIn+0x65
0018e4e4 72a0be99 00000061 11035a85 00000000 msvbvm60!BasicExcepDeferredFillIn+0xd2
00000000 00000000 00000000 00000000 00000000 msvbvm60!ValidateArray+0xb4
!runaway proves that this is indeed the hung thread.
0:000> !runaway
User Mode Time
Thread Time
0:ca8 0 days 0:01:00.325
8:13a4 0 days 0:00:00.171
6:10b4 0 days 0:00:00.062
10:1554 0 days 0:00:00.031
19:1598 0 days 0:00:00.000
18:a88 0 days 0:00:00.000
17:7b0 0 days 0:00:00.000
16:ba0 0 days 0:00:00.000
15:770 0 days 0:00:00.000
14:103c 0 days 0:00:00.000
13:16bc 0 days 0:00:00.000
12:17e4 0 days 0:00:00.000
11:1160 0 days 0:00:00.000
9:1194 0 days 0:00:00.000
7:dc8 0 days 0:00:00.000
5:1510 0 days 0:00:00.000
4:af0 0 days 0:00:00.000
3:e5c 0 days 0:00:00.000
2:10f4 0 days 0:00:00.000
1:ff8 0 days 0:00:00.000
!analyze -hang -v produces the following:
0:000> !analyze -hang -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
*** WARNING: Unable to verify checksum for mscorlib.ni.dll
GetUrlPageData2 (WinHttp) failed: 12152.
FAULTING_IP:
+0
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
CONTEXT: 00000000 -- (.cxr 0x0;r)
eax=00000001 ebx=00000000 ecx=00000000 edx=00000000 esi=0018dcd0 edi=00260f22
eip=755a218a esp=0018d788 ebp=0018d7a0 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00200246
user32!NtUserSetFocus+0x15:
755a218a 83c404 add esp,4
FAULTING_THREAD: 00000000
BUGCHECK_STR: HANG
DEFAULT_BUCKET_ID: APPLICATION_HANG
PROCESS_NAME: OurProcess.exe
ERROR_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>
EXCEPTION_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: OurProcess.exe
ANALYSIS_VERSION: 6.3.9600.16384 (debuggers(dbg).130821-1623) x86fre
MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0xca8 (0)
Current frame:
ChildEBP RetAddr Caller, Callee
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
-- --- ------- --------------------------
0 758.ca8 Unknown
WAIT_CHAIN_COMMAND: ~0s;k;;
BLOCKING_THREAD: 00000ca8
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG
LAST_CONTROL_TRANSFER: from 755eee3e to 755a218a
STACK_TEXT:
0018d788 755eee3e 002609ac 00000001 755eed34 user32!NtUserSetFocus+0x15
0018d7a0 755962fa 00260f22 00000110 002609ac user32!MB_DlgProc+0x10a
0018d7cc 755bf9df 755eed34 00260f22 00000110 user32!InternalCallWinProc+0x23
0018d848 755bf784 00000000 755eed34 00260f22 user32!UserCallDlgProcCheckWow+0xd7
0018d898 755bf889 0160cf70 00000000 00000110 user32!DefDlgProcWorker+0xb7
0018d8b8 755962fa 00260f22 00000110 002609ac user32!DefDlgProcW+0x29
0018d8e4 75596d3a 755bf860 00260f22 00000110 user32!InternalCallWinProc+0x23
0018d95c 7559965e 00000000 76ee3d54 00260f22 user32!UserCallWinProcCheckWow+0x109
0018d9a0 755c206f 0160cf70 00000000 76ee3d54 user32!SendMessageWorker+0x581
0018da74 755bcf4b 75580000 00000008 00000000 user32!InternalCreateDialog+0xb9f
0018daac 755ef73c 75580000 1736f6a8 0010193a user32!InternalDialogBox+0xc1
0018db60 755efa18 00000030 ffffffff ffff0000 user32!SoftModalMessageBox+0x757
0018dcb8 755efc65 0018dcd0 00000000 755efbd1 user32!MessageBoxWorker+0x269
0018dd38 729af829 0018dd50 0010193a 0010193a user32!MessageBoxIndirectA+0x94
0018dd78 729af6a5 0018ddbc 00000000 00000000 msvbvm60!VBMessageBox2+0x92
0018dda0 729af9a0 729af7ce 0018ddbc 0018ddbc msvbvm60!MessageBoxPVoid+0x4b
0018ddd0 729a3d68 00000000 12aa8ef8 0018ddf8 msvbvm60!DlgEnableModeless+0x5e
0018de34 729a3db6 032807d4 002308e4 00000030 msvbvm60!_Scanint+0x13
0018de54 72a0c411 174be0e4 00000000 00000030 msvbvm60!RefMemberIDFromHxmod+0x39
0018de70 72a0c6f3 174be0e4 00000000 00000030 msvbvm60!EbShowError+0x3
0018de94 72a2497c 010831b8 00000000 00000000 msvbvm60!GetErrMsg+0x90
0018deb0 770fb6ad 0018df9c 00000000 0018dfec msvbvm60!SehUpdateStack+0x29
0018ded4 770fb67f 0018df9c 0018f6f8 0018dfec ntdll!ExecuteHandler2+0x26
0018def8 770fb620 0018df9c 0018f6f8 0018dfec ntdll!ExecuteHandler+0x24
0018df84 770b0163 0018df9c 0018dfec 0018df9c ntdll!RtlDispatchException+0x127
0018df84 74f3c42d 0018df9c 0018dfec 0018df9c ntdll!KiUserExceptionDispatcher+0xf
0018e4a0 72a10dcf c000008f 00000001 00000002 KERNELBASE!RaiseException+0x58
0018e4c0 72a0e228 010831b8 800a0061 0018e584 msvbvm60!CEnumConPnts::QueryInterface+0x34
0018e4d4 72a0e28c 010831b8 00000000 00000000 msvbvm60!BasicExcepDeferredFillIn+0x65
0018e4e4 72a0be99 00000061 11035a85 00000000 msvbvm60!BasicExcepDeferredFillIn+0xd2
00000000 00000000 00000000 00000000 00000000 msvbvm60!ValidateArray+0xb4
FOLLOWUP_IP:
msvbvm60!VBMessageBox2+92
729af829 8bd8 mov ebx,eax
SYMBOL_STACK_INDEX: e
SYMBOL_NAME: msvbvm60!VBMessageBox2+92
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: msvbvm60
IMAGE_NAME: msvbvm60.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4a5bda6c
STACK_COMMAND: ~0s ; kb
BUCKET_ID: HANG_msvbvm60!VBMessageBox2+92
FAILURE_BUCKET_ID: APPLICATION_HANG_cfffffff_msvbvm60.dll!VBMessageBox2
ANALYSIS_SOURCE: UM
FAILURE_ID_HASH_STRING: um:application_hang_cfffffff_msvbvm60.dll!vbmessagebox2
FAILURE_ID_HASH: {e6dc63dc-251f-a6d5-c66e-f5e07e418955}
Followup: MachineOwner
---------
Firstly I'd find out what it's doing. When it hangs get a couple of crash dumps and compare the stacks to find out where exactly it's spinning. Also some of the symbols have large offsets (user32!InternalCreateDialog+0xb9f), are you using a symbol server?
After running the Crash/Hang Analyzer in the DebugDiag tool it list that thread 17 is locking.
Running a !locks command in windbg also points to thread 17.
I am up to this point...
0:000> ~17 kb
ChildEBP RetAddr Args to Child
0261fcc4 7c827b69 77e6202c 00000002 0261fd14 ntdll!KiFastSystemCallRet
0261fcc8 77e6202c 00000002 0261fd14 00000001 ntdll!ZwWaitForMultipleObjects+0xc
0261fd70 77e62fbe 00000002 7a3d0468 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
0261fd8c 79f9a9d2 00000002 7a3d0468 00000000 kernel32!WaitForMultipleObjects+0x18
0261fdac 79f9ab07 000f2ab0 0261feb0 000f32b0 mscorwks!SVR::WaitForFinalizerEvent+0x77
0261fdc0 79eac697 0261feb0 00000000 00000000 mscorwks!SVR::GCHeap::FinalizerThreadWorker+0x49
0261fdd4 79eac633 0261feb0 0261fe5c 79fb685f mscorwks!Thread::DoADCallBack+0x32a
0261fe68 79eac553 0261feb0 3aa1ce47 00000000 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
0261fea4 79f4431c 0261feb0 00000000 0010f080 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
0261fecc 79f4432d 79f9aabc 00000008 0261ff14 mscorwks!ManagedThreadBase_NoADTransition+0x32
0261fedc 79f45ecb 79f9aabc 3aa1cff7 00000000 mscorwks!ManagedThreadBase::FinalizerBase+0xd
0261ff14 79fd2733 00000000 00000003 ffffffff mscorwks!SVR::GCHeap::FinalizerThreadStart+0xbb
0261ffb8 77e6482f 000f32b0 00000000 00000000 mscorwks!Thread::intermediateThreadProc+0x49
0261ffec 00000000 79fd26ed 000f32b0 00000000 kernel32!BaseThreadStart+0x34
From my understanding that the stack in thread 17 was trying to run this last
0261fcc4 7c827b69 77e6202c 00000002 0261fd14 ntdll!KiFastSystemCallRet
What do I do from here to go further on my analysis?
It’s a large 32 bits mixed mode MFC 7.0 app on XP, the user tells that he was using a feature which is implemented in managed code.
The crach is in a thread that has acquired the LoaderLock, and seems to orgin from .NET workerthread pool.
0:016> !cs -o -l
-----------------------------------------
DebugInfo = 0x7c97e1a0
Critical section = 0x7c97e174 (ntdll!LdrpLoaderLock+0x0)
LOCKED
LockCount = 0x4
OwningThread = 0x00000260
RecursionCount = 0x1
LockSemaphore = 0x7BC
SpinCount = 0x00000000
OwningThread DbgId = ~16s
OwningThread Stack =
ChildEBP RetAddr Args to Child
0f66e400 7c90df4a 7c8648a2 00000002 0f66e57c ntdll!KiFastSystemCallRet (FPO: [0,0,0])
0f66e404 7c8648a2 00000002 0f66e57c 00000001 ntdll!ZwWaitForMultipleObjects+0xc (FPO: [5,0,0])
0f66e74c 7c83ab50 0f66e774 7c839b39 0f66e77c kernel32!UnhandledExceptionFilter+0x8b9 (FPO: [Non-Fpo])
0f66e754 7c839b39 0f66e77c 00000000 0f66e77c kernel32!BaseThreadStart+0x4d (FPO: [Non-Fpo])
0f66e77c 7c9032a8 0f66e868 0f66ffdc 0f66e884 kernel32!_except_handler3+0x61 (FPO: [Uses EBP] [3,0,7])
0f66e7a0 7c90327a 0f66e868 0f66ffdc 0f66e884 ntdll!ExecuteHandler2+0x26
0f66e850 7c90e48a 00000000 0f66e884 0f66e868 ntdll!ExecuteHandler+0x24
0f66e850 79247eb4 00000000 0f66e884 0f66e868 ntdll!KiUserExceptionDispatcher+0xe (FPO: [2,0,0]) (CONTEXT # 0f66e884)
0f66eb4c 7929a46e 0e715d80 792483ef 0e715d80 mscorwks!Thread::UnhijackThread+0xb (FPO: [0,0,0])
0f66eb54 792483ef 0e715d80 00000000 00000000 mscorwks!Thread::RareEnablePreemptiveGC+0x36 (FPO: [0,0,0])
0f66eb64 792a6ff9 06ee0000 00000000 00000000 mscorwks!Thread::RareDisablePreemptiveGC+0x5f (FPO: [0,0,0])
0f66ec10 79247e14 06ee0000 00000003 00000000 mscorwks!SystemDomain::RunDllMain+0x7d (FPO: [Non-Fpo])
0f66ee98 603d6a2c 00000001 00000003 00000000 mscorwks!ExecuteDLL+0x3c0 (FPO: [Non-Fpo])
0f66eed8 603d70a3 06ee0000 0f66eebc 00000000 mscoreei!CorDllMainWorker+0x153 (FPO: [Non-Fpo])
0f66ef14 79015012 00000000 00000003 00000000 mscoreei!_CorDllMain+0x111 (FPO: [Non-Fpo])
0f66ef30 7c90118a 06ee0000 00000003 00000000 mscoree!ShellShim__CorDllMain+0xad (FPO: [Non-Fpo])
0f66ef50 7c91397b 06ef841e 06ee0000 00000003 ntdll!LdrpCallInitRoutine+0x14
0f66efc8 7c80c136 00000000 793fa180 7c80934a ntdll!LdrShutdownThread+0xd7 (FPO: [Non-Fpo])
0f66f000 792ee8ad 00000000 00000000 792ee78a kernel32!ExitThread+0x3e (FPO: [Non-Fpo])
0f66f020 792edfcb 00000000 00000000 00000000 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x123 (FPO: [Non-Fpo])
Some interesting vales on the stack might be the 06ee0000 and 0f66eebc.
The first is the base address for myMixedModeDll, and the second:
0:016> ln 06ef841e
(06ef841e) myMixedModeDll!CorDllMain | (06ef8424) myMixedModeDll!CDialog::CDialog
Exact matches:
The actual exception should be here:
0:000> .cxr 0f66e884;kb
eax=000000df ebx=00000000 ecx=0e715d80 edx=000003a4 esi=0e715d80 edi=00010000
eip=79247eb4 esp=0f66eb50 ebp=0f66ec10 iopl=0 nv up ei ng nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010282
mscorwks!Thread::UnhijackThread+0xb:
79247eb4 8910 mov dword ptr [eax],edx ds:0023:000000df=????????
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr Args to Child
0f66eb4c 7929a46e 0e715d80 792483ef 0e715d80 mscorwks!Thread::UnhijackThread+0xb
0f66eb54 792483ef 0e715d80 00000000 00000000 mscorwks!Thread::RareEnablePreemptiveGC+0x36
And yes the eax is not good:
0:000> u 79247eae
mscorwks!Thread::UnhijackThread+0x5:
79247eae 8b5178 mov edx,dword ptr [ecx+78h]
79247eb1 8b417c mov eax,dword ptr [ecx+7Ch]
79247eb4 8910 mov dword ptr [eax],edx
Yes, ECX has been restored properly
0:016> dd #ecx+0x78 L1
0e715df8 000003a4
0:016> dd #ecx+0x7c L1
0e715dfc 000000df
0:016> dd #ecx L0x20
0e715d80 0e6f4798 00000000 ffffffff 00000000
0e715d90 00000000 00000020 00000000 0e715da0
0e715da0 0e715da0 0e715da0 00000000 00000000
0e715db0 00000000 000000df 00000000 00000000
0e715dc0 00000000 00000000 00000000 00000000
0e715dd0 00000000 00000000 00000000 00000000
0e715de0 00000000 00000000 00000000 00000000
0e715df0 0e7093e8 00002733 000003a4 000000df
The last error value
0:016> !gle
LastErrorValue: (Win32) 0 (0) - The operation completed successfully.
LastStatusValue: (NTSTATUS) 0xc0000034 - Object Name not found.
This .NET is version 1.1.4322 , and the sos! Claims that thread #16 is not a managed thread.
0:016> !t
ThreadCount: 10
UnstartedThread: 0
BackgroundThread: 10
PendingThread: 0
DeadThread: 0
PreEmptive GC Alloc Lock
ID ThreadOBJ State GC Context Domain Count APT Exception
0 0xc8c 0x001ae598 0x4220 Enabled 0x1b7df804:0x1b7df8d8 0x001fda98 0 STA
5 0xcd4 0x001caea0 0xb220 Enabled 0x00000000:0x00000000 0x001fda98 0 MTA (Finalizer)
8 0xe28 0x0c56ac40 0x220 Enabled 0x00000000:0x00000000 0x001fda98 0 Ukn
10 0x8a8 0x0e5f4b48 0x800220 Enabled 0x1b822518:0x1b824458 0x001fda98 0 MTA (Threadpool Completion Port)
11 0xc18 0x0e6d6a60 0x800220 Enabled 0x1b8651cc:0x1b867008 0x001fda98 0 MTA (Threadpool Completion Port)
12 0xa54 0x00190c28 0x220 Enabled 0x1b5247f0:0x1b52650c 0x001fda98 0 Ukn
13 0xe9c 0x0e6627f8 0x220 Enabled 0x1b5307f0:0x1b53250c 0x001fda98 0 Ukn
14 0xe58 0x0e6b11a0 0x1800220 Enabled 0x00000000:0x00000000 0x001fda98 0 MTA (Threadpool Worker)
15 0x8dc 0x0e6d68a8 0x220 Enabled 0x00000000:0x00000000 0x001fda98 0 Ukn
17 0xbcc 0x0e709378 0x220 Enabled 0x1b52c7f0:0x1b52e50c 0x001fda98 0 Ukn
0:016> !ClrStack
Thread 16
Not a managed thread.
How can I find out more to reveal the cause to this crash ?
Admins told us that there is some prolem in production. They noticed big jump in memory usage and in requests waiting.
I received one crash dump. I need help analysing it.
Using Debug Diagnostic Tool I found this:
Detected possible blocking or leaked critical section at 0x1e5bd320 owned by thread 141 in dllhst3g.exe__Metastorm Process Engine__PID__7444__Date__10_25_2011__Time_01_19_15PM__686__Manual Dump.dmp
Impact of this lock
11,59% of threads blocked
(Threads 97 137 142 143 144 147 207 208 211 212 213 214 215 216 217 218 219 221 222 223 224 225 226 227 228 229 230)
The following functions are trying to enter this critical section
ACECORE+20eb
The following module(s) are involved with this critical section
C:\Program Files\Common Files\Microsoft Shared\OFFICE12\ACECORE.DLL from Microsoft Corporation
From the listed threads only thread 142 is waiting for thread 141. All others are waiting for thread 142.
~141
141 Id: 1d14.1b38 Suspend: 0 Teb: 7fee4000 Unfrozen
Start: msvcrt!_endthreadex+0x2f (77bcb4bc)
Priority: 0 Priority class: 32 Affinity: ff
~141s : Edit: new stacktrace after fixing symbols
kb
2a2efdcc 7c827b89 77e6202c 00000003 2a2efe1c ntdll!KiFastSystemCallRet
2a2efdd0 77e6202c 00000003 2a2efe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
2a2efe78 7739bbd1 00000003 2a2efea0 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
2a2efed4 7739ce36 00000002 2a2eff74 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141
2a2efef0 4a77cb28 00000002 2a2eff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
2a2eff84 77bcb530 33c23fe8 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
2a2effb8 77e6482f 37e3b7e8 00000000 00000000 msvcrt!_endthreadex+0xa3
2a2effec 00000000 77bcb4bc 37e3b7e8 00000000 kernel32!BaseThreadStart+0x34
~142
142 Id: 1d14.1128 Suspend: 0 Teb: 7feb9000 Unfrozen
Start: msvcrt!_endthreadex+0x2f (77bcb4bc)
Priority: 0 Priority class: 32 Affinity: ff
~142s : Edit: new stacktrace after fixing symbols
kb
31b5aa88 7c827b99 7c83d09c 00006c44 00000000 ntdll!KiFastSystemCallRet
31b5aa8c 7c83d09c 00006c44 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
31b5aac8 7c83d0e7 00006c44 00000004 00000000 ntdll!RtlpWaitOnCriticalSection+0x1a3
*** ERROR: Module load completed but symbols could not be loaded for ACECORE.DLL
31b5aae8 3c9e20eb 1e5bd320 31b5ab3c 2ae13a61 ntdll!RtlEnterCriticalSection+0xa8
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 00000000 00000000 00000000 ACECORE+0x20eb
!clrstack
Shows call to System.Data.OleDb.OleDbConnection.Open()
I found it connects to xls
Provider=Microsoft.ACE.OLEDB.12.0;Data Source=c:\temp\somefile.xls;Extended Properties="Excel 8.0;HDR=Yes;"
~97
97 Id: 1d14.730 Suspend: 0 Teb: 7fee5000 Unfrozen
Start: mscorwks!CorExitProcess+0x21ef9 (79f756cf)
Priority: 0 Priority class: 32 Affinity: ff
Edit: new stacktrace after fixing symbols
~97s
kb
27e1ca10 7c827b99 77e61d1e 000018e4 00000000 ntdll!KiFastSystemCallRet
27e1ca14 77e61d1e 000018e4 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
27e1ca84 77e61c8d 000018e4 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
27e1ca98 7769c7ad 000018e4 ffffffff 2386d3a8 kernel32!WaitForSingleObject+0x12
27e1cab4 7778b5cb 24a1c758 2386d3a8 00000000 ole32!GetToSTA+0x7c
27e1cad4 7778c38b 27e1cb9c 27e1cc9c 22d3e674 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xcb
27e1cbb4 776c0585 22d3e674 27e1ccac 27e1cc9c ole32!CRpcChannelBuffer::SendReceive2+0xd3
27e1cc20 776c051a 22d3e674 27e1ccac 27e1cc9c ole32!CAptRpcChnl::SendReceive+0xab
27e1cc74 77ce347f 22d3e674 27e1ccac 27e1cc9c ole32!CCtxComChnl::SendReceive+0x1a9
27e1cc90 77ce352f 45e02be4 27e1ccd8 0600016e rpcrt4!NdrProxySendReceive+0x43
27e1d080 77ce35a6 776762b8 776794ba 27e1d0b8 rpcrt4!NdrClientCall2+0x206
27e1d0a0 77c65037 00000014 00000004 27e1d0d0 rpcrt4!ObjectStublessClient+0x8b
27e1d0b0 776ad951 45e02be4 00000000 27e1d89c rpcrt4!ObjectStubless+0xf
27e1d0d0 776acb4b 77794960 00000001 00000000 ole32!CProcessActivator::CCICallback+0x6d
27e1d0f0 776acafc 77794960 27e1d6f8 00000000 ole32!CProcessActivator::AttemptActivation+0x2c
27e1d12c 776ada3b 77794960 27e1d6f8 00000000 ole32!CProcessActivator::ActivateByContext+0x4f
27e1d154 776aaf9e 77794960 00000000 27e1d89c ole32!CProcessActivator::CreateInstance+0x49
27e1d194 4a777108 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
27e1d210 776aaf9e 443dd8f8 00000000 27e1d89c comsvcs!CSTAPoolActivator::CreateInstance+0x5a9
27e1d250 4a766303 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
27e1d2a0 776aaf9e 341d3168 00000000 27e1d89c comsvcs!CStdContextActivator::CreateInstance+0x221
27e1d2e0 77727f8a 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
27e1d310 776f5c55 0009a0f8 00000000 27e1d89c ole32!CSurrogateProcessActivator::CreateInstance+0xf7
27e1d344 776aaf9e 7779487c 00000000 27e1d89c ole32!CClientContextActivator::CreateInstance+0xc9
27e1d384 776ab12f 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
27e1d68c 776a67ba 361f8c54 00000000 00000015 ole32!ICoCreateInstanceEx+0x3f8
27e1d6c0 7769b9b3 361f8c54 00000000 00000000 ole32!CComActivator::DoCreateInstance+0x6a
27e1ddc8 4a7516d8 45bfeb58 361f8c54 00000000 ole32!CComActivator::StandardCreateInstance+0x7c
27e1de3c 4a751fc6 00000000 361f8c40 00000000 comsvcs!CClassFactoryWrapper::ActivateOnMachine+0xaf
27e1de74 7a078d9b 361f8c40 00000000 79edda70 comsvcs!CClassFactoryWrapper::CreateInstance+0x80
27e1e118 7a07a1e6 361f8c40 00000000 00000000 mscorwks!ComClassFactory::CreateInstanceFromClassFactory+0x102
27e1e174 7a0bf10a 1e772994 00000000 27e1e1b4 mscorwks!ComClassFactory::CreateInstance+0x91
27e1e184 7a0c870b 1e772994 1e772994 79faa672 mscorwks!AllocateComObject_ForManaged+0x1e
27e1e1b4 79e9c82b 1e772994 234b20e8 79faa737 mscorwks!AllocateObject+0x38
27e1e1c0 79faa737 05ff2627 00000001 072336fc mscorwks!MethodTable::Allocate+0x35
27e1e260 792c25c3 1e772994 27e1e278 792c257c mscorwks!CRemotingServices::AllocateUninitializedObject+0xdf
27e1e278 792c1951 0f5b1b50 27e1e2d0 03100120 mscorlib_ni+0x2025c3
27e1e290 792c143e 44194464 00000000 00000016 mscorlib_ni+0x201951
27e1e2ac 79e71e04 00000001 27e1e334 79faa569 mscorlib_ni+0x20143e
27e1e2b8 79faa569 792c1400 072336fc 00000000 mscorwks!CTPMethodTable__CallTargetHelper3+0xf
27e1e334 79faa5d8 072336fc 00000000 00000001 mscorwks!CTPMethodTable::CallTarget+0xdd
27e1e348 79faa614 790fd65c 072336fc 00000000 mscorwks!CTPMethodTable::CallTarget+0x14
27e1e368 7a04b578 1e772994 00000000 00000001 mscorwks!CRemotingServices::CreateProxyOrObject+0x38
27e1e414 1e923c99 0f5b1a58 0f5b1b1c 27e1e464 mscorwks!JIT_NewCrossContextHelper+0xa9
WARNING: Frame IP not in any known module. Following frames may be wrong.
27e1e430 776e2fae 00000000 00000064 00000001 0x1e923c99
27e1e464 79e71b4c 27e1ef70 00000064 00000001 ole32!CoGetContextToken+0x29
27e1e494 79e821b9 27e1ee70 00000008 27e1ee20 mscorwks!CallDescrWorker+0x33
27e1e514 7a0f851b 27e1ee70 00000008 27e1ee20 mscorwks!CallDescrWorkerWithHandler+0xa3
27e1e5e4 79270454 00000001 00000000 00000000 mscorwks!CMessage::Dispatch+0x162
27e1e63c 7977c16e 00000001 00000000 075d788c mscorlib_ni+0x1b0454
27e1e658 6744d146 0752309c 0f5b19d8 0752304c mscorlib_ni+0x6bc16e
27e1e690 197cb7c7 0d0e3640 79e7a6b8 27e1ea80 System_EnterpriseServices_ni+0x5d146
27e1e70c 197f3d04 27e1eae4 0071f13b 36c6b460 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2b7c7)
27e1e74c 77720df0 27e1eae4 35ccbed0 00000000 System_EnterpriseServices_Wrapper_197f0000!System::EnterpriseServices::Thunk::FilteringCallbackFunction+0x44
27e1e798 7772189c 00000000 35ccbed0 197f3cc0 ole32!EnterForCallback+0xc4
27e1e8f8 776f0418 27e1e7d0 197f3cc0 27e1eae4 ole32!SwitchForCallback+0x1a3
27e1e924 7769c194 35ccbed0 197f3cc0 27e1eae4 ole32!PerformCallback+0x54
27e1e9bc 776e316c 36c6b460 197f3cc0 27e1eae4 ole32!CObjectContext::InternalContextCallback+0x159
27e1ea0c 79e71d8b 36c6b470 197f3cc0 27e1eae4 ole32!CObjectContext::ContextCallback+0x85
27e1ea68 197cbc82 776e30e7 0f5b1a28 072323f0 mscorwks!PInvokeCalliReturnFromCall
27e1ea88 197cbc82 0010c8f0 00000000 00000008 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2bc82)
27e1eab4 197cbb51 07231a74 27e1eb2c 79282f85 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2bc82)
27e1eac0 79282f85 27e1eb44 00000000 00000000 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2bb51)
27e1eb2c 797f5a0d 03100788 072323f0 075d7830 mscorlib_ni+0x1c2f85
00000000 00000000 00000000 00000000 00000000 mscorlib_ni+0x735a0d
And all other threads waiting for thread 142 have same stack trace.
I don't know what thread 141 is doing. Thread 142 cannot open connection to xls file. And not sure why other threads wait for thread 142.
Maybe you have any ideas?
EDIT:
analyze -v -hang
GetPageUrlData failed, server returned HTTP status 404
URL requested: http://watson.microsoft.com/StageOne/dllhst3g_exe/5_2_3790_3959/unknown/0_0_0_0/00000000.htm?Retriage=1
FAULTING_IP:
+1e02faf0157df58
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 0000008d
BUGCHECK_STR: HANG
PROCESS_NAME: dllhst3g.exe
ERROR_CODE: (NTSTATUS) 0xcfffffff -
EXCEPTION_CODE: (NTSTATUS) 0xcfffffff -
MOD_LIST:
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0xc2c (17)
Current frame:
ChildEBP RetAddr Caller,Callee
1974f450 655f1110 (MethodDesc 0x651b7efc +0x30 System.Data.ProviderBase.WrappedIUnknown.ReleaseHandle())
1974f93c 792e5e4f (MethodDesc 0x79107064 +0xf System.Runtime.InteropServices.SafeHandle.Dispose(Boolean))
1974f944 792e5d6b (MethodDesc 0x79107030 +0x1b System.Runtime.InteropServices.SafeHandle.Finalize())
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
0 1d14.71c Speculated (Triage) -->
17 1d14.c2c Critical Section -->
141 1d14.1b38 Event
WAIT_CHAIN_COMMAND: ~0s;k;;~17s;k;;~141s;k;;
BLOCKING_THREAD: 00001b38
DEFAULT_BUCKET_ID: APPLICATION_HANG_WRONG_SYMBOLS
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_WRONG_SYMBOLS
LAST_CONTROL_TRANSFER: from 7c827b89 to 7c82847c
STACK_TEXT:
2a2efdcc 7c827b89 77e6202c 00000003 2a2efe1c ntdll!KiFastSystemCallRet
2a2efdd0 77e6202c 00000003 2a2efe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
2a2efe78 7739bbd1 00000003 2a2efea0 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
2a2efed4 7739ce36 00000002 2a2eff74 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141
2a2efef0 4a77cb28 00000002 2a2eff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
2a2eff84 77bcb530 33c23fe8 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
2a2effb8 77e6482f 37e3b7e8 00000000 00000000 msvcrt!_endthreadex+0xa3
2a2effec 00000000 77bcb4bc 37e3b7e8 00000000 kernel32!BaseThreadStart+0x34
FOLLOWUP_IP:
ntdll!KiFastSystemCallRet+0
7c82847c c3 ret
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: ntdll!KiFastSystemCallRet+0
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: ntdll
IMAGE_NAME: ntdll.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4cc18322
STACK_COMMAND: ~141s ; kb
BUCKET_ID: HANG_ntdll!KiFastSystemCallRet+0
FAILURE_BUCKET_ID: APPLICATION_HANG_WRONG_SYMBOLS_cfffffff_ntdll.dll!KiFastSystemCallRet
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/dllhst3g_exe/5_2_3790_3959/45d69678/unknown/0_0_0_0/bbbbbbb4/cfffffff/00000000.htm?Retriage=1
Followup: MachineOwner
Not sure I understand it. But thread 17 is finalizer thread. So looks like it is blocked? And I thinkg it as again some kind of access to xls?
~17s
kb
ChildEBP RetAddr Args to Child
1974f08c 7c827b99 7c83d09c 00003a98 00000000 ntdll!KiFastSystemCallRet
1974f090 7c83d09c 00003a98 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
1974f0cc 7c83d0e7 00003a98 00000004 00000001 ntdll!RtlpWaitOnCriticalSection+0x1a3
1974f0ec 2ae1fa94 29202124 32787727 00000000 ntdll!RtlEnterCriticalSection+0xa8
WARNING: Stack unwind information not available. Following frames may be wrong.
1974f160 4c856487 2263eff4 00000000 00000005 ACEOLEDB!DllGetClassObject+0xdc2d
1974f174 4c862d06 2263eff4 4c8633c4 2263efe8 oledb32!CACMDynamic::CmFinalRelease+0x50
1974f17c 4c8633c4 2263efe8 2263efe8 1974f1e4 oledb32!CDCM::FinalRelease+0x1b
1974f198 4c8633f6 00000000 2263efe8 1974f1b8 oledb32!ATL::CComPolyObject::~CComPolyObject+0x2a
1974f1a8 4c88d5ad 2263efe8 2370a39c 1974f1f4 oledb32!ATL::CComPolyObject::Release+0x25
1974f1b8 4a757681 1a271fa0 2263efe8 2370a39c oledb32!CDCMCreator::DestroyResource+0xf
1974f1f4 4a75793e 2370a39c 2263efe8 00000000 comsvcs!CHolder::SafeDispenserDriver::DestroyResource+0x20
1974f20c 4a758a35 23151cd0 00000000 2fa44808 comsvcs!CHolder::ProcessDestroyList+0x2e
1974f238 4c88d30e 23151cd0 2263efe8 2263efe8 comsvcs!CHolder::FreeResource+0x7f
1974f268 4c878a3a 2263efe8 00000000 271aa824 oledb32!CDCMCreator::ReleaseResource+0x31
1974f288 4c8545e7 271aa824 00000000 4c85456a oledb32!CDPO::ReturnDCMToPool+0x89
1974f294 4c85456a 271aa818 271aa818 1974f320 oledb32!CDPO::FinalRelease+0xb
1974f2b0 4c88e78a 271aa818 1974f2d0 4c85218c oledb32!ATL::CComPolyObject::~CComPolyObject+0x2a
1974f2bc 4c85218c 00000001 00000000 000eaaf8 oledb32!ATL::CComPolyObject::`scalar deleting destructor'+0xd
1974f2d0 4c8521aa 271aa818 1974f330 79f63ff9 oledb32!ATL::CComPolyObject::Release+0x27
1974f2dc 79f63ff9 271aa824 3b6a3777 000eaaf8 oledb32!ATL::CComContainedObject::Release+0x11
1974f330 79f640ac 271aa828 3b6a3733 000eaaf8 mscorwks!ReleaseTransitionHelper+0x5f
1974f374 79f64110 271aa828 00000000 3b6a37ef mscorwks!SafeReleaseHelper+0x8c
1974f3a8 79f53b5d 271aa828 00000000 3b6a3017 mscorwks!SafeRelease+0x2f
1974f450 655f1110 000eaaf8 00000000 1974f470 mscorwks!MarshalNative::Release+0xb0
1974f460 79e71b4c 1974f4dc 000c5958 1974f4f0 System_Data_ni!_bidW103 (System_Data_ni+0x4a1110)
1974f470 79e821b9 1974f540 00000000 1974f510 mscorwks!CallDescrWorker+0x33
1974f4f0 79e96531 1974f540 00000000 1974f510 mscorwks!CallDescrWorkerWithHandler+0xa3
1974f634 79e96564 655f10e0 1974f6bc 1974f6a4 mscorwks!MethodDesc::CallDescr+0x19c
1974f650 79e96c4c 655f10e0 1974f6bc 1974f6a4 mscorwks!MethodDesc::CallTargetWorker+0x1f
1974f670 79eccd2a 1974f6a4 0b402780 00000004 mscorwks!MethodDescCallSite::Call_RetObjPtr+0x1c
1974f770 79eccd5e 0b402780 000eaaf8 000eaaf8 mscorwks!SafeHandle::RunReleaseMethod+0x89
1974f870 79eccbf4 00000001 0d0e3640 79e7a1c8 mscorwks!SafeHandle::Release+0x11b
1974f89c 79f83999 3b6a3d7b 00000000 000eaaf8 mscorwks!SafeHandle::Dispose+0x23
1974f93c 792e5e4f 1974f970 792e5d6b 0b3ac100 mscorwks!SafeHandle::Finalize+0xab
1974f944 792e5d6b 0b3ac100 ffffffff 00000000 mscorlib_ni+0x225e4f
1974f970 79f7169a 1974f9c4 79ef465c 1974fc1c mscorlib_ni+0x225d6b
1974f9d4 79f7175b 0b402780 792e5d50 08000000 mscorwks!MethodTable::GetObjCreateDelegate+0xaf
1974f9f4 79f71609 0b402780 0b402780 00000000 mscorwks!MethodTable::CallFinalizer+0xa0
1974fa08 79fd46a6 0b402780 3b6a3e1f 00000000 mscorwks!SVR::CallFinalizer+0xa6
1974fa58 79fd45d7 000eaaf8 00000415 1974fadf mscorwks!SVR::GCHeap::TraceGCSegments+0x1b0
1974fae0 79f5832f 0740eeb4 00000000 1974feb0 mscorwks!SVR::GCHeap::TraceGCSegments+0x2f6
1974faf4 79e9848f 1974fd04 000eaaf8 00000000 mscorwks!SVR::ProfScanRootsHelper+0x69
1974fb08 79e9842b 1974feb0 1974fb90 79fa6a6b mscorwks!Thread::DoADCallBack+0x32a
1974fb9c 79e98351 1974feb0 3b6a3f9f 000eaaf8 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
1974fbd8 79ec4322 1974feb0 00000000 1974fc98 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
1974fbe8 79f581f3 1974feb0 1974fc8c 79fa6a6b mscorwks!Thread::RaiseCrossContextException+0x434
1974fc98 79f58279 1a9cb678 79ec430e 1974feb0 mscorwks!Thread::DoADCallBack+0xcd
1974fcb8 79f58265 1974feb0 1974fd20 79f582e4 mscorwks!Thread::DoADCallBack+0x322
1974fcc4 79f582e4 1a9cb678 79f58316 1974fd04 mscorwks!ManagedThreadBase::FinalizerAppDomain+0x25
1974fd20 79fd45d7 000eaaf8 00000000 1974fda7 mscorwks!SVR::GCHeap::TraceGCSegments+0x251
1974fda8 79fd48a8 00000000 00000000 1974feb0 mscorwks!SVR::GCHeap::TraceGCSegments+0x2f6
1974fdc0 79e9848f 1974feb0 00000000 00000000 mscorwks!SVR::GCHeap::FinalizerThreadWorker+0xb7
1974fdd4 79e9842b 1974feb0 1974fe5c 79fa6a6b mscorwks!Thread::DoADCallBack+0x32a
1974fe68 79e98351 1974feb0 3b6a3ae3 00000000 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
1974fea4 79f074d4 1974feb0 00000000 1a9cb678 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
1974fecc 79f074e5 79fd4809 00000008 1974ff14 mscorwks!ManagedThreadBase_NoADTransition+0x32
1974fedc 79f090b3 79fd4809 3b6a3b53 00000000 mscorwks!ManagedThreadBase::FinalizerBase+0xd
1974ff14 79f75715 00000000 00000007 ffffffff mscorwks!SVR::GCHeap::FinalizerThreadStart+0xbb
1974ffb8 77e6482f 000eb528 00000000 00000000 mscorwks!Thread::intermediateThreadProc+0x49
1974ffec 00000000 79f756cf 000eb528 00000000 kernel32!BaseThreadStart+0x34
You need to find who owns the following critical section that your thread is waiting on:
1974f0ec 2ae1fa94 29202124 32787727 00000000 ntdll!RtlEnterCriticalSection+0xa8
You can do an automated critical section analysis with call stacks:
!locks -v
This will dump all critical section locks that are in a locked state and the call stacks of the threads, you then need to scan each call stack for each lock to see if say Thread A is waiting on lock 1 which is owned by Thread B, Thread B is waiting on lock 2 which is owned by Thread A.
Hope this helps