I have been getting a stackoverflow exception in my program which may be originating from a thirdparty libary, microsoft.sharepoint.client.runtime.dll.
Using adplus to create the crash dump, I'm facing the problem that I'm struggling to get any information from it when i open it in windbg. This is what I get as a response:
> 0:000> .restart /f
Loading Dump File [C:\symbols\FULLDUMP_FirstChance_epr_Process_Shut_Down_DocumentumMigrator.exe__0234_2011-11-17_15-19-59-426_0d80.dmp]
User Mini Dump File with Full Memory: Only application data is available
Comment: 'FirstChance_epr_Process_Shut_Down'
Symbol search path is: C:\symbols
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Machine Name:
Debug session time: Thu Nov 17 15:19:59.000 2011 (UTC + 2:00)
System Uptime: 2 days 2:44:48.177
Process Uptime: 0 days 0:13:05.000
.........................................WARNING: rsaenh overlaps cryptsp
.................WARNING: rasman overlaps apphelp
......
..WARNING: webio overlaps winhttp
.WARNING: credssp overlaps mswsock
.WARNING: IPHLPAPI overlaps mswsock
.WARNING: winnsi overlaps mswsock
............
wow64cpu!CpupSyscallStub+0x9:
00000000`74e42e09 c3 ret
Any ideas as to how i can get more information from the dump, or how to use it to find where my stackoverflow error is occuring?
The problem you are facing is that the process is 32-bit, but you are running on 64-bit, therefore your dump is a 64-bit dump. To make use of the dump you have to run the following commands:
.load wow64exts
.effmach x86
!analyze -v
The last command should give you a meaningful stack trace.
This page provides lots of useful information and method to analyze the problem.
http://www.dumpanalysis.org/blog/index.php/2007/09/11/crash-dump-analysis-patterns-part-26/
You didn't mention if your code is managed or unmanaged. Assuming it is unmanaged. In debugger:
.symfix
.reload
~*kb
Look through the call stack for all threads and identify thread that caused SO. It is easy to identify the thread with SO, because the call stack will be extra long. Switch to that thread using command ~<N>s, where is thread number, dump more of the call stack using command k 200 to dump up to 200 lines of call stack. At the very bottom of the call stack you should be able to see the code that originated the nested loop.
If your code is managed, use SOS extension to dump call stacks.
Related
I am trying to hit an external web service using Progress OE 11.5. When I execute the code from GUI Procedure Editor or the CHUI Procedure Editor, it crashes when calling the API
RUN ProcessTrack IN hTrackPortType(INPUT lcRequest, OUTPUT lcResponse) no-error.
I don't get any errors. The progress GUI window just crashes.
When I traced the logs it has "C0000005 ACCESS_VIOLATION" exception. Any idea why this is caused? But when I access the same web services from SoapUI or from a Python program it works fine. I am not sure if Progress OpenEdge has any access restrictions to contact the apis.
I have the full stack trace here.
=====================================================
PROGRESS stack trace as of Fri Aug 07 12:26:40 2020
=====================================================
Progress OpenEdge Release 11.5 build 1114 on WINNT
Startup parameters:
-pf C:\Progressx86\OpenEdge\startup.pf,-cpinternal ISO8859-1,-cpstream ISO8859-1,-cpcoll Basic,-cpcase Basic,-d mdy,-numsep 44,-numdec 46,(end .pf),-param C:\.....\api_request.p
Exception code: C0000005 ACCESS_VIOLATION
Fault address: 025C21CC 1C3:0034002D
Registers:
EAX:086496B8
EBX:00000002
ECX:03100000
EDX:03100000
ESI:59DF2175
EDI:085AA1E0
CS:EIP:0023:025C21CC
SS:ESP:002B:00F4BFA0 EBP:00F4BFD0
DS:002B ES:002B FS:0053 GS:002B
Flags:00210206
Debugging dll: C:\Progressx86\OpenEdge\bin\DBGHELP.DLL
Symbol Path:
C:\Progressx86\OpenEdge\bin;C:\Progressx86\OpenEdge\pdbfiles
Call Stack:
Address Frame
025C21CC 00F4BF9C 0000:00000000
085AA1E0 00F4BFD0 0000:00000000
59DF27DB 00F4BFDC WSDLAttribute::getHandle+3F52B
59DA415C 00F4F130 WSDLArray_Empty+23ABC
59DCD9A0 00F4F144 WSDLAttribute::getHandle+1A6F0
59E502A8 00F4F198 WSDLAttribute::getHandle+9CFF8
59E5032C 00F4F1CC WSDLAttribute::getHandle+9D07C
59DCF04E 00F4F204 WSDLAttribute::getHandle+1BD9E
59D9B724 00F4F240 WSDLArray_Empty+1B084
59E50403 00F4F260 WSDLAttribute::getHandle+9D153
59D55D6F 00F4F2A0 csp_tweakFileURL+312F
** ABL Stack Trace **
--> C:\....\p56215_api_request.ped at line 54 (C:\.....\p56215_api_request.ped)
adecomm/_runcode.p at line 665 (adecomm/_runcode.r)
ExecuteRun adeedit/_proedit.p at line 3613 (adeedit/_proedit.r)
RunFile adeedit/_proedit.p at line 10625 (adeedit/_proedit.r)
USER-INTERFACE-TRIGGER adeedit/_proedit.p at line 1985 (adeedit/_proedit.r)
adeedit/_proedit.p at line 12280 (adeedit/_proedit.r)
_edit.p at line 408 (C:\Progressx86\OpenEdge\gui\_edit.r)
** Persistent procedures/Classes **
** PROPATH **
.,C:\Progressx86\OpenEdge\gui,C:\Progressx86\OpenEdge\gui\ablunit.pl,C:\Progressx86\OpenEdge\gui\adecomm.pl,C:\Progressx86\OpenEdge\gui\adecomp.pl,C:\Progressx86\OpenEdge\gui\adedesk.pl,C:\Progressx86\OpenEdge\gui\adedict.pl,C:\Progressx86\OpenEdge\gui\adeedit.pl,C:\Progressx86\OpenEdge\gui\adeicon.pl,C:\Progressx86\OpenEdge\gui\aderes.pl,C:\Progressx86\OpenEdge\gui\adeshar.pl,C:\Progressx86\OpenEdge\gui\adeuib.pl,C:\Progressx86\OpenEdge\gui\adeweb.pl,C:\Progressx86\OpenEdge\gui\adexml.pl,C:\Progressx86\OpenEdge\gui\dataadmin.pl,C:\Progressx86\OpenEdge\gui\OpenEdge.BusinessLogic.pl,C:\Progressx86\OpenEdge\gui\OpenEdge.Core.pl,C:\Progressx86\OpenEdge\gui\OpenEdge.ServerAdmin.pl,C:\Progressx86\OpenEdge\gui\prodict.pl,C:\Progressx86\OpenEdge\gui\protools.pl,C:\Progressx86\OpenEdge,C:\Progressx86\OpenEdge\bin
** Databases (logical/type/physical) **
** End of Protrace **
This KnowledgeBase post indicates that this is a known error. If you run a version below 11.7.1 you should consider upgrading to the latest version of 11.7 (currently 11.7.6). If you run a version later than 11.7.1 that's mentioned in the article you should consider contacting Progress support.
EDIT: since running 11.5 upgrading should be a priority!
Yesterday we had crash of PostgreSQL 9.5.14 running on Debian 8 (Linux xxxxxx 3.16.0-7-amd64 #1 SMP Debian 3.16.59-1 (2018-10-03) x86_64 GNU/Linux) - Segmentation fault. Database closed all connections and reinitialized itself staying ~1 minute in recovery mode.
PostgreSQL log:
2018-10-xx xx:xx:xx UTC [580-2] LOG: server process (PID 16461) was
terminated by signal 11: Segmentation fault
kern.log:
Oct xx xx:xx:xx xxxxxxxx kernel: [117977.301353] postgres[16461]:
segfault at 7efd3237db90 ip 00007efd3237db90 sp 00007ffd26826678 error
15 in libc-2.19.so[7efd322a2000+1a1000]
According to libc documentation (https://support.novell.com/docs/Tids/Solutions/10100304.html) error code 15 means:
NX_EDEADLK 15 resource deadlock would occur - which does not tell me much.
Could you tell me please if we can do something to avoid this problem in the future? Because this server is of course production one.
All packages are up to date currently. Upgrade of PG is unfortunately not the option. Server runs on Google Compute Engine.
error code 15 means: NX_EDEADLK 15
No, it doesn't mean that. This answer explains how to interpret 15 here.
It's bits 0, 1, 2, 3 set => protection fault, write access, user mode, use of reserved bit. Most likely your postgress process attempted to write to some wild pointer.
if we can do something to avoid this problem in the future?
The only thing you can do is find the bug and fix it, or upgrade to a release of postgress where that bug is already fixed (and hope that no new ones were introduced).
To understand where the bug might be, you should check whether a core dump was produced (if not, do enable them). If you have the core, use gdb /path/to/postgress /path/to/core, and then where GDB command. That will give you crash stack trace, which may allow you to find similar bug reports.
I've got a server running around 500 powershell processes. Each of these processes are designed to make WMI calls across our environment. I've been careful to verify that I do not use up all of the server's available memory or CPU. When I have all 500 processes running, I'm at around 70% memory usage.
Just in case anybody is wondering how the individual processes are handled, they are executed using a gearman job worker. Basically a shell python script that calls a powershell script...times 500.
The issue i'm running into is that some of my powershell processes are crashing after running a few hours.
Some of the errors that I'm getting are:
A new guard page for the stack cannot be created
When I open event viewer, I see these events when processes crash
Fault bucket , type 0
Event Name: PowerShell
Response: Not available
Cab Id: 0
Problem signature:
P1: powershell.exe
P2: 6.3.9600.16394
P3: System.OutOfMemoryException
P4: System.OutOfMemoryException
P5: oft.PowerShell.ConsoleHost.ReportExceptionFallback
P6: lization.EncodingTable.nativeCreateOpenFileMapping
P7: Consol.. main thread
P8:
P9:
P10:
Attached files:
These files may be available here:
C:\path
Analysis symbol:
Rechecking for solution: 0
Report Id: ID
Report Status: 2048
Hashed bucket:
I'm guessing it has something to do with powershell running out of memory, but the server is not peaked, and not all processes crash, it is sporadic.
Any help would be appreciated.
Here are more crash results, the powershell fault module names are different from time to time:
Problem Event Name: APPCRASH
Application Name: powershell.exe
Application Version: 6.3.9600.16384
Application Timestamp: 52158733
Fault Module Name: ntdll.dll
Fault Module Version: 6.3.9600.16408
Fault Module Timestamp: 523d45fa
Exception Code: c00000fd
Exception Offset: 00069abb
OS Version: 6.3.9600.2.0.0.272.7
Locale ID: 1033
Additional Information 1: 624b
Additional Information 2: 624b484d3cf74536f98239c741379147
Additional Information 3: a901
Additional Information 4: a901f876e92d1eb79eb3a513defef0c6
Problem signature:
Problem Event Name: APPCRASH
Application Name: powershell.exe
Application Version: 6.3.9600.16384
Application Timestamp: 52158733
Fault Module Name: combase.dll
Fault Module Version: 6.3.9600.16408
Fault Module Timestamp: 523d3001
Exception Code: c00000fd
Exception Offset: 0001a360
OS Version: 6.3.9600.2.0.0.272.7
Locale ID: 1033
Additional Information 1: 81ca
Additional Information 2: 81cae32566783b059420874b47802c3e
Additional Information 3: b637
Additional Information 4: b6375e6f6a866fc9d00393d4649231b8
have you looked at your max memory allocation per shell?
get-item WSMan:\localhost\Shell\MaxMemoryPerShellMB
and if its too low changing this;
set-item WSMan:\localhost\Shell\MaxMemoryPerShellMB 2048
Doesn't .Net have a limit of memory?
If you're using TaskManager to check on memory usage, you might try Process Explorer instead. It sometimes gives very different results.
Thanks everyone for the responses, it turns out that I had a memory leak in my powershell code that was causing memory usage to spike every now and then. Since I was not watching the server at every second, I missed when the memory usage spiked.
An interesting note, it appears that Powershell will not use more then 80% of available memory on a server before killing its own processes.
I had to increase the available memory to 56GB and now I'm not running into any issues whatsoever. I've been running 600 powershell processes for a week now and have not had one crash on me.
I'm getting a BSOD on my Windows XP SP3 image (running on Workstation 8) all of a sudden. Here is the message:
stop: 0x000000D1 (0x00000008, 0x00000002, 0x00000000, 0x8616A7c5)
I've tried to debug the D1 error - what do the other values represent? I'm hoping they can point me to a solution to my problem. Any ideas what the last 4 numbers represent?
Make sure that the computer is correctly configured to save a crash dump file. The 1-st parametr - memory referenced 0x00000008 bad address in both user and kernel mode at any IRQL. The 4-th parametr - address that referenced memory 0x8616A7c5. Instruction in a module that is loaded at this address causes an error.
download win-debugger from Microsoft and then open the crash dump files. you could also see the calling stack and know which process cause the BSOD.
Loading Dump File [C:\Crash_Mode__Date_12-05-2009__Time_15-54-2727\PID-4056__CCNET.EXE__1st_chance_Process_Shut_Down__full_13d0_2009-12-06_00-33-14-734_0fd8.dmp]
User Mini Dump File with Full Memory: Only application data is available
Comment: '1st_chance_Process_Shut_Down_exception_in_CCNET.EXE_running_on_TEST218'
Symbol search path is: srvE:\symbolshttp://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Server 2003 Version 3790 (Service Pack 2) MP (2 procs) Free x64
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Machine Name:
Debug session time: Sun Dec 6 00:33:14.000 2009 (GMT+8)
System Uptime: 32 days 12:43:52.414
Process Uptime: 0 days 8:44:37.000
..........................WARNING: mscorjit overlaps mscoree
..............................WARNING: wldap32 overlaps dnsapi
..........WARNING: rasapi32 overlaps dnsapi
...WARNING: tapi32 overlaps rasapi32
.WARNING: rtutils overlaps rasman
..............WARNING: setupapi overlaps winsta
....wow64cpu!CpupSyscallStub+0x9:
00000000`78b842d9 c3 ret
why this happen?
Not related to CLR, instead it's 32-on-64 as described here http://www.dumpanalysis.org/blog/index.php/2007/09/11/crash-dump-analysis-patterns-part-26/ -- in short use the following:
.load wow64exts
.effmach x86
With those, kb and !analyze -v will give better results.
I hadve seen the same thing latly, I do not know for sure but it's probably some WOW64 artifact or possibly due to some more aggressive anti-exploitation techniques. At leat on Win32, even though the load address of a DLL may be different on occasion, if a DLL is mapped in antother process (like ntdll/kernel32) when your process start's if it is statically linking these DLL's also, it would always load at the same address until the next reboot.
It seems probable that more recent CLR exe's are able to remap various module's per-execution, I know that this is an issue on MSVC10 and Windows7 but perhaps it's been ported to additonal platforms for CLR applications.