Powershell crashing due to memory constants - powershell

Question

I've got a server running around 500 powershell processes. Each of these processes are designed to make WMI calls across our environment. I've been careful to verify that I do not use up all of the server's available memory or CPU. When I have all 500 processes running, I'm at around 70% memory usage.
Just in case anybody is wondering how the individual processes are handled, they are executed using a gearman job worker. Basically a shell python script that calls a powershell script...times 500.
The issue i'm running into is that some of my powershell processes are crashing after running a few hours.
Some of the errors that I'm getting are:
A new guard page for the stack cannot be created
When I open event viewer, I see these events when processes crash
Fault bucket , type 0
Event Name: PowerShell
Response: Not available
Cab Id: 0
Problem signature:
P1: powershell.exe
P2: 6.3.9600.16394
P3: System.OutOfMemoryException
P4: System.OutOfMemoryException
P5: oft.PowerShell.ConsoleHost.ReportExceptionFallback
P6: lization.EncodingTable.nativeCreateOpenFileMapping
P7: Consol.. main thread
P8:
P9:
P10:
Attached files:
These files may be available here:
C:\path
Analysis symbol:
Rechecking for solution: 0
Report Id: ID
Report Status: 2048
Hashed bucket:
I'm guessing it has something to do with powershell running out of memory, but the server is not peaked, and not all processes crash, it is sporadic.
Any help would be appreciated.
Here are more crash results, the powershell fault module names are different from time to time:
Problem Event Name: APPCRASH
Application Name: powershell.exe
Application Version: 6.3.9600.16384
Application Timestamp: 52158733
Fault Module Name: ntdll.dll
Fault Module Version: 6.3.9600.16408
Fault Module Timestamp: 523d45fa
Exception Code: c00000fd
Exception Offset: 00069abb
OS Version: 6.3.9600.2.0.0.272.7
Locale ID: 1033
Additional Information 1: 624b
Additional Information 2: 624b484d3cf74536f98239c741379147
Additional Information 3: a901
Additional Information 4: a901f876e92d1eb79eb3a513defef0c6
Problem signature:
Problem Event Name: APPCRASH
Application Name: powershell.exe
Application Version: 6.3.9600.16384
Application Timestamp: 52158733
Fault Module Name: combase.dll
Fault Module Version: 6.3.9600.16408
Fault Module Timestamp: 523d3001
Exception Code: c00000fd
Exception Offset: 0001a360
OS Version: 6.3.9600.2.0.0.272.7
Locale ID: 1033
Additional Information 1: 81ca
Additional Information 2: 81cae32566783b059420874b47802c3e
Additional Information 3: b637
Additional Information 4: b6375e6f6a866fc9d00393d4649231b8

Answer 1

have you looked at your max memory allocation per shell?
get-item WSMan:\localhost\Shell\MaxMemoryPerShellMB
and if its too low changing this;
set-item WSMan:\localhost\Shell\MaxMemoryPerShellMB 2048

Answer 2

Doesn't .Net have a limit of memory?

Answer 3

If you're using TaskManager to check on memory usage, you might try Process Explorer instead. It sometimes gives very different results.

Answer 4

Thanks everyone for the responses, it turns out that I had a memory leak in my powershell code that was causing memory usage to spike every now and then. Since I was not watching the server at every second, I missed when the memory usage spiked.
An interesting note, it appears that Powershell will not use more then 80% of available memory on a server before killing its own processes.
I had to increase the available memory to 56GB and now I'm not running into any issues whatsoever. I've been running 600 powershell processes for a week now and have not had one crash on me.