PowerShell Job throttle blocking the server - powershell

Question

I'm experiencing a strange issue with our script server being overloaded and running out of resources. We have a script that copies data from one location to another, this is defined in a large input file that contains over 200 lines of text in the format 'Source path, Destination path'.
We are now in the process of trying to throttle the maximum jobs we kick of at once and I think it's working fine. But for some reason or another we're still running out of resources on the server when the input file contains over 94 lines. This became apparent after some testing.
We tried to upgrade our Windows 2008 R2 server with PowerShell 4.0 to 4 processors and 8 GB of RAM, but no luck. So I assume my throttling isn't working as designed.
Error code:
Insufficient system resources exist to complete the requested service.
The code:
$MaxThreads = 4
$FunctionFeed = Import-Csv -Path $File -Delimiter ',' -Header 'Source', 'Destination'
$Jobs=#()
Function Wait-MaxRunningJobs {
Param (
$Name,
[Int]$MaxThreads
)
Process {
$Running = #($Name | where State -eq Running)
while ($Running.Count -ge $MaxThreads) {
$Finished = Wait-Job -Job $Name -Any
$Running = #($Name | where State -eq Running)
}
}
}
$ScriptBlock = {
Try {
Robocopy.exe $Using:Line.Source $Using:Line.Destination $Using:Line.File /MIR /Z /R:3 /W:15 /NP /MT:8 | Out-File $Using:LogFile
[PSCustomObject]#{
Source = if ($Using:Line.Source) {$Using:Line.Source} else {'NA'}
Target = if ($Using:Line.Destination) {$Using:Line.Destination} else {'NA'}
}
}
Catch {
"Robocopy | ERROR: $($Error[0].Exception.Message)" |
Out-File -LiteralPath $Using:LogFile
throw $($Error[0].Exception.Message)
}
}
ForEach ($Line in $FunctionFeed) {
$LogParams = #{
LogFolder = $LogFolder
Name = $Line.Destination + '.log'
Date = 'ScriptStartTime'
Unique = $True
}
$LogFile = New-LogFileNameHC #LogParams
' ' >> $LogFile # Avoid not being able to write to log
$Jobs += Start-Job -Name RoboCopy -ScriptBlock $ScriptBlock
Wait-MaxRunningJobs -Name $Jobs -MaxThreads $MaxThreads
}
if ($Jobs) {
Wait-Job -Job $Jobs
$JobResults = $Jobs | Receive-Job
}
Am I missing something here? Thank you for your help.

Answer 1

You're using background jobs, which actually run in remote sessions on the local machine. Remote sessions are intentionally resource restricted, according to settings set in the session configuration. You can check the current settings using
Get-PSSessionConfiguration
And adjust the settings to increase the resources available to the sessions with
Set-PSSessionConfiguration
You may need to do some testing to determine exactly what resource limit you're hitting, and what adjustments need to be made for this particular application to work.

Answer 2

Fixed the problem by enlarging the MaxMemoryPerShellMB for remote sessions from 1GB to 2 GB as described here. Keep in mind that Start-Job is using a remote PowerShell session as mjolinor already indicated, so this variable is applicable to PowerShell jobs.
Solution:
# 'System.OutOfMemoryException error message' when running Robocopy and over 94 PowerShell-Jobs:
Get-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB # Default 1024
Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 2048
# Set PowerShell plugins memory from 1 GB to 2 GB
Get-Item WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxMemoryPerShellMB # Default 1024
Set-Item WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxMemoryPerShellMB 2048
Restart-Service winrm