Powershell Job Memory Consumption Issue - powershell

Question

I've been struggling with this for a week now and have exhausted all the methods and options I have found online. I am hoping someone here will be able to help me out with this.
I am using powershell to start 8 jobs, each job running FFmpeg to stream a 7 minute file to a remote RTMP server. This is pulling from a file on the disk and each job uses a different file. The command is in a do while loop so that it is constantly restreaming.
This is causing the shell I launched the jobs from to accumulate a massive amount of memory, consuming all that it can. In 24 hours it consumed 30 of the 32 GB of my server.
Here is my launch code, any help would be appreciated.
start-job -Name v6 -scriptblock {
do { $d = $true; $f = Invoke-Expression -Command "ffmpeg -re -i `"C:\Shares\Matthew\180p_3000k.mp4`" -vcodec copy -acodec copy -f flv -y rtmp://<ip>/<appName>/<streamName>"; $f = $null }
while ($d = $true)
}
I've tried to receive the jobs and pipe it to out-null, I've tried setting $f to $null before starting the do while loop, and some other things I found online but to no avail. Thanks everyone for your time!

Answer 1

Better late than never I guess. I've had the same problem with huge memory consumption when running ffmpeg in Powershell jobs. The core of the issue is that a Powershell job will store any/all output into memory, and ffmpeg is extremely happy to log output to both standard output and standard error streams.
My solution was to add the parameter "-loglevel quiet" to ffmpeg. Alternatively you could redirect both the standard and error streams to null (it's not enough to redirect just the standard stream). For more on how to redirect the standard streams, refer to this question: Redirection of standard and error output appending to the same log-file

Answer 2

There are many ways to invoke a PowerShell script or an expression.
One of my favorites is using RunspacePool. In a RunspacePool each task is started with a new runspace. When the task completes the runspace is disposed of. This should keep memory consumption fairly constant. It is not the easiest method to get your head around, but here is a bare-bone example that might work for you.
$Throttle = 10 # Maximum jobs that can run simultaneously in a runpool
$maxjobs = 8 # Maximum jobs that will run simultaneously
[System.Collections.ArrayList]$jobs = #()
$script:currentjobs = 0
$jobindex = 0
$jobaction = "ffmpeg -re -i `"C:\Shares\Matthew\180p_3000k.mp4`" -vcodec copy -acodec copy -f flv -y rtmp://<ip>/<appName>/<streamName>"
$sessionstate = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
$runspace = [runspacefactory]::CreateRunspacePool(1, $Throttle, $sessionstate, $Host)
$runspace.Open()
function QueueJob{
param($jobindex)
$job = "" | Select-Object JobID,Script,Errors,PS,Runspace,Handle
$job.JobID = $jobindex
$job.Script = $jobaction
$job.PS = [System.Management.Automation.PowerShell]::create()
$job.PS.RunspacePool = $runspace
[void]$job.PS.AddScript($job.Script)
$job.Runspace = $job.PS.BeginInvoke()
$job.Handle = $job.Runspace.AsyncWaitHandle
Write-Host "----------------- Starting Job: $("{0:D4}" -f $job.JobID) --------------------" -ForegroundColor Blue
$jobs.add($job)
$script:currentjobs++
}
Function ServiceJobs{
foreach($job in $Jobs){
If ($job.Runspace.isCompleted) {
$result = $job.PS.EndInvoke($job.Runspace)
$job.PS.dispose()
$job.Runspace = $null
$job.PS = $null
$script:currentjobs--
Write-Host "----------------- Job Completed: $("{0:D4}" -f $job.JobID) --------------------" -ForegroundColor Yellow
Write-Host $result
$Jobs.Remove($job)
return
}
}
}
while ($true) {
While ($script:currentjobs -le $maxjobs){
QueueJob $jobindex
$jobindex++
}
ServiceJobs
Start-Sleep 1
}
$runspace.Close()
[gc]::Collect()
It is using an infinite loop without proper termination and is lacking any sort of error checking, but hopefully it is enough to demonstrate the technique.