PowerShell: How 'Receive-Job' pulls output from the job's code block in detail? - powershell

Question

Please have a look at this test script and the conclusions I've made about how 'Receive-Job' works in detail.
I have still issues to figure out, how exaclty 'Receive-Job' pulls the streams from the code block.
<# .SYNOPSIS Test the console output and variable capturing of Write- cmdlet calls in a code block used by 'Start-Job'
.NOTES
.NET Version 4.7.2
PSVersion 5.1.16299.431
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
BuildVersion 10.0.16299.431
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
#>
Set-StrictMode -Version latest
if ($host.Name -inotmatch 'consolehost') { Clear-Host }
$errorBuffer = $null
$warningBuffer = $null
$outBuffer = $null
$infoBuffer = $null
# Start the job
$job = Start-Job -ScriptBlock {
Set-StrictMode -Version latest
PowerShell starts this script block in its own process, like it would start an external executable.
Therfore PowerShell can only map stdout/success and stderr/error from the codeblock to the PowerShell's success (1) and error (2) streams in the script's process.
Those two streams will be passed by Receive-Job and can be redirected in the Receive-Job line as expected.
And those two streams can be stored into variables by Receive-Job on request. (-OutVariable -ErrorVariable)
Additionally, Receive-Job can caputure the PowerShell streams info (stream 6) and warning (stream 3) and can store them in variables, too. (-WarningVariable -InformationVariable)
But storing those streams in the variables is no redirection.
Every call of a Write- cmdlet can display a message on the console, independent to the -variable swiches.
A visible message on the console depends only on the Write- cmdlet's own preferences and possible redirection in the Write- cmdlet call.
# This will, by default, output to the console over stream 6 (info), and always get captured in $infoBuffer.
Write-Host "***WRITE_HOST***" # 6> $null # Supresses the output to the console.
# This will not output to the console over stream 6 (info) by default, but always get captured in $infoBuffer.
$InformationPreference = 'Continue' # Outputs to the console, default is 'SilentlyContinue'.
Write-Information "***INFO***" # 6> $null # Supresses the output to the console for preference 'Continue'.
$InformationPreference = "SilentlyContinue"
# This will not output to the console over stream 5 (debug) by default, and can't get captured in a variable.
$DebugPreference = 'Continue' # Outputs to the console, default is 'SilentlyContinue'.
Write-Debug "***DEBUG***" # 5> $null # Supresses the output to the console for preference 'Continue'.
$DebugPreference = "SilentlyContinue"
# This will not output to the console over stream 4 (verbose), by default, and can't get captured in a variable.
$VerbosePreference = 'Continue' # Outputs to the console, default is 'SilentlyContinue'.
Write-Verbose "***Verbose***" # 4> $null # Supresses the output to the console for preference 'Continue'.
$VerbosePreference = 'SilentlyContinue'
# This will, by default, output to the console over stream 3 (warning), but get captured in $warningBuffer only for
# preference 'Continue'.
#$WarningPreference = 'SilentlyContinue' # Supresses console output AND variable capturing, default is 'Continue'.
Write-Warning "***WARNING***" # 3> $null # Supresses the warning output to the console for preference
#$WarningPreference = 'Continue' # 'Continue'.
# This will output to the console over stream 2 (error), and always get captured in $errorBuffer, if not redirected
# in the code block.
# For 'Receive-Job -ErrorAction Stop' it would raise an execption, the content in $errorBuffer is quite useless then.
Write-Error '***ERROR***' # 2> $null # Supresses the output AND variable capturing, but you can supress/redirect
# this stream in the 'Receive-Job' line without breaking the variable
# capturing: 'Receive-Job ... -ErrorVariable errorBuffer 2> $null'
# These will output to the console over stream 1 (success), and always get captured in $result and $outBuffer, if
# not redirected in the code block.
Write-Output '***OUTPUT***' # 1> $null # Supresses the output AND variable capturing, but you can supress/redirect
Write-Output '***NEXT_OUTPUT***' # this stream in the 'Receive-Job' line without breaking the variable
"***DIRECT_OUT***" # capturing: '$result = Receive-Job ... -OutVariable outBuffer 1> $null'
}
# Wait for the job to finish
Wait-Job -Job $job
try
{
# Working only outside the code block, this is a workaround for catching ALL output.
#$oldOut = [Console]::Out
#$stringWriter = New-Object IO.StringWriter
#[Console]::SetOut($stringWriter)
# Pull the buffers from the code block
$result = Receive-Job <#-ErrorAction Stop#> `
-Job $job `
-ErrorVariable errorBuffer `
-WarningVariable warningBuffer `
-OutVariable outBuffer `
-InformationVariable infoBuffer `
# 1> $null #2> $null # Only the success and error streams can be redirected here, other
# streams are not available.
# Restore the console
#[Console]::SetOut($oldOut)
# Get all catched output
#$outputOfAllWriteFunctions = $stringWriter.ToString()
}
catch
{
Write-Host "EXCEPTION: $_" -ForegroundColor Red
}
finally
{
Write-Host "error: $errorBuffer"
Write-Host "warning: $warningBuffer"
Write-Host "out: $outBuffer"
Write-Host "info: $infoBuffer"
Write-Host "result: $result"
#Write-Host "`noutputOfAllWriteFunctions:`n";Write-Host "$outputOfAllWriteFunctions" -ForegroundColor Cyan
Remove-Job -Job $job
}
My final conclusions:
Because the code block of Start-Job runs in its own process, it can't write to the scripts process console directly.
The code block is wrapped by a capture mechanism, which captures all 6 PS streams in buffers.
A call of Receive-Job uses inter process communication to get all those streams.
Receive-Job passes through stream 1 and 2 and makes them to its own output and therefore avaiable for redirection.
Receive-Job uses Write-Error to write stream 2 to the console, and therfore Receive-Job will raise an exception for parameter -ErrorAction Stop.
Then Write-Error uses Console.Out.WriteLine() to write to the console in red.
Then Receive-Job checks for variable storing and stores stream 1 (success), 2 (error), 3 (warning) and 6 (info).
Finally Receive-Job uses Console.Out.WriteLine() to write stream 1, 3, 4, 5 and 6 with different ForegroundColors to the console.
That's why you can capture ALL those 6 stream outputs with Console.SetOut(), even the error stream output, for which I had expected Console.SetError() would be needed.
But there is an issue in those conclusions:
The output of Write-Host is written to the console by default and its output is added to the information variable.
So Write-Host maybe just write into stream 6.
But the output of Write-Information is not visible on the console by default, but is also added to the information variable.
So Write-Information can't just share the same IPC pipe with Write-Host.
And Write-Warning can write to the console and the variable independently, so only one stream/pipe couldn't be used here, too.
Have a look at my diagram for that issue.
Receive-Job output transport diagram:
You can verify the diagram by redirecting stream 1-6 in the code block and stream 1 or 2 in the script.
|<-------- code block process -------->|<-- IPC -->|<-------------------- script process ------------------->|
Method Preference Stream Stream/Variable Console output
Write-Out * --> 1 --> PIPE 1 --> 1 --> Console.Out.Write(gray)
PIPE 1 --> Out Variable
Write-Error * --> 2 --> PIPE 2 --> 2 --> Console.Out.Write(red)
PIPE 2 --> Error Variable
Write-Warning Continue ----??????---> PIPE 3 --> Warning Variable
Write-Warning Continue --> 3 --> PIPE 4 --> Console.Out.Write(yellow)
Write-Verbose Continue --> 4 --> PIPE 4 --> Console.Out.Write(yellow)
Write-Debug Continue --> 5 --> PIPE 4 --> Console.Out.Write(yellow)
Write-Information Continue --> 6 --> PIPE 6 --> Console.Out.Write(gray)
Write-Information * ----??????---> PIPE 5 --> Information Variable
Write-Host * ----??????---> PIPE 5 --> Information Variable
Write-Host * --> 6 --> PIPE 6 --> Console.Out.Write(gray)
IPC : Inter Process Communication
* : always, independent from Preference or has no own Preference
There is no redirection you can add after Write-Information or Write-Warning to prevent storing in their variables.
If you'd redirect 3 and 6 after the methods, then it would only affect the console output, not the variable storing.
Only when $InformationPreference (not default) or $WarningPreference (default) are set to Continue, they write into stream 6 or 3, whose are always written in gray or yellow color to the console of the script process.
And only Write-Warning needs preference Continue to store in its variable, Write-Informations always writes to its variable.
Question:
How can 'Write-Warning' and 'Write-Information' pass their output to their assigned variables in the script process ?
(They can't use stream 7,8,9, since they don't exists in windows.)
Best practice:
After the call of Job-Start you should Start-Sleep 1-3 seconds to give the code block time to start or fail.
Then use Receive-Job the first time to get the current progress, start debug info, warning or errors.
You should not use Wait-Job, but use your own loop to check for the job's running state and check a timeout by yourself.
In that own wait loop, you call Receive-Job every X seconds to get progress, debug and error information from the code block process.
When the job's state is finished or failed, you call Receive-Job a last time to get the remaining data of all the buffers.
To redirect/capture stream 1 (success) and 2 (error) you can use normal redirection in the Receive-Job line or storing to the variables.
To capture stream 3 (warning) and 6 (info & Write-Host) you have to use the variable storing.
You can't redirect or capture stream 4 (verbose) or 5 (debug) directly, but you could redirect (4>&1 or 5>&1) those streams in the code block to stream 1 (success) to add them to the output variable.
To supress console output of Write-Output or Write-Error, you can just redirect stream 1 or 2 in the Receive-Job line.
You don't have to supress console output of Write-Information, Write-Verbose or Write-Debug, since they don't write to console with their default preferences.
If you want to capture the output of Write-Information in the assigned variable without console output, you have to redirect stream 6: Write-Information <message> 6>$null.
To supress console output of Write-Warning or Write-Host, you have to redirect stream 3 or 6 in their call lines: Write-Warning <message> 3>$null and Write-Host <message> 6>$null.
Be aware:
If you redirect stream success (1) or error (2) in the code block, they will not be tranfered to the script process, not written to the console and not be stored in the output or error variable.

Answer 1

You are a bit hard to follow with your terminology use but I will do my best with my limited experience.
The output of Write-Host is written to the console by default and its output is added to the information variable.
So Write-Host maybe just write into stream 6.
But the output of Write-Information is not visible on the console by default, but is also added to the information variable.
So Write-Information can't just share the same IPC pipe with Write-Host.
First of all, I read somewhere (do not remember so cannot link, sorry) and confirmed for myself that Write-Host and Write-Information do, in fact, use the same stream. However, Write-Host is, essentially, a special case of Write-Information whereby it ignores the preference variable and always writes. So I would expect Write-Information to show up in its respective variable when the respective preference variable is set properly.
And Write-Warning can write to the console and the variable independently, so only one stream/pipe couldn't be used here, too.
This observation is likely a design choice. (I am guessing here) I expect it works similar to the Tee-Object cmdlet so it can, indeed, write to the console and variable despite only being one stream.
$result = 'some string' | Tee-Object -Variable var
Write-Host $result
Write-Host $var
# same string in both variables