Capture a literal string on stderr in powershell - powershell

My application should write it's errors as literal JSON objects on stderr. This is proving difficult with PowerShell (5, 6 or 7) since PowerShell seems to want to prevent you from writing to stderr and, if you do succeed, it changes what you write.
In all examples we are running the following from within a powershell/pwsh console:
./test.ps1 2> out.json
test.ps1
Write-Error '{"code": "foo"}'
out.json
[91mWrite-Error: [91m{"code": "foo"}[0m
PowerShell is changing my stderr output. Bad PowerShell.
test.ps1
$Host.UI.WriteErrorLine('{"code": "foo"}')
out.json
PowerShell not writing to stderr (or >2 is not capturing it). Bad PowerShell.
test.ps1
[Console]::Error.WriteLine('{"code": "foo"}')
out.json
PowerShell not writing to stderr (or >2 is not capturing it). Bad PowerShell.
Update
I now understand that PowerShell does not have a stderr but rather numbered streams of which 2 corresponds to Write-Error and [Console]::Error.WriteLine() output and is sent to stderr of the pwsh/powershell.exe process IFF that output is redirected.
In short, stderr only exists outside of powershell and you can only access it via redirection:
pwsh ./test.ps1 2> out.json
Inside of powershell you can only redirect 2> the output from Write-Error. [Console]::Error.WriteLine() is not captured internally but sent to the console.

Problem
When you write to the error stream, Powershell creates an ErrorRecord object for each message. When you redirect the error stream and output it, Powershell formats it like an error message by default. The sub strings like [91m are ANSI escape sequences that colorize the message when written to the console.
Solution
To output plain text messages, convert the error records to strings before redirecting them to the file:
./test.ps1 2>&1 | ForEach-Object {
if( $_ -is [System.Management.Automation.ErrorRecord] ) {
# Message from the error stream
# -> convert error message to plain text and redirect (append) to file
"$_" >> out.json
}
else {
# Message from the success stream
# -> already a String, so output it directly
$_ # Shortcut for Write-Output $_
}
}
Remarks:
2>&1 merges the error stream with the success stream, so we can process both by the pipeline.
$_ is the current object processed by ForEach-Object. It is of type ErrorRecord, when the message is from the error stream of "test.ps1". It is of type String, when the message is from the success stream of "test.ps1".
Using the -is operator we check the type of the message to handle messages originating from the error stream differently than those from the success stream.
"$_" uses string interpolation to convert the ErrorRecord to the plain text message.
The >> operator redirects to the given file, but appends instead of overwriting.
Bonus code - a reusable cmdlet
If we regularly need to redirect error streams as plain text to a file, it makes sense to wrap the whole thing in a reusable cmdlet:
Function Out-ErrorMessageToFile {
[CmdletBinding()]
param (
[Parameter( Mandatory )] [String] $FilePath,
[Parameter( Mandatory, ValueFromPipeline )] [PSObject] $InputObject,
[Parameter( )] [Switch] $Append
)
begin {
if( ! $Append ) {
$null > $FilePath # Create / clear the file
}
}
process {
if( $InputObject -is [System.Management.Automation.ErrorRecord] ) {
# Message from the error stream
# -> convert error message to plain text and redirect (append) to file
"$InputObject" >> $FilePath
}
else {
# Message from the success stream
# -> already a String, so output it directly
$InputObject # Shortcut for Write-Output $InputObject
}
}
}
Usage examples:
# Overwrite "out.json"
./test.ps1 2>&1 | Out-ErrorMessageToFile out.json
# Append to "out.json"
./test.ps1 2>&1 | Out-ErrorMessageToFile out.json -Append

Related

Suppress NativeCommandError output when redirecting stdout and stderr to separate files

I have the following files:
test.ps1
& e:\test.bat > stdout.txt 2> stderr.txt
test.bat
#echo off
echo write to stdout
echo write to stderr >&2
When I call test.ps1 like this:
powershell -ExecutionPolicy bypass e:\test.ps1
The output files look like this:
stdout.txt
write argument to stdout
stderr.txt
test.bat : write to stderr
At E:\test.ps1:5 char:1
+ & "$application" "$argument" > $stdout 2> $stderr
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (write to stderr :String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
There is an answer how to prevent the NativeCommandError output from being written to file when redirecting both stdout and stderr to the same file, but how can I achieve that when writing to different files?
To build on the helpful comments:
In PowerShell (Core) 7+, your command would work as expected.
In Windows PowerShell, unfortunately, stderr lines are implicitly formatted as if they were PowerShell errors when a 2> redirection is involved, resulting in the noisy output you saw.
The solution is to merge stderr into stdout with 2>&1, and separate the collected lines into stdout and stderr by their type, allowing you to stringify the stderr lines, which PowerShell wraps in [System.Management.Automation.ErrorRecord] instances, via their .ToString() method, which results in their text content only.
Apart from being cumbersome, this approach requires collecting all output in memory first.
$stdout, $stderr = (& e:\test.bat 2>&1).Where({ $_ -is [string] }, 'Split')
$stdout > stdout.txt
$stderr.ForEach('ToString') > stderr.txt
Complementing mklement0's helpful answer, here is a function Tee-StdErr for PowerShell 5 and older versions that you can chain in-between to redirect stderr to a file without NativeCommandError, while forwarding regular output to the next pipeline command (or redirect it to a file if > is used):
Function Tee-StdErr {
[CmdletBinding()]
param (
[Parameter(Mandatory)] [string] $Path,
[Parameter(Mandatory, ValueFromPipeline)] $InputObject
)
begin {
$file = if( $item = New-Item -Path $Path -Force ) {
[IO.StreamWriter]::new( $item.FullName )
}
}
process {
if( $InputObject -is [Management.Automation.ErrorRecord] ) {
# stringify error message and write to file
if( $file ) { $file.WriteLine( "$InputObject" ) }
}
else {
# pass stdout through
$InputObject
}
}
end {
$file | ForEach-Object Dispose
}
}
Usage:
& .\test.bat 2>&1 | Tee-StdErr stderr.txt > stdout.txt
# Alternatively pipe stdout to other commands:
& .\test.bat 2>&1 | Tee-StdErr stderr.txt | Set-Content stdout.txt
2>&1 merges the error (#2) stream with the success (#1) stream, so both can be processed by the next pipeline command
Tee-StdErr tests whether the current pipeline object ($InputObject) is an ErrorRecord and if so, it stringifies ("$_") it and writes it to the error file. Otherwise the current object is a string from the success stream which is passed through by using PowerShell's implicit output feature (just naming the variable outputs it).

When running a command in powershell how can I prepend a date/time for all output on stdout/stderr?

Is it possible in powershell when running a script to add a date prefix to all log output?
I know that it would be possible to do something like:
Write-Host "$(Get-Date -format 'u') my log output"
But I dont want to have to call some function for each time we output a line. Instead I want to modify all output when running any script or command and have the time prefix for every line.
To insert a date in front of all output, that is stdout, stderr and the PowerShell-specific streams, you can use the redirection operator *>&1 to redirect (merge) all streams of a command or scriptblock, pipe to Out-String -Stream to format the stream objects into lines of text and then use ForEach-Object to process each line and prepend the date.
Let me start with a simple example, a more complete solution can be found below.
# Run a scriptblock
&{
# Test output to all possible streams, using various formatting methods.
# Added a few delays to test if the final output is still streaming.
"Write $($PSStyle.Foreground.BrightGreen)colored`ntext$($PSStyle.Reset) to stdout"
Start-Sleep -Millis 250
[PSCustomObject]#{ Answer = 42; Question = 'What?' } | Format-Table
Start-Sleep -Millis 250
Get-Content -Path not-exists -EA Continue # produce a non-terminating error
Start-Sleep -Millis 250
Write-Host 'Write to information stream'
Start-Sleep -Millis 250
Write-Warning 'Write to warning stream'
Start-Sleep -Millis 250
Write-Verbose 'Write to verbose stream' -Verbose
Start-Sleep -Millis 250
$DebugPreference = 'Continue' # To avoid prompt, needed for Windows Powershell
Write-Debug 'Write to debug stream'
} *>&1 | Out-String -Stream | ForEach-Object {
# Add date in front of each output line
$date = Get-Date -Format "yy\/MM\/dd H:mm:ss"
foreach( $line in $_ -split '\r?\n' ) {
"$($PSStyle.Reset)[$date] $line"
}
}
Output in PS 7.2 console:
Using Out-String we use the standard PowerShell formatting system to have the output look normally, as it would appear without redirection (e. g. things like tables stay intact). The -Stream parameter is crucial to keep the streaming output behaviour of PowerShell. Without this parameter, output would only be received once the whole scriptblock has completed.
While the output already looks quite nice, there are some minor issues:
The verbose, warning and debug messages are not colored as usual.
The word "text" in the 2nd line should be colored in green. This isn't working due to the use of $PSStyle.Reset. When removed, the colors of the error message leak into the date column, which looks far worse. It can be fixed, but it is not trivial.
The line wrapping isn't right (it wraps into the date column in the middle of the output).
As a more general, reusable solution I've created a function Invoke-WithDateLog that runs a scriptblock, captures all of its output, inserts a date in front of each line and outputs it again:
Function Invoke-WithDateLog {
[CmdletBinding()]
param(
[Parameter(Mandatory)]
[scriptblock] $ScriptBlock,
[Parameter()]
[string] $DateFormat = '[yy\/MM\/dd H:mm:ss] ',
[Parameter()]
[string] $DateStyle = $PSStyle.Foreground.BrightBlack,
[Parameter()]
[switch] $CatchExceptions,
[Parameter()]
[switch] $ExceptionStackTrace,
[Parameter()]
[Collections.ICollection] $ErrorCollection
)
# Variables are private so they are not visible from within the ScriptBlock.
$private:ansiEscapePattern = "`e\[[0-9;]*m"
$private:lastFmt = ''
& {
if( $CatchExceptions ) {
try { & $scriptBlock }
catch {
# The common parameter -ErrorVariable doesn't work in scripted cmdlets, so use our own error variable parameter.
if( $null -ne $ErrorCollection ) {
$null = $ErrorCollection.Add( $_ )
}
# Write as regular output, colored like an error message.
"`n" + $PSStyle.Formatting.Error + "EXCEPTION ($($_.Exception.GetType().FullName)):`n $_" + $PSStyle.Reset
# Optionally write stacktrace. Using the -replace operator we indent each line.
Write-Debug ($_.ScriptStackTrace -replace '^|\r?\n', "`n ") -Debug:$ExceptionStackTrace
}
}
else {
& $scriptBlock
}
} *>&1 | ForEach-Object -PipelineVariable record {
# Here the $_ variable is either:
# - a string in case of simple output
# - an instance of one of the System.Management.Automation.*Record classes (output of Write-Error, Write-Debug, ...)
# - an instance of one of the Microsoft.PowerShell.Commands.Internal.Format.* classes (output of a Format-* cmdlet)
if( $_ -is [System.Management.Automation.ErrorRecord] ) {
# The common parameter -ErrorVariable doesn't work in scripted cmdlets, so use our own error variable parameter.
if( $null -ne $ErrorCollection ) {
$null = $ErrorCollection.Add( $_ )
}
}
$_ # Forward current record
} | Out-String -Stream | ForEach-Object {
# Here the $_ variable is always a (possibly multiline) string of formatted output.
# Out-String doesn't add any ANSI escape codes to colorize Verbose, Warning and Debug messages,
# so we have to do it by ourselfs.
$overrideFmt = switch( $record ) {
{ $_ -is [System.Management.Automation.VerboseRecord] } { $PSStyle.Formatting.Verbose; break }
{ $_ -is [System.Management.Automation.WarningRecord] } { $PSStyle.Formatting.Warning; break }
{ $_ -is [System.Management.Automation.DebugRecord] } { $PSStyle.Formatting.Debug; break }
}
# Prefix for each line. It resets the ANSI escape formatting before the date.
$prefix = $DateStyle + (Get-Date -Format $DateFormat) + $PSStyle.Reset
foreach( $line in $_ -split '\r?\n' ) {
# Produce the final, formatted output.
$prefix + ($overrideFmt ?? $lastFmt) + $line + ($overrideFmt ? $PSStyle.Reset : '')
# Remember last ANSI escape sequence (if any) of current line, for cases where formatting spans multiple lines.
$lastFmt = [regex]::Match( $line, $ansiEscapePattern, 'RightToLeft' ).Value
}
}
}
Usage example:
# To differentiate debug and verbose output from warnings
$PSStyle.Formatting.Debug = $PSStyle.Foreground.Yellow
$PSStyle.Formatting.Verbose = $PSStyle.Foreground.BrightCyan
Invoke-WithDateLog -CatchExceptions -ExceptionStackTrace {
"Write $($PSStyle.Foreground.Green)colored`ntext$($PSStyle.Reset) to stdout"
[PSCustomObject]#{ Answer = 42; Question = 'What?' } | Format-Table
Get-Content -Path not-exists -EA Continue # produce a non-terminating error
Write-Host 'Write to information stream'
Write-Warning 'Write to warning stream'
Write-Verbose 'Write to verbose stream' -Verbose
Write-Debug 'Write to debug stream' -Debug
throw 'Critical error'
}
Output in PS 7.2 console:
Notes:
The code requires PowerShell 7+.
The date formatting can be changed through parameters -DateFormat (see formatting specifiers) and -DateStyle (ANSI escape sequence for coloring).
Script-terminating errors such as created by throwing an exception or using Write-Error -EA Stop, are not logged by default. Instead they bubble up from the scriptblock as usual. You can pass parameter -CatchExceptions to catch exceptions and log them like regular non-terminating errors. Pass -ExceptionStackTrace to also log the script stacktrace, which is very useful for debugging.
Scripted cmdlets such as this one don't set the automatic variable $? and also don't add errors to the automatic $Error variable when an error is written via Write-Error. Neither the common parameter -ErrorVariable works. To still be able to collect error information I've added parameter -ErrorCollection which can be used like this:
$scriptErrors = [Collections.ArrayList]::new()
Invoke-WithDateLog -CatchExceptions -ExceptionStackTrace -ErrorCollection $scriptErrors {
Write-Error 'Write to stderr' -EA Continue
throw 'Critical error'
}
if( $scriptErrors ) {
# Outputs "Number of errors: 2"
"`nNumber of errors: $($scriptErrors.Count)"
}
The objects generated by Write-Host already come with a timestamp, you can use Update-TypeData to override the .ToString() Method from the InformationRecord Class and then redirect the output from the Information Stream to the Success Stream.
Update-TypeData -TypeName System.Management.Automation.InformationRecord -Value {
return $this.TimeGenerated.ToString('u') + $this.MessageData.Message.PadLeft(10)
} -MemberType ScriptMethod -MemberName ToString -Force
'Hello', 'World', 123 | Write-Host 6>&1

Is it possible to customize the verbose message prefix in Powershell?

The current verbose message prefix is simply VERBOSE:
I would like to modify it to VERBOSE[N]:, where N is the current thread Id.
Is it possible?
This behavior (or rather the format string) is hard-coded into the default PowerShell host and there are no hooks to override it. You'd have to implement your own host or modify the calling code to use a proper logging framework, neither of which are particularly simple.
If you at least control the outermost invocation, you have the option to redirect the verbose stream output, and we can use this in combination with a cmdlet to "sort of" customize things:
function Verbosi-Tee {
[CmdletBinding()]
Param (
[Parameter(ValueFromPipeline = $true)]
$o
)
Process {
if ($o -is [System.Management.Automation.VerboseRecord]) {
Write-Verbose "[$([System.Threading.Thread]::CurrentThread.ManagedThreadId)] $($o.Message)"
} else {
$o
}
}
}
Sample use:
$VerbosePreference = "continue"
$x = (&{
Write-Verbose "This is verbose."
Write-Output "This is just regular output."
} >4&1 | Verbosi-Tee) # redirect, then pipe
"We captured the output in `$x: $x"
Output (on my system):
VERBOSE: [16] This is verbose.
We captured the output in $x: This is just regular output.
The name of the cmdlet is a lie because this doesn't in fact implement a full tee, but a good pun is its own reward.

How to pass large input to powershell

Assuming powershell has a limit of N characters in its command, how can I pass more than N chars to the powershell cmdlet? Based on https://support.microsoft.com/en-in/kb/830473 link, it seems that the character limit is 8191 but it says that for cmd.exe, not sure what is the size limit for powershell. So if I have an input of size more than >8k, can I redirect the input to the powershell to circumvent this problem (solution based on what is mentioned in the referenced document).
Eg:
powershell console $> echo “a very long string” // the whole command including the echo and the very long string totalling less than 8192 chars on the powershell console. When I execute this I get the whole string as the output on the console
powershell console $> echo “a very long string // Try to add characters to the very long string, powershell doesn’t allow me to add more chars to the very long string if the total goes above 8192 since I guess I have reached the limit on the number of characters I can enter.
What I want:
powershell console $> echo // Place my input (which is more than 8192 chars) in a file and provide that as an input to echo and echo should display the complete string on the console thereby circumventing the limitation of the number of chars in a command.
The command echo I have used is only for representation purpose and I want to use a custom cmdlet instead of that so please consider this a valid scenario.
Edit 2:
psm1 file:
Function DoSomething {
[CmdletBinding()]
Param(
[Parameter(
Mandatory = $False)
]
[string]$v1,
[Parameter(
Mandatory = $False)
]
[string]$v2)
Begin {}
Process {
Write-Output "hello $v1 | $v2"
}
}
Text File Content say content.txt(short for representation purpose but assume this can be more than 8k characters):
-v1 "t1" -v2 "qwe"
Now when I do
powershell Console$> DoSomething (Get-Content content.txt)
the output that I get is
hello -v1 "t1" -v2 "qwe" |
I expect the output to be
hello -v1 "t1" | -v2 "qwe"
so that the execution of the cmdlet can happen without any issues. I tried this with the example of more than 8k characters in the text file and it is able to print the output, it is just that the parameters aren't getting separated. The command to provide the input to the cmdlet doesn't have to be Get-Content, it can be anything as long as it works.
You misunderstand how parameters in PowerShell functions work. The output of Get-Content is an array of strings (one string for each line in the file), but the entire array is passed to the first parameter. Also, a string isn't magically split so that the substrings can go to several parameters. How should PowerShell know which way to split the string?
A better way to deal with such input data is to have your function accept input from the pipeline:
Function DoSomething {
[CmdletBinding()]
Param(
[Parameter(
Mandatory=$false,
ValueFromPipeline=$true,
ValueFromPipelineByPropertyName=$true
)]
[string]$v1,
[Parameter(
Mandatory=$false,
ValueFromPipeline=$true,
ValueFromPipelineByPropertyName=$true
)]
[string]$v2
)
Process {
Write-Output "hello $v1 | $v2"
}
}
and define the data as a CSV (column names matching the parameter names):
v1,v2
"-v1 ""t1""","-v2 ""qwe"""
so that you can pipe the data into the function:
Import-Csv content.csv | DoSomething
With the function built like this you could also define the data as a hashtable and splat it:
$data = #{
'v1' = '-v1 "t1"'
'v2' = '-v2 ""qwe"'
}
DoSomething #data
For more information about function parameters see about_Parameters and about_Functions_Advanced_Parameters.
Else you can pass a path file where content of this file is used in your script ps. No limit then...

Appropriate logging in Powershell

If I have a powershell script say called caller.ps1 which looks like this
.\Lib\library.ps1
$output = SomeLibraryFunction
where library.ps1 looks like the following
function SomeLibraryFunction()
{
Write-Output "Some meaningful function logging text"
Write-Output "return value"
}
What I'd like to achieve is a way in which the library function can return it's value but also add some logging messages that allow the caller to process those internal messages as they see fit. The best I can think of is to write both to the pipeline and then the caller will have an array with the actual return value plus the internal messages which may be of use to a logger the calling script has.
Am I going about this problem the right way? Is there a better way to achieve this?
It's usually not a good idea to mix logging messages with actual output. Consumers of your function then have to do a lot of filtering to find the object they really want.
Your function should write logging messages to the verbose output stream. If the caller wants to see those messages, it can by specifying the -Verbose switch to your function.
function SomeLibraryFunction()
{
[CmdletBinding()]
param(
)
Write-Verbose "Some meaningful function logging text"
Write-Output "return value"
}
In PowerShell 3+, consumers can redirect your verbose messages to the output stream or a file:
# Show verbose logging messages on the console
SomeLibraryFunction -Verbose
# Send verbose logging messages to a file
SomeLibraryFunction -Verbose 4> verbose.txt
# redirect verbose message to the output stream
SomeLibraryFunction -Verbose 4>&1
Other options include:
Writing to a well-known file
Writing to the Event Log
Use Start-Transcript to create a log of the session.
Something like Tee-Object might be helpful to you
function SomeLibraryFunction()
{
$filepath = "C:\temp\log.txt"
write-output "Some meaningful function logging text" | Tee-Object -FilePath $filepath -Append
write-output "return value" | Tee-Object -FilePath $filepath -Append
}
For more information on Tee-Object look here
You could use an If statement based on a variable like $logging=$true but i could see that getting messy.
Another Approach
If you are looking for more of an optional solution then maybe you could use something like this Start-Transcript and Stop-Transcript which creates a record of all or part of a Windows PowerShell session in a text file.
function SomeLibraryFunction()
{
write-output "Some meaningful function logging text"
write-output "return value"
}
$logging = $True
If ($logging){
Start-Transcript -Path C:\temp\log.txt -Append
}
SomeLibraryFunction
If ($logging){
Stop-Transcript
}
This would just show that you could toggle the Start and Stop. You could even set the switch with a paramater passed to the script if you chose.
NOTE The output might be more that you are looking for but at least give it a try. Also, this will not work in the Powershell ISE as you will get an error Start-Transcript : This host does not support transcription.
Another way to do this would be to return a compound object that includes the results and the log information. This is then easy to pick apart.
function MyFunc
{
# initialize
$result = $null
$log = #()
# write to the log
$log += "This will be written to the log"
$log += "So will this"
# record the result
$result = $true
# return result and log combined into object
return New-Object -TypeName PSObject -Property #{ result = $result; log = $log -join "`r`n" }
}
# Call the function and get the results
$MyFuncResult = MyFunc
# Display the returned result value
Write-Host ( "MyFunc Result = {0}" -f $MyFuncResult.Result )
# Display the log
write-host ( "MyFunc Log = {0}" -f $MyFuncResult.Log )
Alternatively, if you want to avoid the object, pass in a log variable by reference. The function can write to the log variable and the changes will be visible in the calling scope. To do this, you need to add the [ref] prefix to the function definition AND the function call. When you write to the variable in the function you need to refer to the .value property.
function MyFunc2 ([ref]$log)
{
# initialize
$result = $null
# write to the log
$log.value += "`r`nThis will be written to the log"
$log.value += "`r`nSo will this"
# record the result
$result = $true
# return result and log combined into object
return $result
}
# Call the function and get the results
$log = "before MyFunc2"
$MyFuncResult = MyFunc2([ref]$log)
$log += "`nafter MyFunc2"
# Display the returned result value
write-host ( "MyFunc2 result = {0}" -f $MyFuncResult )
# Display the log
write-host ( "MyFunc2 Log = {0}" -f $Log )