Trying to use PowerShell to capture the running status of the "Nessus Essentials" software product. Simply trying to capture product status: running, not running, or other. Getting the below error each time. I've tried changing -like to -match and changing string [warn] [scanner] Not linked to a manager to various other shorter versions, with wildcards and without, to no avail. I still get several lines of an ugly error message when all I want is one line with the string Not linked to a manager returned to console with nothing beneath that.
Pertinent snippet working incorrectly:
} elseif(($agentStatus.stdOut -like "[warn] [scanner] Not linked to a manager")) {
Throw "Not linked to a manager"
The Error:
The Code:
Function Start-ProcessGetStreams {
[CmdLetBinding()]
Param(
[System.IO.FileInfo]$FilePath,
[string[]]$ArgumentList
)
$pInfo = New-Object System.Diagnostics.ProcessStartInfo
$pInfo.FileName = $FilePath
$pInfo.Arguments = $ArgumentList
$pInfo.RedirectStandardError = $true
$pInfo.RedirectStandardOutput = $true
$pinfo.UseShellExecute = $false
$pInfo.CreateNoWindow = $true
$pInfo.WindowStyle = [System.Diagnostics.ProcessWindowStyle]::Hidden
$proc = New-Object System.Diagnostics.Process
$proc.StartInfo = $pInfo
Write-Verbose "Starting $FilePath"
$proc.Start() | Out-Null
Write-Verbose "Waiting for $($FilePath.BaseName) to complete"
$proc.WaitForExit()
$stdOut = $proc.StandardOutput.ReadToEnd()
$stdErr = $proc.StandardError.ReadToEnd()
$exitCode = $proc.ExitCode
Write-Verbose "Standard Output: $stdOut"
Write-Verbose "Standard Error: $stdErr"
Write-Verbose "Exit Code: $exitCode"
[PSCustomObject]#{
"StdOut" = $stdOut
"Stderr" = $stdErr
"ExitCode" = $exitCode
}
}
Function Get-NessusStatsFromStdOut {
Param(
[string]$stdOut
)
$stats = New-Object System.Collections.Hashtable
$StdOut -split "`r`n" | % {
if($_ -like "*:*") {
$result = $_ -split ":"
$stats.add(($result[0].Trim() -replace "[^A-Za-z0-9]","_").ToLower(),$result[1].Trim())
}
}
Return $stats
}
Function Get-DateFromEpochSeconds {
Param(
[int]$seconds
)
$utcTime = (Get-Date 01.01.1970)+([System.TimeSpan]::fromseconds($seconds))
Return Get-Date $utcTime.ToLocalTime() -Format "yyyy-MM-dd HH:mm:ss"
}
Try {
$nessusExe = Join-Path $env:ProgramFiles -ChildPath "Tenable\Nessus\nessuscli.exe" -ErrorAction Stop
} Catch {
Throw "Cannot find NessusCli.exe"
}
Write-Host "Getting Agent Status..."
$agentStatus = Start-ProcessGetStreams -FilePath $nessusExe -ArgumentList "managed status"
If($agentStatus.stdOut -eq "" -and $agentStatus.StdErr -eq "") {
Throw "No Data Returned from NessusCli"
} elseif($agentStatus.StdOut -eq "" -and $agentStatus.StdErr -ne "") {
Throw "StdErr: $($agentStatus.StdErr)"
} elseif(($agentStatus.stdOut -like "[warn] [scanner] Not linked to a manager")) {
Throw "Not linked to a manager"
} elseif(-not($agentStatus.stdOut -like "*Running: *")) {
Throw "StdOut: $($agentStatus.StdOut)"
} else {
$stats = Get-NessusStatsFromStdOut -stdOut $agentStatus.StdOut
If($stats.last_connection_attempt -as [int]) { $stats.last_connection_attempt = Get-DateFromEpochSeconds $stats.last_connection_attempt }
If($stats.last_connect -as [int]) { $stats.last_connect = Get-DateFromEpochSeconds $stats.last_connect }
If($stats.last_scanned -as [int]) { $stats.last_connect = Get-DateFromEpochSeconds $stats.last_scanned }
}
$stats | Out-Host
Note: Code above is courtesy of here, I've only made a change to the path of Nessus, and I am adding the attempt to capture that it's not connected to a manager.
Modify your code so that it separates standard output from error and so that it handles each line separately.
The following is how to capture standard output (excluding else statements and error handling) of a program (according to your $Proc variable)
if ($proc.Start())
{
while (!$proc.StandardOutput.EndOfStream)
{
$StreamLine = $proc.StandardOutput.ReadLine()
if (![string]::IsNullOrEmpty($StreamLine))
{
# TODO: Duplicate code in this scope as needed or rewrite to use multiline regex
$WantedLine = [regex]::Match($StreamLine, "(?<wanted>.*Not linked to a manager.*)")
$Capture = $WantedLine.Groups["wanted"]
if ($Capture.Success)
{
Write-Output $Capture.Value
}
}
}
}
After that deal with error output separately:
$StandardError = $Proc.StandardError.ReadToEnd()
if (![string]::IsNullOrEmpty($StandardError))
{
# Or use Write-Error
Write-Output $StandardError
}
using below snippet :
#loop until we have found the newly added domain
$Stop=''
while (-not $Stop)
{
$stop = Get-AzureADDomain -Name $direct_routing_domain -ErrorAction Ignore | Out-Null
if (!$stop)
{
Write-Host "Domain is not provisioned yet - retrying in 60 seconds..." -ForegroundColor Yellow
Start-Sleep -Seconds 60
}
}
Write-Host "Domain is provisioned " -ForegroundColor Green
The output gives an error on stdout:
Get-AzureADDomain : Error occurred while executing GetDomain
Code: Request_ResourceNotFound
...
So here comes the question: how can I surpress this output ? I believe that | out-null should be enough. Or is this a bug in AzureAd 2.0.0 module ?
Out-Null does not ignore errors. Out-Null redirects stdout (known as stream #1) to null. You can try doing a try catch (catch provides easy exceptions handling)
try {}
catch {}
doc. https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_try_catch_finally?view=powershell-5.1
or redirect an error stream to void
Write-Error "ASD" 2>$null
doc. https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_redirection?view=powershell-5.1
You can use this:
$stop = $true
try {Get-AzureADDomain -Name $direct_routing_domain}
catch {
$stop = $false
}
OR
$stop = $true
$domain = Get-AzureADDomain | Where-Object {$_.Name -eq $direct_routing_domain}
if ($null -eq $domain) {
$stop = $false
}
You can use a try{}..catch{} on this, but remember to add -ErrorAction Stop (not 'Ignore') to the cmdlet to ensure also non-terminating exceptions are caught in the catch block:
# loop until we have found the newly added domain
$isProvisioned = $false
while (-not $isProvisioned) {
try {
$isProvisioned = [bool](Get-AzureADDomain -Name $direct_routing_domain -ErrorAction Stop)
}
catch { $isProvisioned = $false }
if (!$isProvisioned) {
Write-Host "Domain is not provisioned yet - retrying in 60 seconds..." -ForegroundColor Yellow
Start-Sleep -Seconds 60
}
}
Write-Host "Domain is provisioned " -ForegroundColor Green
I'm doing a BITS transfer of daily imagery from a web server and I keep getting random drops during the transfer.
As it's cycling through the downloads I get the occasional "The connection was closed prematurely" or "An error occurred in the secure channel support". There are about 180 images in each folder and this happens for maybe 5-10% of them. I need to retry the download for those that didn't complete.
My code to do so follows - my imperfect work-around is to run the loop twice but I'm hoping to find a better solution.
# Set the URL where the images are located
$url = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'
# Set the local path where the images will be stored
$path = 'C:\images\Wind_Waves\latest\'
# Create a list of all assets returned from $url
$site = Invoke-WebRequest -UseBasicParsing -Uri $url
# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links | Where-Object{ $_.tagName -eq 'A' -and $_.href.ToLower().EndsWith("jpg") }
# Create a list of all href items from the table & call it $images
$images = $table.href
# Enumerate all of the images - for troubleshooting purposes - can be removed
$images
# Check to make sure there are images available for download - arbitrarily picked more than 2 $images
if($images.count -gt 2){
# Delete all of the files in the "latest" folder
Remove-Item ($path + "*.*") -Force
# For loop to check to see if we already have the image and, if not, download it
ForEach ($image in $images)
{
if(![System.IO.File]::Exists($path + $image)){
Write-Output "Downloading: " $image
Start-BitsTransfer -Source ($url + $image) -Destination $path -TransferType Download -RetryInterval 60
Start-Sleep 2
}
}
Get-BitsTransfer | Where-Object {$_.JobState -eq "Transferred"} | Complete-BitsTransfer
} else {
Write-Output "No images to download"}
I don't see any error handling in your code to resume/retry/restart on fail.
Meaning why is there no try/catch in the loop or the Get?
If the Get is on per download job in the loop, why is it outside the loop?
Download is the default for TransferType, so no need to specify, it normally will generate an error if you do.
So, something like this. I did test this, but never got a fail. Yet, I have a very high-speed speed internet connection. If you are doing this inside an enterprise, edge devices (filters, proxies, could also be slowing things down, potentially forcing timeouts.)
$url = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'
$path = 'D:\Temp\images\Wind_Waves\latest'
$site = Invoke-WebRequest -UseBasicParsing -Uri $url
# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links |
Where-Object{
$_.tagName -eq 'A' -and
$_.href.ToLower().EndsWith('jpg')
}
<#
# Create a list of all href items from the table & call it $images
Enumerate all of the images - for troubleshooting purposes - can be removed
Assign and display using variable squeezing
#>
($images = $table.href)
<#
Check to make sure there are images available for download - arbitrarily
picked more than 2 $images
#>
if($images.count -gt 2)
{
Remove-Item ($path + '*.*') -Force
ForEach ($image in $images)
{
Try
{
Write-Verbose -Message "Downloading: $image" -Verbose
if(![System.IO.File]::Exists($path + $image))
{
$StartBitsTransferSplat = #{
Source = ($url + $image)
Destination = $path
RetryInterval = 60
}
Start-BitsTransfer #StartBitsTransferSplat -ErrorAction Stop
Start-Sleep 2
}
Get-BitsTransfer |
Where-Object {$PSItem.JobState -eq 'Transferred'} |
Complete-BitsTransfer
}
Catch
{
$PSItem.Exception.Message
Write-Warning -Message "Download of $image not complete or failed. Attempting a resume/retry" -Verbose
Get-BitsTransfer -Name $image | Resume-BitsTransfer
}
}
}
else
{
Write-Warning -Message 'No images to download'
$PSItem.Exception.Message
}
See the help files
Resume-BitsTransfer
Module: bitstransfer Resumes a BITS transfer job.
# Example 1: Resume all BITS transfer jobs owned by the current user
Get-BitsTransfer | Resume-BitsTransfer
# Example 2: Resume a new BITS transfer job that was initially suspended
$Bits = Start-BitsTransfer -DisplayName "MyJob" -Suspended
Add-BitsTransfer -BitsJob $Bits -ClientFileName C:\myFile -ServerFileName http://www.SomeSiteName.com/file1
Resume-BitsTransfer -BitsJob $Bits -Asynchronous
# Example 3: Resume the BITS transfer by the specified display name
Get-BitsTransfer -Name "TestJob01" | Resume-BitsTransfer
Here's a somewhat modified version of the above code. It appears the BITS transfer job object goes away when the error occurs, so there is no use in trying to find/resume that job. Instead, I wrapped the entire Try-Catch block in a while loop with an exit when the file is downloaded.
$url = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'
$path = 'D:\Temp\images\Wind_Waves\latest'
$site = Invoke-WebRequest -UseBasicParsing -Uri $url
$MaxRetries = 3 # Initialize the maximum number of retry attempts.
# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links |
Where-Object {
$_.tagName -eq 'A' -and
$_.href.ToLower().EndsWith('jpg')
}
<#
# Create a list of all href items from the table & call it $images
Enumerate all of the images - for troubleshooting purposes - can be removed
Assign and display using variable squeezing
#>
($images = $table.href)
<#
Check to make sure there are images available for download - arbitrarily
picked more than 2 $images
#>
if ($images.count -gt 2) {
Remove-Item ($path + '*.*') -Force
ForEach ($image in $images) {
# Due to occasional failures to transfer, wrap the BITS transfer in a while loop
# re-initialize the exit counter for each new image
$retryCount = 0
while ($retryCount -le $MaxRetries){
Try {
Write-Verbose -Message "Downloading: $image" -Verbose
if (![System.IO.File]::Exists($path + $image)) {
$StartBitsTransferSplat = #{
Source = ($url + $image)
Destination = $path
RetryInterval = 60
}
Start-BitsTransfer #StartBitsTransferSplat -ErrorAction Stop
Start-Sleep 2
}
# To get here, the transfer must have finished, so set the counter
# greater than the max value to exit the loop
$retryCount = $MaxRetries + 1
} # End Try block
Catch {
$PSItem.Exception.Message
$retryCount += 1
Write-Warning -Message "Download of $image not complete or failed. Attempting retry #: $retryCount" -Verbose
} # End Catch Block
} # End While loop for retries
} # End of loop over images
} # End of test for new images
else {
Write-Warning -Message 'No images to download'
$PSItem.Exception.Message
} # End of result for no new images
Here is a combination of the code that postanote provided and a Do-While loop to retry the download up to 5x if an error is thrown.
$url = 'https://www.nrlmry.navy.mil/archdat/global/stitched/MoS_2/navgem/wind_waves/latest/'
$path = 'D:\Temp\images\Wind_Waves\latest'
$site = Invoke-WebRequest -UseBasicParsing -Uri $url
# Create a table subset from the $site of all files returned with a .jpg extension
$table = $site.Links |
Where-Object{
$_.tagName -eq 'A' -and
$_.href.ToLower().EndsWith('jpg')
}
<#
# Create a list of all href items from the table & call it $images
Enumerate all of the images - for troubleshooting purposes - can be removed
Assign and display using variable squeezing
#>
($images = $table.href)
<# Check to make sure there are images available for download - arbitrarily
picked more than 2 $images #>
if($images.count -gt 2)
{
Remove-Item ($path + '*.*') -Force
ForEach ($image in $images)
{
# Create a Do-While loop to retry downloads up to 5 times if they fail
$Stoploop = $false
[int]$Retrycount = "0"
do{
Try
{
Write-Verbose -Message "Downloading: $image" -Verbose
if(![System.IO.File]::Exists($path + $image))
{
$StartBitsTransferSplat = #{
Source = ($url + $image)
Destination = $path
RetryInterval = 60
}
Start-BitsTransfer #StartBitsTransferSplat -ErrorAction Stop
Start-Sleep 10
$Stoploop = $true
}
Get-BitsTransfer |
Where-Object {$PSItem.JobState -eq 'Transferred'} |
Complete-BitsTransfer
}
Catch
{
if ($Retrycount -gt 5){
$PSItem.Exception.Message
Write-Warning -Message "Download of $image not complete or failed." -Verbose
$Stoploop = $true
}
else {
Write-Host "Could not download the image, retrying..."
Start-Sleep 10
$Retrycount = $Retrycount + 1
}
}
}
While ($Stoploop -eq $false)
}
}
else
{
Write-Warning -Message 'No images to download'
$PSItem.Exception.Message
}
I'm writing a script that will watch a directory for any new mp4 files, then convert the file using HandBrake's CLI tool. The logic that watches the directory for changes works by itself, but if I drop a large video into the "watched" directory the conversion fails since it kicks off as soon as it sees a new file, before the file has time to finish copying.
I'm using a do until loop to check if the file is locked/downloading and then continues once the file is unlocked/writable. The loop works as a stand-alone script, but when used inside the filesystem watcher the script it will halt without any errors on this line:
[System.IO.FileStream] $fs = $convertingFile.OpenWrite();
This occurs regardless if $ErrorActionPreference = "SilentlyContinue" is commented out. I've been unable to gather any log output to see why the script is halting using Start-Transcript or Out-File.
How should I best gather error info about why the script halts once it hits this line?
Bonus: Why might this script not provide error information?
$ErrorActionPreference = "SilentlyContinue"
function Start-FileSystemWatcher {
[CmdletBinding()]
param(
[Parameter()]
[string]$Path,
[Parameter()]
[ValidateSet('Changed','Created','Deleted','Renamed')]
[string[]]$EventName,
[Parameter()]
[string]$Filter,
[Parameter()]
[System.IO.NotifyFilters]$NotifyFilter,
[Parameter()]
[switch]$Recurse,
[Parameter()]
[scriptblock]$Action
)
#region Build FileSystemWatcher
$FileSystemWatcher = New-Object System.IO.FileSystemWatcher
if (-not $PSBoundParameters.ContainsKey('Path')) {
$Path = $PWD
}
$FileSystemWatcher.Path = $Path
if ($PSBoundParameters.ContainsKey('Filter')) {
$FileSystemWatcher.Filter = $Filter
}
if ($PSBoundParameters.ContainsKey('NotifyFilter')) {
$FileSystemWatcher.NotifyFilter = $NotifyFilter
}
if ($PSBoundParameters.ContainsKey('Recurse')) {
$FileSystemWatcher.IncludeSubdirectories = $True
}
if (-not $PSBoundParameters.ContainsKey('EventName')) {
$EventName = 'Changed','Created','Deleted','Renamed'
}
if (-not $PSBoundParameters.ContainsKey('Action')) {
$Action = {
switch ($Event.SourceEventArgs.ChangeType) {
'Renamed' {
$Object = "{0} was {1} to {2} at {3}" -f $Event.SourceArgs[-1].OldFullPath,
$Event.SourceEventArgs.ChangeType,
$Event.SourceArgs[-1].FullPath,
$Event.TimeGenerated
}
Default {
$Object = "{0} was {1} at {2}" -f $Event.SourceEventArgs.FullPath,
$Event.SourceEventArgs.ChangeType,
$Event.TimeGenerated
}
}
$WriteHostParams = #{
ForegroundColor = 'Green'
BackgroundColor = 'Black'
Object = $Object
}
Write-Host #WriteHostParams
}
}
$ObjectEventParams = #{
InputObject = $FileSystemWatcher
Action = $Action
}
foreach ($Item in $EventName) {
$ObjectEventParams.EventName = $Item
$ObjectEventParams.SourceIdentifier = "File.$($Item)"
Write-Verbose "Starting watcher for Event: $($Item)"
$Null = Register-ObjectEvent #ObjectEventParams
}
}
$FileSystemWatcherParams = #{
Path = 'X:\share\scripts\ps\converter\input'
Recurse = $True
NotifyFilter = 'FileName'
Verbose = $True
Action = {
$Item = Get-Item $Event.SourceEventArgs.FullPath
$WriteHostParams = #{
ForegroundColor = 'Green'
BackgroundColor = 'Black'
}
$inputFile = "${PWD}\input\$($Item.Name)".trim()
$outputFile = "${PWD}\output\$($Item.Name)".trim()
$logFile = "${PWD}\log\$($Item.Name).txt"
$testLogFile = "${PWD}\log\$($Item.Name)(t).txt"
function mp4-Func {
Start-Transcript -path $logFile
Write-Host "New mp4 file detected..."
$convertingFile = New-Object -TypeName System.IO.FileInfo -ArgumentList $inputFile
$locked = 1
do {
[System.IO.FileStream] $fs = $convertingFile.OpenWrite();
if (!$?) {
Write-Host "Can't convert yet, file appears to be loading..."
sleep 2
}
else {
$fs.Dispose()
$locked = 0
}
} until ($locked -eq 0)
Write-Host "File unlocked and ready for conversion."
HandBrake
Stop-Transcript
$WriteHostParams.Object = "Finished converting: $($Item.Name)"
}
function HandBrake {
.\HandBrakeCLI.exe --input "$inputFile" `
--output "$outputFile" `
--format av_mp4 `
--encoder x264 `
--vb 1700 `
--two-pass `
--aencoder copy:aac `
--ab 320 `
--arate 48 `
--mixdown stereo
}
switch -regex ($Item.Extension) {
'\.mp4' { mp4-Func }
}
Write-Host #WriteHostParams
}
}
#( 'Created') | ForEach-Object {
$FileSystemWatcherParams.EventName = $_
Start-FileSystemWatcher #FileSystemWatcherParams
}
I think you'll find that $ErrorActionPreference only impact errors at the level of cmdlets, while you hitting a problem in non-cmdlet code. For that, you'll probably need a try/catch construct.
Based on Burt_Harris' answer (please upvote his answer over this one), I changed the "do while" loop so that it would use try/catch rather than an if/else statement. By using $.Exception.Message and $.Exception.ItemName I was able to better understand why the script was halting at that particular line.
Working code:
Do {
Try {
[System.IO.FileStream] $fs = $convertingFile.OpenWrite()
$fs.Dispose()
$locked = 0
}
Catch {
$ErrorMessage = $_.Exception.Message
$FailedItem = $_.Exception.ItemName
Write-Host $ErrorMessage
Write-Host $FailedItem
Write-Host "Can't convert yet, file appears to be loading..."
sleep 5
continue
}
} until ($locked -eq 0)
I'm trying to pipe a string into a program's STDIN without any trailing linefeeds (unless that string itself actually ends in a linefeed). I tried googling around, but I only found people trying to print to the console without a trailing linefeed, in which case Write-Host takes a parameter -NoNewLine. However, to pipe it on to another program, I need Write-Output or similar which doesn't have such a parameter. Now it seems like Write-Output isn't even the problem:
Z:\> (Write-Output "abc").Length
3
But as soon as I pipe it to another program and read the string there, I get an additional linefeed. For instance, I tried this Ruby snippet:
Z:\> Write-Output "abc" | ruby -e "p ARGF.read"
"abc\n"
I checked that the actual string received is abc\n. The same happens in several other languages (at least C#, Java and Python), so I believe it's an issue with PowerShell, not the language doing the reading.
As a further test, I replaced Write-Output itself with another Ruby script:
Z:\> ruby -e "$> << 'abc'"
abcZ:\>
(That is, there is definitely no \n on the script's STDOUT.)
But again, when I pipe it into another script:
Z:\> ruby -e "$> << 'abc'" | ruby -e "p ARGF.read"
"abc\n"
I'm fairly convinced that it's the pipe which adds the linefeed. How do I avoid that? I actually want to be able to control whether the input ends in a linefeed or not (by including it in the input or omitting it).
(For reference, I also tested strings which already contain a trailing linefeed, and in that case the pipe doesn't add another one, so I guess it just ensures a trailing linefeed.)
I originally encountered this in PowerShell v3, but I'm now using v5 and still have the same issue.
Introduction
Here is my Invoke-RawPipeline function (get latest version from this Gist).
Use it to pipe binary data between processes' Standard Output and Standard Input streams. It can read input stream from file/pipeline and save resulting output stream to file.
It requires PsAsync module to be able to launch and pipe data in multiple processes.
In case of issues use -Verbose switch to see debug output.
Examples
Redirecting to file
Batch:
findstr.exe /C:"Warning" /I C:\Windows\WindowsUpdate.log > C:\WU_Warnings.txt
PowerShell:
Invoke-RawPipeline -Command #{Path = 'findstr.exe' ; Arguments = '/C:"Warning" /I C:\Windows\WindowsUpdate.log'} -OutFile 'C:\WU_Warnings.txt'
Redirecting from file
Batch:
svnadmin load < C:\RepoDumps\MyRepo.dump
PowerShell:
Invoke-RawPipeline -InFile 'C:\RepoDumps\MyRepo.dump' -Command #{Path = 'svnadmin.exe' ; Arguments = 'load'}
Piping strings
Batch:
echo TestString | find /I "test" > C:\SearchResult.log
PowerShell:
'TestString' | Invoke-RawPipeline -Command #{Path = 'find.exe' ; Arguments = '/I "test"'} -OutFile 'C:\SearchResult.log'
Piping between multiple processes
Batch:
ipconfig | findstr /C:"IPv4 Address" /I
PowerShell:
Invoke-RawPipeline -Command #{Path = 'ipconfig'}, #{Path = 'findstr' ; Arguments = '/C:"IPv4 Address" /I'} -RawData
Code:
<#
.Synopsis
Pipe binary data between processes' Standard Output and Standard Input streams.
Can read input stream from file and save resulting output stream to file.
.Description
Pipe binary data between processes' Standard Output and Standard Input streams.
Can read input stream from file/pipeline and save resulting output stream to file.
Requires PsAsync module: http://psasync.codeplex.com
.Notes
Author: beatcracker (https://beatcracker.wordpress.com, https://github.com/beatcracker)
License: Microsoft Public License (http://opensource.org/licenses/MS-PL)
.Component
Requires PsAsync module: http://psasync.codeplex.com
.Parameter Command
An array of hashtables, each containing Command Name, Working Directory and Arguments
.Parameter InFile
This parameter is optional.
A string representing path to file, to read input stream from.
.Parameter OutFile
This parameter is optional.
A string representing path to file, to save resulting output stream to.
.Parameter Append
This parameter is optional. Default is false.
A switch controlling wheither ovewrite or append output file if it already exists. Default is to overwrite.
.Parameter IoTimeout
This parameter is optional. Default is 0.
A number of seconds to wait if Input/Output streams are blocked. Default is to wait indefinetely.
.Parameter ProcessTimeout
This parameter is optional. Default is 0.
A number of seconds to wait for process to exit after finishing all pipeline operations. Default is to wait indefinetely.
Details: https://msdn.microsoft.com/en-us/library/ty0d8k56.aspx
.Parameter BufferSize
This parameter is optional. Default is 4096.
Size of buffer in bytes for read\write operations. Supports standard Powershell multipliers: KB, MB, GB, TB, and PB.
Total number of buffers is: Command.Count * 2 + InFile + OutFile.
.Parameter ForceGC
This parameter is optional.
A switch, that if specified will force .Net garbage collection.
Use to immediately release memory on function exit, if large buffer size was used.
.Parameter RawData
This parameter is optional.
By default function returns object with StdOut/StdErr streams and process' exit codes.
If this switch is specified, function will return raw Standard Output stream.
.Example
Invoke-RawPipeline -Command #{Path = 'findstr.exe' ; Arguments = '/C:"Warning" /I C:\Windows\WindowsUpdate.log'} -OutFile 'C:\WU_Warnings.txt'
Batch analog: findstr.exe /C:"Warning" /I C:\Windows\WindowsUpdate.log' > C:\WU_Warnings.txt
.Example
Invoke-RawPipeline -Command #{Path = 'findstr.exe' ; WorkingDirectory = 'C:\Windows' ; Arguments = '/C:"Warning" /I .\WindowsUpdate.log'} -RawData
Batch analog: cd /D C:\Windows && findstr.exe /C:"Warning" /I .\WindowsUpdate.log
.Example
'TestString' | Invoke-RawPipeline -Command #{Path = 'find.exe' ; Arguments = '/I "test"'} -OutFile 'C:\SearchResult.log'
Batch analog: echo TestString | find /I "test" > C:\SearchResult.log
.Example
Invoke-RawPipeline -Command #{Path = 'ipconfig'}, #{Path = 'findstr' ; Arguments = '/C:"IPv4 Address" /I'} -RawData
Batch analog: ipconfig | findstr /C:"IPv4 Address" /I
.Example
Invoke-RawPipeline -InFile 'C:\RepoDumps\Repo.svn' -Command #{Path = 'svnadmin.exe' ; Arguments = 'load'}
Batch analog: svnadmin load < C:\RepoDumps\MyRepo.dump
#>
function Invoke-RawPipeline
{
[CmdletBinding()]
Param
(
[Parameter(ValueFromPipeline = $true)]
[ValidateNotNullOrEmpty()]
[ValidateScript({
if($_.psobject.Methods.Match.('ToString'))
{
$true
}
else
{
throw 'Can''t convert pipeline object to string!'
}
})]
$InVariable,
[Parameter(ValueFromPipelineByPropertyName = $true)]
[ValidateScript({
$_ | ForEach-Object {
$Path = $_.Path
$WorkingDirectory = $_.WorkingDirectory
if(!(Get-Command -Name $Path -CommandType Application -ErrorAction SilentlyContinue))
{
throw "Command not found: $Path"
}
if($WorkingDirectory)
{
if(!(Test-Path -LiteralPath $WorkingDirectory -PathType Container -ErrorAction SilentlyContinue))
{
throw "Working directory not found: $WorkingDirectory"
}
}
}
$true
})]
[ValidateNotNullOrEmpty()]
[array]$Command,
[Parameter(ValueFromPipelineByPropertyName = $true)]
[ValidateScript({
if(!(Test-Path -LiteralPath $_))
{
throw "File not found: $_"
}
$true
})]
[ValidateNotNullOrEmpty()]
[string]$InFile,
[Parameter(ValueFromPipelineByPropertyName = $true)]
[ValidateScript({
if(!(Test-Path -LiteralPath (Split-Path $_)))
{
throw "Folder not found: $_"
}
$true
})]
[ValidateNotNullOrEmpty()]
[string]$OutFile,
[Parameter(ValueFromPipelineByPropertyName = $true)]
[switch]$Append,
[Parameter(ValueFromPipelineByPropertyName = $true)]
[ValidateRange(0, 2147483)]
[int]$IoTimeout = 0,
[Parameter(ValueFromPipelineByPropertyName = $true)]
[ValidateRange(0, 2147483)]
[int]$ProcessTimeout = 0,
[Parameter(ValueFromPipelineByPropertyName = $true)]
[long]$BufferSize = 4096,
[Parameter(ValueFromPipelineByPropertyName = $true)]
[switch]$RawData,
[Parameter(ValueFromPipelineByPropertyName = $true)]
[switch]$ForceGC
)
Begin
{
$Modules = #{PsAsync = 'http://psasync.codeplex.com'}
'Loading modules:', ($Modules | Format-Table -HideTableHeaders -AutoSize | Out-String) | Write-Verbose
foreach($module in $Modules.GetEnumerator())
{
if(!(Get-Module -Name $module.Key))
{
Try
{
Import-Module -Name $module.Key -ErrorAction Stop
}
Catch
{
throw "$($module.Key) module not available. Get it here: $($module.Value)"
}
}
}
function New-ConsoleProcess
{
Param
(
[string]$Path,
[string]$Arguments,
[string]$WorkingDirectory,
[switch]$CreateNoWindow = $true,
[switch]$RedirectStdIn = $true,
[switch]$RedirectStdOut = $true,
[switch]$RedirectStdErr = $true
)
if(!$WorkingDirectory)
{
if(!$script:MyInvocation.MyCommand.Path)
{
$WorkingDirectory = [System.AppDomain]::CurrentDomain.BaseDirectory
}
else
{
$WorkingDirectory = Split-Path $script:MyInvocation.MyCommand.Path
}
}
Try
{
$ps = New-Object -TypeName System.Diagnostics.Process -ErrorAction Stop
$ps.StartInfo.Filename = $Path
$ps.StartInfo.Arguments = $Arguments
$ps.StartInfo.UseShellExecute = $false
$ps.StartInfo.RedirectStandardInput = $RedirectStdIn
$ps.StartInfo.RedirectStandardOutput = $RedirectStdOut
$ps.StartInfo.RedirectStandardError = $RedirectStdErr
$ps.StartInfo.CreateNoWindow = $CreateNoWindow
$ps.StartInfo.WorkingDirectory = $WorkingDirectory
}
Catch
{
throw $_
}
return $ps
}
function Invoke-GarbageCollection
{
[gc]::Collect()
[gc]::WaitForPendingFinalizers()
}
$CleanUp = {
$IoWorkers + $StdErrWorkers |
ForEach-Object {
$_.Src, $_.Dst |
ForEach-Object {
if(!($_ -is [System.Diagnostics.Process]))
{
Try
{
$_.Close()
}
Catch
{
Write-Error "Failed to close $_"
}
$_.Dispose()
}
}
}
}
$PumpData = {
Param
(
[hashtable]$Cfg
)
# Fail hard, we don't want stuck threads
$Private:ErrorActionPreference = 'Stop'
$Src = $Cfg.Src
$SrcEndpoint = $Cfg.SrcEndpoint
$Dst = $Cfg.Dst
$DstEndpoint = $Cfg.DstEndpoint
$BufferSize = $Cfg.BufferSize
$SyncHash = $Cfg.SyncHash
$RunspaceId = $Cfg.Id
# Setup Input and Output streams
if($Src -is [System.Diagnostics.Process])
{
switch ($SrcEndpoint)
{
'StdOut' {$InStream = $Src.StandardOutput.BaseStream}
'StdIn' {$InStream = $Src.StandardInput.BaseStream}
'StdErr' {$InStream = $Src.StandardError.BaseStream}
default {throw "Not valid source endpoint: $_"}
}
}
else
{
$InStream = $Src
}
if($Dst -is [System.Diagnostics.Process])
{
switch ($DstEndpoint)
{
'StdOut' {$OutStream = $Dst.StandardOutput.BaseStream}
'StdIn' {$OutStream = $Dst.StandardInput.BaseStream}
'StdErr' {$OutStream = $Dst.StandardError.BaseStream}
default {throw "Not valid destination endpoint: $_"}
}
}
else
{
$OutStream = $Dst
}
$InStream | Out-String | ForEach-Object {$SyncHash.$RunspaceId.Status += "InStream: $_"}
$OutStream | Out-String | ForEach-Object {$SyncHash.$RunspaceId.Status += "OutStream: $_"}
# Main data copy loop
$Buffer = New-Object -TypeName byte[] $BufferSize
$BytesThru = 0
Try
{
Do
{
$SyncHash.$RunspaceId.IoStartTime = [DateTime]::UtcNow.Ticks
$ReadCount = $InStream.Read($Buffer, 0, $Buffer.Length)
$OutStream.Write($Buffer, 0, $ReadCount)
$OutStream.Flush()
$BytesThru += $ReadCount
}
While($readCount -gt 0)
}
Catch
{
$SyncHash.$RunspaceId.Status += $_
}
Finally
{
$OutStream.Close()
$InStream.Close()
}
}
}
Process
{
$PsCommand = #()
if($Command.Length)
{
Write-Verbose 'Creating new process objects'
$i = 0
foreach($cmd in $Command.GetEnumerator())
{
$PsCommand += New-ConsoleProcess #cmd
$i++
}
}
Write-Verbose 'Building I\O pipeline'
$PipeLine = #()
if($InVariable)
{
[Byte[]]$InVarBytes = [Text.Encoding]::UTF8.GetBytes($InVariable.ToString())
$PipeLine += New-Object -TypeName System.IO.MemoryStream -ArgumentList $BufferSize -ErrorAction Stop
$PipeLine[-1].Write($InVarBytes, 0, $InVarBytes.Length)
[Void]$PipeLine[-1].Seek(0, 'Begin')
}
elseif($InFile)
{
$PipeLine += New-Object -TypeName System.IO.FileStream -ArgumentList ($InFile, [IO.FileMode]::Open) -ErrorAction Stop
if($PsCommand.Length)
{
$PsCommand[0].StartInfo.RedirectStandardInput = $true
}
}
else
{
if($PsCommand.Length)
{
$PsCommand[0].StartInfo.RedirectStandardInput = $false
}
}
$PipeLine += $PsCommand
if($OutFile)
{
if($PsCommand.Length)
{
$PsCommand[-1].StartInfo.RedirectStandardOutput = $true
}
if($Append)
{
$FileMode = [System.IO.FileMode]::Append
}
else
{
$FileMode = [System.IO.FileMode]::Create
}
$PipeLine += New-Object -TypeName System.IO.FileStream -ArgumentList ($OutFile, $FileMode, [System.IO.FileAccess]::Write) -ErrorAction Stop
}
else
{
if($PsCommand.Length)
{
$PipeLine += New-Object -TypeName System.IO.MemoryStream -ArgumentList $BufferSize -ErrorAction Stop
}
}
Write-Verbose 'Creating I\O threads'
$IoWorkers = #()
for($i=0 ; $i -lt ($PipeLine.Length-1) ; $i++)
{
$SrcEndpoint = $DstEndpoint = $null
if($PipeLine[$i] -is [System.Diagnostics.Process])
{
$SrcEndpoint = 'StdOut'
}
if($PipeLine[$i+1] -is [System.Diagnostics.Process])
{
$DstEndpoint = 'StdIn'
}
$IoWorkers += #{
Src = $PipeLine[$i]
SrcEndpoint = $SrcEndpoint
Dst = $PipeLine[$i+1]
DstEndpoint = $DstEndpoint
}
}
Write-Verbose "Created $($IoWorkers.Length) I\O worker objects"
Write-Verbose 'Creating StdErr readers'
$StdErrWorkers = #()
for($i=0 ; $i -lt $PsCommand.Length ; $i++)
{
$StdErrWorkers += #{
Src = $PsCommand[$i]
SrcEndpoint = 'StdErr'
Dst = New-Object -TypeName System.IO.MemoryStream -ArgumentList $BufferSize -ErrorAction Stop
}
}
Write-Verbose "Created $($StdErrWorkers.Length) StdErr reader objects"
Write-Verbose 'Starting processes'
$PsCommand |
ForEach-Object {
$ps = $_
Try
{
[void]$ps.Start()
}
Catch
{
Write-Error "Failed to start process: $($ps.StartInfo.FileName)"
Write-Verbose "Can't launch process, killing and disposing all"
if($PsCommand)
{
$PsCommand |
ForEach-Object {
Try{$_.Kill()}Catch{} # Can't do much if kill fails...
$_.Dispose()
}
}
Write-Verbose 'Closing and disposing I\O streams'
. $CleanUp
}
Write-Verbose "Started new process: Name=$($ps.Name), Id=$($ps.Id)"
}
$WorkersCount = $IoWorkers.Length + $StdErrWorkers.Length
Write-Verbose 'Creating sync hashtable'
$sync = #{}
for($i=0 ; $i -lt $WorkersCount ; $i++)
{
$sync += #{$i = #{IoStartTime = $nul ; Status = $null}}
}
$SyncHash = [hashtable]::Synchronized($sync)
Write-Verbose 'Creating runspace pool'
$RunspacePool = Get-RunspacePool $WorkersCount
Write-Verbose 'Loading workers on the runspace pool'
$AsyncPipelines = #()
$i = 0
$IoWorkers + $StdErrWorkers |
ForEach-Object {
$Param = #{
BufferSize = $BufferSize
Id = $i
SyncHash = $SyncHash
} + $_
$AsyncPipelines += Invoke-Async -RunspacePool $RunspacePool -ScriptBlock $PumpData -Parameters $Param
$i++
Write-Verbose 'Started working thread'
$Param | Format-Table -HideTableHeaders -AutoSize | Out-String | Write-Debug
}
Write-Verbose 'Waiting for I\O to complete...'
if($IoTimeout){Write-Verbose "Timeout is $IoTimeout seconds"}
Do
{
# Check for pipelines with errors
[array]$FailedPipelines = Receive-AsyncStatus -Pipelines $AsyncPipelines | Where-Object {$_.Completed -and $_.Error}
if($FailedPipelines)
{
"$($FailedPipelines.Length) pipeline(s) failed!",
($FailedPipelines | Select-Object -ExpandProperty Error | Format-Table -AutoSize | Out-String) | Write-Debug
}
if($IoTimeout)
{
# Compare I\O start time of thread with current time
[array]$LockedReaders = $SyncHash.Keys | Where-Object {[TimeSpan]::FromTicks([DateTime]::UtcNow.Ticks - $SyncHash.$_.IoStartTime).TotalSeconds -gt $IoTimeout}
if($LockedReaders)
{
# Yikes, someone is stuck
"$($LockedReaders.Length) I\O operations reached timeout!" | Write-Verbose
$SyncHash.GetEnumerator() | ForEach-Object {"$($_.Key) = $($_.Value.Status)"} | Sort-Object | Out-String | Write-Debug
$PsCommand | ForEach-Object {
Write-Verbose "Killing process: Name=$($_.Name), Id=$($_.Id)"
Try
{
$_.Kill()
}
Catch
{
Write-Error 'Failed to kill process!'
}
}
break
}
}
Start-Sleep 1
}
While(Receive-AsyncStatus -Pipelines $AsyncPipelines | Where-Object {!$_.Completed}) # Loop until all pipelines are finished
Write-Verbose 'Waiting for all pipelines to finish...'
$IoStats = Receive-AsyncResults -Pipelines $AsyncPipelines
Write-Verbose 'All pipelines are finished'
Write-Verbose 'Collecting StdErr for all processes'
$PipeStdErr = $StdErrWorkers |
ForEach-Object {
$Encoding = $_.Src.StartInfo.StandardOutputEncoding
if(!$Encoding)
{
$Encoding = [System.Text.Encoding]::Default
}
#{
FileName = $_.Src.StartInfo.FileName
StdErr = $Encoding.GetString($_.Dst.ToArray())
ExitCode = $_.Src.ExitCode
}
} |
Select-Object #{Name = 'FileName' ; Expression = {$_.FileName}},
#{Name = 'StdErr' ; Expression = {$_.StdErr}},
#{Name = 'ExitCode' ; Expression = {$_.ExitCode}}
if($IoWorkers[-1].Dst -is [System.IO.MemoryStream])
{
Write-Verbose 'Collecting final pipeline output'
if($IoWorkers[-1].Src -is [System.Diagnostics.Process])
{
$Encoding = $IoWorkers[-1].Src.StartInfo.StandardOutputEncoding
}
if(!$Encoding)
{
$Encoding = [System.Text.Encoding]::Default
}
$PipeResult = $Encoding.GetString($IoWorkers[-1].Dst.ToArray())
}
Write-Verbose 'Closing and disposing I\O streams'
. $CleanUp
$PsCommand |
ForEach-Object {
$_.Refresh()
if(!$_.HasExited)
{
Write-Verbose "Process is still active: Name=$($_.Name), Id=$($_.Id)"
if(!$ProcessTimeout)
{
$ProcessTimeout = -1
}
else
{
$WaitForExitProcessTimeout = $ProcessTimeout * 1000
}
Write-Verbose "Waiting for process to exit (Process Timeout = $ProcessTimeout)"
if(!$_.WaitForExit($WaitForExitProcessTimeout))
{
Try
{
Write-Verbose 'Trying to kill it'
$_.Kill()
}
Catch
{
Write-Error "Failed to kill process $_"
}
}
}
Write-Verbose "Disposing process object: Name=$($_.StartInfo.FileName)"
$_.Dispose()
}
Write-Verbose 'Disposing runspace pool'
# http://stackoverflow.com/questions/21454252/how-to-cleanup-resources-in-a-dll-when-powershell-ise-exits-like-new-pssession
$RunspacePool.Dispose()
if($ForceGC)
{
Write-Verbose 'Forcing garbage collection'
Invoke-GarbageCollection
}
if(!$RawData)
{
New-Object -TypeName psobject -Property #{Result = $PipeResult ; Status = $PipeStdErr}
}
else
{
$PipeResult
}
}
}
Bruteforce approach: feed binary data to process' stdin. I've tested this code on the cat.exe from UnixUtils and it seems to do what you want:
# Text to send
$InputVar = "No Newline, No NewLine,`nNewLine, No NewLine,`nNewLine, No NewLine"
# Buffer & initial size of MemoryStream
$BufferSize = 4096
# Convert text to bytes and write to MemoryStream
[byte[]]$InputBytes = [Text.Encoding]::UTF8.GetBytes($InputVar)
$MemStream = New-Object -TypeName System.IO.MemoryStream -ArgumentList $BufferSize
$MemStream.Write($InputBytes, 0, $InputBytes.Length)
[Void]$MemStream.Seek(0, 'Begin')
# Setup stdin\stdout redirection for our process
$StartInfo = New-Object -TypeName System.Diagnostics.ProcessStartInfo -Property #{
FileName = 'MyLittle.exe'
UseShellExecute = $false
RedirectStandardInput = $true
}
# Create new process
$Process = New-Object -TypeName System.Diagnostics.Process
# Assign previously created StartInfo properties
$Process.StartInfo = $StartInfo
# Start process
[void]$Process.Start()
# Pipe data
$Buffer = New-Object -TypeName byte[] -ArgumentList $BufferSize
$StdinStream = $Process.StandardInput.BaseStream
try
{
do
{
$ReadCount = $MemStream.Read($Buffer, 0, $Buffer.Length)
$StdinStream.Write($Buffer, 0, $ReadCount)
$StdinStream.Flush()
}
while($ReadCount -gt 0)
}
catch
{
throw 'Houston, we have a problem!'
}
finally
{
# Close streams
$StdinStream.Close()
$MemStream.Close()
}
# Cleanup
'Process', 'StdinStream', 'MemStream' |
ForEach-Object {
(Get-Variable $_ -ValueOnly).Dispose()
Remove-Variable $_ -Force
}
Do it a simple way create a cmd process and execute it
$cmdArgs = #('/c','something.exe','arg1', .. , 'arg2' , $anotherArg , '<', '$somefile.txt' )
&'cmd.exe' $cmdArgs
Worked perfect for piping information into stdin that I wanted,
To clear up a fundamental misconception in some of the comments: the "powershell commands" in a pipeline are cmdlets and each one runs within the process space of the single powershell. Thus, objects are being passed as is within the same process (on multiple threads) UNLESS you invoke an external command. Then the passed objects are converted to strings by the appropriate formatting cmdlet (if not already string objects). These strings are then converted to a stream of characters with each string having an appended \n. So it is not "the pipeline" adding the \n but the implicit conversion to text for input to the "legacy" command.
The basic problem in the question is that the asker is trying to get object like behaviour (e.g. a string with no trailing \n) on a character (byte) stream input. The standard input stream of a (console) process supplies characters (bytes) one at a time. The input routines collect these individual characters into a single string (typically) terminated when a \n is received. Whether the \n is returned as part of the string is up to the input routine. When the standard input stream is redirected to a file or pipe, the input routines mostly have no knowledge of this. So there is no way to determine the difference between a complete string with no \n and an incomplete string with more characters and the \n still to come.
Possible solutions (to the string delimiting problem, not the powershell added \n problem) would be to have some sort of time-out on the standard input reads. The end of a string could be signaled by no received characters for a certain time. Alternatively, if you had low enough level access to the pipe you could try to have atomic reads and writes. In this way a blocked read would return exactly what was written. Unfortunately, both of these methods have timing problems when running within a multitasking environment. If the delay is long then efficiency falls but if it is too short then it can be fooled by delays caused by process priority scheduling. Scheduling priorities can also interfere with atomic reads and writes if the writing process writes another line before the reading process has read the current one. It would need some sort of synchronizing system.
The only other way to signal that there are no more characters coming on the current line would be to close the pipe (EOF) but this is a once only method so you can only send one string (trailing \n or not). (This is how Ruby knows when the input is finished both in the initial example and in the Invoke-RawPipeline example.) It is possible that this is actually your intention (send only one string with or without trailing \n) in which case you could simply concatenate all the input (retaining or re-inserting any embedded \n) and throw away the last \n.
A possible solution to the powershell added \n problem for multiple strings would be to redefine your encoding of "a string" by terminating each string object with an otherwise invalid character sequence. \0 could be used if you had per character input (no good for C-like line input) otherwise maybe \377 (0xff). This would allow the input routines of your legacy command to "know" when a string ended. The sequence \0\n (or \377\n) would be the "end" of the string and everything before that (including a trailing \n or not, possibly using multiple reads) would be the string. I am assuming that you have some control over the input routines (e.g. you wrote the program) since any off the shelf program reading from standard input would typically be expecting a \n (or EOF) to delimit its input.
I will admit to having zero experience with the ruby -e "puts ARGF.read" command you are using after the pipe, but I think I can prove that the pipe doesn't adding a newline.
# check length of string without newline after pipe
Write-Output "abc" | %{Write-Host "$_ has a length of: $($_.Length)" }
#check of string with newline length after pipe
Write-Output "def`n" | %{Write-Host "$($_.Length) is the length of $_" -NoNewline }
#write a string without newline (suppressing newline on Write-Host)
Write-Output 'abc' | %{ Write-Host $_ -NoNewline; }
#write a string with newline (suppressing newline on Write-Host)
Write-Output "def`n" | %{ Write-Host $_ -NoNewline; }
#write a final string without newline (suppressing newline on Write-Host)
Write-Output 'ghi' | %{ Write-Host $_ -NoNewline; }
This gives me an output of:
abc has a length of: 3
4 is the length of def
abcdef
ghi
I think you might want to start looking at the ruby -e "put AGRF.read" command and see if it is adding a newline after each read.