There is a script that has CmdletBinding attribute, which effectively makes it an “advanced” script. Inside the script I'm processing data in a pipeline in parallel, and I want the -WhatIf parameter to be passed down to the processing script block when I pass it to the script invocation.
Simplified code:
#Requires -Version 7.2
[CmdletBinding(SupportsShouldProcess = $true)]
param()
Get-ChildItem | ForEach-Object -Parallel {
if ($PSCmdlet.ShouldProcess("target", "operation")) {
Write-Host "Processing"
}
}
PS C:\> script.ps1 -WhatIf
InvalidOperation:
Line |
2 | if ($PSCmdlet.ShouldProcess("target", "operation")) {
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| You cannot call a method on a null-valued expression.
This does not work because $PSCmdlet is not defined in the script block.
When I replace $PSCmdlet with ($using:PSCmdlet), I get another error (only when -WhatIf is provided):
MethodInvocationException:
Line |
2 | if (($using:PSCmdlet).ShouldProcess("target", "operation")) {
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Exception calling "ShouldProcess" with "2" argument(s): "The WriteObject and WriteError methods cannot be called from outside the overrides of the BeginProcessing, ProcessRecord, and EndProcessing methods, and they can only be called from within the same thread. Validate that the cmdlet makes these calls correctly,
or contact Microsoft Customer Support Services."
Obviously, this happens because script blocks are executed in seaprate threads (“they can only be called from within the same thread”).
How to properly handle -WhatIf inside the script blocks of Foreach-Object -Parallel?
I've read this official article and seen this comment to the PowerShell issue #13816. Maybe another related issue: #14984.
As a side-note: specifying -WhatIf to the ForEach-Object itself doesn't make any difference in this case. This is also noticed here: https://thedavecarroll.com/powershell/foreach-object-whatif/#script-blocks-and--whatif
The closest I could get to something working, as you can see it is very cumbersome. First, as stated, Risk Management parameters do not seem to work in ForEach-Object -Parallel nor Start-ThreadJob. Using the function from this answer, -WhatIf seem to work properly, however the workaround does require an inner script block also supporting SupportsShouldProcess.
This is how the code would look:
[CmdletBinding(SupportsShouldProcess)]
param()
Get-ChildItem | Invoke-Parallel {
& {
[CmdletBinding(SupportsShouldProcess)]
param()
if ($PSCmdlet.ShouldProcess($_, "doing something")) {
Write-Host "Processing"
}
} -WhatIf:$WhatIf
} -Variables #{ WhatIf = [bool] $PSBoundParameters['WhatIf'] }
Related
I've encountered this issue in a longer script and have simplified here to show the minimal code required to reproduce it (I think). It outputs numbers followed by letters:
1 a
1 b
1 c...
2 a
2 b
2 c...
all the way to "500 z"
Function Write-HelloWorld
{
Param($number)
write-host -Object $number
}
$numbers = 1..500
$letters = "a".."z"
$Function = get-command Write-HelloWorld
$numbers | ForEach-Object -Parallel {
${function:Write-HelloWorld} = $using:Function
foreach($letter in $using:letters) {
Write-HelloWorld -number "$_ $letter"
}
}
I'm seeing 2 types of sporadically (not every time I run it):
"The term 'write-host' is not recognized as a name of a cmdlet, function, script file, or executable program." As understand it, write-host should always be available. Adding the line "Import-Module Microsoft.PowerShell.Utility" just before the call to write-host didn't help
Odd output like the below, specifically all the "write-host :" lines.
Santiago Squarzon's helpful answer demonstrates the problem with your approach well and links to a GitHub issue explaining the underlying problem (runspace affinity); however, that demonstration isn't the right solution (it wasn't meant to be), as it uses explicit synchronization to allow only one thread at a time to call the function, which negates the benefits of parallelism.
As for a solution:
You must pass a string representation of your Write-HelloWorld's function body to the ForEach-Object -Parallel call:
Function Write-HelloWorld
{
Param($number)
write-host -Object $number
}
$numbers = 1..500
$letters = "a".."z"
# Get the body of the Write-HelloWorld function *as a string*
# Alternative, as suggested by #Santiago:
# $funcDefString = (Get-Command -Type Function Write-HelloWorld).Definition
$funcDefString = ${function:Write-HelloWorld}.ToString()
$numbers | ForEach-Object -Parallel {
# Redefine the Write-HelloWorld function in this thread,
# using the *string* representation of its body.
${function:Write-HelloWorld} = $using:funcDefString
foreach($letter in $using:letters) {
Write-HelloWorld -number "$_ $letter"
}
}
${function:Write-HelloWorld} is an instance of namespace variable notation, which allows you to both get a function (its body as a [scriptblock] instance) and to set (define) it, by assigning either a [scriptblock] or a string containing the function body.
By passing a string, the function is recreated in the context of each thread, which avoids cross-thread issues that can arise when you pass a [System.Management.Automation.FunctionInfo] instance, as output by Get-Command, which contains a [scriptblock] that is bound to the runspace in which it was defined (i.e., the caller's; that script block is said to have affinity with the caller's runspace), and calling this bound [scriptblock] instance from other threads (runspaces) isn't safe.
By contrast, by redefining the function in each thread, via a string, a thread-specific [scriptblock] instance bound to that thread is created, which can safely be called.
In fact, you appear to have found a loophole, given that when you attempt to use a [scriptblock] instance directly with the $using: scope, the command by design breaks with an explicit error message:
A ForEach-Object -Parallel using variable cannot be a script block.
Passed-in script block variables are not supported with ForEach-Object -Parallel,
and can result in undefined behavior
In other words: PowerShell shouldn't even let you do what you attempted to do, but unfortunately does, as of PowerShell Core 7.2.7, resulting in the obscure failures you saw - see GitHub issue #16461.
Potential future improvement:
An enhancement is being discussed in GitHub issue #12240 to support copying the caller's state to the parallel threads on demand, which would automatically make the caller's functions available, without the need for manual redefinition.
Note, this answer is meant to prove a point but does not provide the correct solution to the problem.
See mklement0's helpful answer for the proper way to solve this by simply passing the function's definition as string to the runspaces. See also GitHub Issue #4003 for more details.
It's a very bad idea to pass in a reference object and use it without thread safety, here is proof that by simply adding thread safety to your code the problem is solved:
function Write-HelloWorld {
param($number)
Write-Host -Object $number
}
$numbers = 1..500
$letters = "a".."z"
$Function = Get-Command Write-HelloWorld
$numbers | ForEach-Object -Parallel {
$refObj = $using:Function
[System.Threading.Monitor]::Enter($refObj)
${function:Write-HelloWorld} = $using:Function
foreach($letter in $using:letters) {
Write-HelloWorld -number "$_ $letter"
}
[System.Threading.Monitor]::Exit($refObj)
}
To be precise, this issue is related to Runspace Affinity, all Runspaces are trying to send the invocation back to the origin Runspace thread hence poor PowerShell collapses.
I've encountered this issue in a longer script and have simplified here to show the minimal code required to reproduce it (I think). It outputs numbers followed by letters:
1 a
1 b
1 c...
2 a
2 b
2 c...
all the way to "500 z"
Function Write-HelloWorld
{
Param($number)
write-host -Object $number
}
$numbers = 1..500
$letters = "a".."z"
$Function = get-command Write-HelloWorld
$numbers | ForEach-Object -Parallel {
${function:Write-HelloWorld} = $using:Function
foreach($letter in $using:letters) {
Write-HelloWorld -number "$_ $letter"
}
}
I'm seeing 2 types of sporadically (not every time I run it):
"The term 'write-host' is not recognized as a name of a cmdlet, function, script file, or executable program." As understand it, write-host should always be available. Adding the line "Import-Module Microsoft.PowerShell.Utility" just before the call to write-host didn't help
Odd output like the below, specifically all the "write-host :" lines.
Santiago Squarzon's helpful answer demonstrates the problem with your approach well and links to a GitHub issue explaining the underlying problem (runspace affinity); however, that demonstration isn't the right solution (it wasn't meant to be), as it uses explicit synchronization to allow only one thread at a time to call the function, which negates the benefits of parallelism.
As for a solution:
You must pass a string representation of your Write-HelloWorld's function body to the ForEach-Object -Parallel call:
Function Write-HelloWorld
{
Param($number)
write-host -Object $number
}
$numbers = 1..500
$letters = "a".."z"
# Get the body of the Write-HelloWorld function *as a string*
# Alternative, as suggested by #Santiago:
# $funcDefString = (Get-Command -Type Function Write-HelloWorld).Definition
$funcDefString = ${function:Write-HelloWorld}.ToString()
$numbers | ForEach-Object -Parallel {
# Redefine the Write-HelloWorld function in this thread,
# using the *string* representation of its body.
${function:Write-HelloWorld} = $using:funcDefString
foreach($letter in $using:letters) {
Write-HelloWorld -number "$_ $letter"
}
}
${function:Write-HelloWorld} is an instance of namespace variable notation, which allows you to both get a function (its body as a [scriptblock] instance) and to set (define) it, by assigning either a [scriptblock] or a string containing the function body.
By passing a string, the function is recreated in the context of each thread, which avoids cross-thread issues that can arise when you pass a [System.Management.Automation.FunctionInfo] instance, as output by Get-Command, which contains a [scriptblock] that is bound to the runspace in which it was defined (i.e., the caller's; that script block is said to have affinity with the caller's runspace), and calling this bound [scriptblock] instance from other threads (runspaces) isn't safe.
By contrast, by redefining the function in each thread, via a string, a thread-specific [scriptblock] instance bound to that thread is created, which can safely be called.
In fact, you appear to have found a loophole, given that when you attempt to use a [scriptblock] instance directly with the $using: scope, the command by design breaks with an explicit error message:
A ForEach-Object -Parallel using variable cannot be a script block.
Passed-in script block variables are not supported with ForEach-Object -Parallel,
and can result in undefined behavior
In other words: PowerShell shouldn't even let you do what you attempted to do, but unfortunately does, as of PowerShell Core 7.2.7, resulting in the obscure failures you saw - see GitHub issue #16461.
Potential future improvement:
An enhancement is being discussed in GitHub issue #12240 to support copying the caller's state to the parallel threads on demand, which would automatically make the caller's functions available, without the need for manual redefinition.
Note, this answer is meant to prove a point but does not provide the correct solution to the problem.
See mklement0's helpful answer for the proper way to solve this by simply passing the function's definition as string to the runspaces. See also GitHub Issue #4003 for more details.
It's a very bad idea to pass in a reference object and use it without thread safety, here is proof that by simply adding thread safety to your code the problem is solved:
function Write-HelloWorld {
param($number)
Write-Host -Object $number
}
$numbers = 1..500
$letters = "a".."z"
$Function = Get-Command Write-HelloWorld
$numbers | ForEach-Object -Parallel {
$refObj = $using:Function
[System.Threading.Monitor]::Enter($refObj)
${function:Write-HelloWorld} = $using:Function
foreach($letter in $using:letters) {
Write-HelloWorld -number "$_ $letter"
}
[System.Threading.Monitor]::Exit($refObj)
}
To be precise, this issue is related to Runspace Affinity, all Runspaces are trying to send the invocation back to the origin Runspace thread hence poor PowerShell collapses.
Is there any command to list all functions I've created in a script?
Like i created function doXY and function getABC or something like this.
Then I type in the command and it shows:
Function doXY
Function getABC
Would be a cool feature^^
Thanks for all your help.
You can have PowerShell parse your script, and then locate the function definitions in the resulting Abstract Syntax Tree (AST).
Get-Command is probably the easiest way to access the AST:
# Use Get-Command to parse the script
$myScript = Get-Command .\path\to\script.ps1
$scriptAST = $myScript.ScriptBlock.AST
# Search the AST for function definitions
$functionDefinitions = $scriptAST.FindAll({
$args[0] -is [Management.Automation.Language.FunctionDefinitionAst]
}, $false)
# Report function name and line number in the script
$functionDefinitions |ForEach-Object {
Write-Host "Function '$($_.Name)' found on line $($_.StartLineNumber)!"
}
You can also use this to analyze the functions' contents and parameters if necessary.
Where your script is named things.ps1, something like...
cat ./things.ps1 | grep function
For MacOS/Linux or...
cat ./things.ps1 | select-string function
For Windows.
This is a built-in feature as shown in the PowerShell help files.
About_Providers
Similar questions have been asked before. So, this is a potential duplicate of:
How to get a list of custom Powershell functions?
Answers... Using the PSDrive feature
# To get a list of available functions
Get-ChildItem function:\
# To remove a powershell function
# removes `someFunction`
Remove-Item function:\someFunction
Or
Function Get-MyCommands {
Get-Content -Path $profile | Select-String -Pattern "^function.+" | ForEach-Object {
[Regex]::Matches($_, "^function ([a-z.-]+)","IgnoreCase").Groups[1].Value
} | Where-Object { $_ -ine "prompt" } | Sort-Object
}
Or this one
Get List Of Functions From Script
$currentFunctions = Get-ChildItem function:
# dot source your script to load it to the current runspace
. "C:\someScript.ps1"
$scriptFunctions = Get-ChildItem function: | Where-Object { $currentFunctions -notcontains $_ }
$scriptFunctions | ForEach-Object {
& $_.ScriptBlock
}
As for this...
Thanks, this is kind of what i want, but it also shows functions like
A:, B:, Get-Verb, Clear-Host, ...
That is by design. If you want it another way, then you have to code that.
To get name of functions in any script, it has to be loaded into memory first, then you can dot source the definition and get the internals. If you just want the function names, you can use regex to get them.
Or as simple as this...
Function Show-ScriptFunctions
{
[cmdletbinding()]
[Alias('ssf')]
Param
(
[string]$FullPathToScriptFile
)
(Get-Content -Path $FullPathToScriptFile) |
Select-String -Pattern 'function'
}
ssf -FullPathToScriptFile 'D:\Scripts\Format-NumericRange.ps1'
# Results
<#
function Format-NumericRange
function Flush-NumberBuffer
#>
This function will parse all the functions included in a .ps1 file, and then will return objects for each function found.
The output can be piped directly into Invoke-Expression to load the retuned functions into the current scope.
You can also provide an array of desired names, or a Regular Expression to constrain the results.
My use case was I needed a way for loading individual functions from larger scripts, that I don't own, so I could do pester testing.
Note: only tested in PowerShell 7, but I suspect it will work in older versions too.
function Get-Function {
<#
.SYNOPSIS
Returns a named function from a .ps1 file without executing the file
.DESCRIPTION
This is useful where you have a blended file containing functions and executed instructions.
If neither -Names nor -Regex are provided then all functions in the file are returned.
Returned objects can be piped directly into Invoke-Expression which will place them into the current scope.
Returns an array of objects with the following
- .ToString()
- .Name
- .Parameters
- .Body
- .Extent
- .IsFilter
- .IsWorkFlow
- .Parent
.PARAMETER -Names
Array of Strings; Optional
If provided then function objects of these names will be returned
The name must exactly match the provided value
Case Insensitive.
.PARAMETER -Regex
Regular Expression; Optional
If provided then function objects with names that match will be returned
Case Insensitive
.EXAMPLE
Get all the functions names included in the file
Get-Function -name TestA | select name
.EXAMPLE
Import a function into the current scope
Get-Function -name TestA | Invoke-Expression
#>
param (
$File = "c:\fullpath\SomePowerShellScriptFile.ps1"
,
[alias("Name", "FunctionNames", "Functions")]
$Names
,
[alias("NameRegex")]
$Regex
) # end function
# get the script and parse it
$Script = Get-Command /Users/royomi/Documents/dev/javascript/BenderBot_AI/Import-Function.ps1
$AllFunctions = $Script.ScriptBlock.AST.FindAll({$args[0] -is [Management.Automation.Language.FunctionDefinitionAst]}, $false)
# return all requested functions
$AllFunctions | Where-Object { `
( $Names -and $Names -icontains $_.Name ) `
-or ( $Regex -and $Names -imatch $Regex ) `
-or (-not $Names -and -not $Regex) `
} # end where-object
} # end function Get-Function
Having some problems getting a Start-Job script block to output to a file. The following three lines of code work without any problem:
$about_name = "C:\0\ps_about_name.txt"
$about = get-help about_* | select Name,Synopsis
if (-not (Test-
Path $about_name)) { ($about | select Name | sort Name | Out-String).replace("[Aa]bout_", "") > $about_name }
The file is created in C:\0\
But I need to do a lot of collections like this, so I naturally looked at stacking them in parallel as separate jobs. I followed online examples and so put the last line in the above as a script block invoked by Start-Job:
Start-Job { if (-not (Test-Path $about_name)) { { ($about | select Name | sort Name | Out-String).replace("[Aa]bout_", "") > $about_name } }
The Job is created, goes to status Running, and then to status Completed, but no file is created. Without Start-Job, all works, with Start-Job, nothing... I've tried a lot of variations on this but cannot get it to create the file. Can someone advise what I am doing wrong in this please?
IMO, the simplest way to get around this problem by use of the $using scope modifier.
$about_name = "C:\0\ps_about_name.txt"
$about = get-help about_* | select Name,Synopsis
$sb = { if (-not (Test-Path $using:about_name)) {
$using:about.Name -replace '^about_' | Sort-Object > $using:about_name
}
}
Start-Job -Scriptblock $sb
Explanation:
$using allows you to access local variables in a remote command. This is particularly useful when running Start-Job and Invoke-Command. The syntax is $using:localvariable.
This particular problem is a variable scope issue. Start-Job creates a background job with its own scope. When using -Scriptblock parameter, you are working within that scope. It does not know about variables defined in your current scope/session. Therefore, you must use a technique that will define the variable within the scope, pass in the variable's value, or access the local scope from the script block. You can read more about scopes at About_Scopes.
As an aside, character sets [] are not supported in the .NET .Replace() method. You need to switch to -replace to utilize those. I updated the code to perform the replace using -replace case-insensitively.
HCM's perfectly fine solution uses a technique that passes the value into the job's script block. By defining a parameter within the script block, you can pass a value into that parameter by use of -ArgumentList.
Another option is to just define your variables within the Start-Job script block.
$sb = { $about_name = "C:\0\ps_about_name.txt"
$about = get-help about_* | select Name,Synopsis
if (-not (Test-Path $about_name)) {
$about.Name -replace '^about_' | Sort-Object > $about_name
}
}
Start-Job -Scriptblock $sb
You've got to send your parameters to your job.
This does not work:
$file = "C:\temp\_mytest.txt"
start-job {"_" | out-file $file}
While this does:
$file = "C:\temp\_mytest.txt"
start-job -ArgumentList $file -scriptblock {
Param($file)
"_" | out-file $file
}
I find myself often tacking |select -first 10 onto the end of commands, and I'd like to shorten that to |s10, |s50, and a couple other variants. So I'd like to do the equivalent of set-alias s10 select-object -first 10.
The standard way you "alias" with bound parameters is to write a function and forward #args along with the extra params. But if I write a function that pipes $input through select-object, I lose streaming.
I could write a begin/process/end function, but I don't know if/how I can forward each of those to equivalents in select-object. I could write my own begin/process/end implementation of select-object that just implements the -first behavior, but that's just wrong...
(My fallback is to add a tab-completion to expand s10, but I'd really rather learn how I can implement a proper function.)
How can I implement a function that forwards to select-object with a parameter I want added but doesn't break pipelining?
I found this source https://blogs.technet.microsoft.com/heyscriptingguy/2011/03/01/proxy-functions-spice-up-your-powershell-core-cmdlets/
In this case, a shortcut/alias for Select-String -First 10, it comes down to:
$metadata = New-Object System.Management.Automation.CommandMetaData (Get-Command Select-Object)
[System.Management.Automation.ProxyCommand]::Create($MetaData) | Out-File -FilePath prxyfunctions.psm1
Open the prxyfunctions.psm1 module file and wrap the complete content in the new function called S10
function S10 {
[CmdletBinding(DefaultParameterSetName = 'DefaultParameter', HelpUri = 'https://go.microsoft.com/fwlink/?LinkID=113387', RemotingCapability = 'None')]
param(
<abbreviated...>
.ForwardHelpTargetName Microsoft.PowerShell.Utility\Select-Object
.ForwardHelpCategory Cmdlet
#>
}
Then in the Begin{} section add one statement $PSBoundParameters.Add('First','10') like below.
begin {
try {
$outBuffer = $null
if ($PSBoundParameters.TryGetValue('OutBuffer', [ref]$outBuffer)) {
$PSBoundParameters['OutBuffer'] = 1
}
$PSBoundParameters.Add('First','10')
$wrappedCmd = $ExecutionContext.InvokeCommand.GetCommand('Microsoft.PowerShell.Utility\Select-Object', [System.Management.Automation.CommandTypes]::Cmdlet)
$scriptCmd = {& $wrappedCmd #PSBoundParameters }
$steppablePipeline = $scriptCmd.GetSteppablePipeline($myInvocation.CommandOrigin)
$steppablePipeline.Begin($PSCmdlet)
}
catch {
throw
}
}
That's it. Save the file, import the module, type a nice short command like gci c:\|s10 and get just 10 results.
If you really want to make things error proof, more coding is needed. If S10 -First 2 is used you'll get a nice error thrown.
EDIT in response to #PetSerAl 's useful comments
Some cmdlets further in the pipeline might not be able to handle the proxied function, for instance Sort-Object. Compare the output of these two lines
-join (20..1 | Select -First 10 | Sort)
11121314151617181920
-join (20..1 | S10 | Sort)
<nothing>
-join (20..1 | S10 -Wait | Sort)
11121314151617181920
It is possible to work around that by using the -Wait parameter on the commandline. Or code the Wait parameter in the proxy function $PSBoundParameters.Add('Wait',$true)
When working with large collections this is unfortunate because it disables the Select-Object feature that stops the pipeline after x elements, resulting in more processing and longer waiting.
never use aliases in production scripts is what I would say (and is considered best practice). If its a bit of test code or something quick and dirty that no-one else will ever use, fair enough, but never in production scripts. Aliases can be removed, changed to run other commands and leave you with unintended results as they are user specific.