I normally do the following to invoke a script block containing $_:
$scriptBlock = { $_ <# do something with $_ here #> }
$theArg | ForEach-Object $scriptBlock
In effect, I am creating a pipeline which will give $_ its value (within the Foreach-Object function invocation).
However, when looking at the source code of the LINQ module, it defines and uses the following function to invoke the delegate:
# It is actually surprisingly difficult to write a function (in a module)
# that uses $_ in scriptblocks that it takes as parameters. This is a strange
# issue with scoping that seems to only matter when the function is a part
# of a module which has an isolated scope.
#
# In the case of this code:
# 1..10 | Add-Ten { $_ + 10 }
#
# ... the function Add-Ten must jump through hoops in order to invoke the
# supplied scriptblock in such a way that $_ represents the current item
# in the pipeline.
#
# Which brings me to Invoke-ScriptBlock.
# This function takes a ScriptBlock as a parameter, and an object that will
# be supplied to the $_ variable. Since the $_ may already be defined in
# this scope, we need to store the old value, and restore it when we are done.
# Unfortunately this can only be done (to my knowledge) by hitting the
# internal api's with reflection. Not only is this an issue for performance,
# it is also fragile. Fortunately this appears to still work in PowerShell
# version 2 through 3 beta.
function Invoke-ScriptBlock {
[CmdletBinding()]
param (
[Parameter(Position=1,Mandatory=$true)]
[ScriptBlock]$ScriptBlock,
[Parameter(ValueFromPipeline=$true)]
[Object]$InputObject
)
begin {
# equivalent to calling $ScriptBlock.SessionState property:
$SessionStateProperty = [ScriptBlock].GetProperty('SessionState',([System.Reflection.BindingFlags]'NonPublic,Instance'))
$SessionState = $SessionStateProperty.GetValue($ScriptBlock, $null)
}
}
process {
$NewUnderBar = $InputObject
$OldUnderBar = $SessionState.PSVariable.GetValue('_')
try {
$SessionState.PSVariable.Set('_', $NewUnderBar)
$SessionState.InvokeCommand.InvokeScript($SessionState, $ScriptBlock, #())
}
finally {
$SessionState.PSVariable.Set('_', $OldUnderBar)
}
}
}
This strikes me as a bit low-level. Is there a recommended, safe way of doing this?
You can invoke scriptblocks with the ampersand. No need to use Foreach-Object.
$scriptblock = {## whatever}
& $scriptblock
#(1,2,3) | % { & {write-host $_}}
To pass parameters:
$scriptblock = {write-host $args[0]}
& $scriptblock 'test'
$scriptBlock = {param($NamedParam) write-host $NamedParam}
& $scriptBlock -NamedParam 'test'
If you're going to be using this inside of Invoke-Command, you could also usin the $using construct.
$test = 'test'
$scriptblock = {write-host $using:test}
Related
The function Select-WriteHost from an answer to another Stackoverflow question (see code below) will redirect/capture Write-Host output:
Example:
PS> $test = 'a','b','c' |%{ Write-Host $_ } | Select-WriteHost
a
b
c
PS> $test
a
b
c
However, if I add -NoNewLine to Write-Host, Select-WriteHost will ignore it:
PS> $test = 'a','b','c' |%{ Write-Host -NoNewLine $_ } | Select-WriteHost
abc
PS> $test
a
b
c
Can anyone figure out how to modify Select-WriteHost (code below) to also support -NoNewLine?
function Select-WriteHost
{
[CmdletBinding(DefaultParameterSetName = 'FromPipeline')]
param(
[Parameter(ValueFromPipeline = $true, ParameterSetName = 'FromPipeline')]
[object] $InputObject,
[Parameter(Mandatory = $true, ParameterSetName = 'FromScriptblock', Position = 0)]
[ScriptBlock] $ScriptBlock,
[switch] $Quiet
)
begin
{
function Cleanup
{
# Clear out our proxy version of write-host
remove-item function:\write-host -ea 0
}
function ReplaceWriteHost([switch] $Quiet, [string] $Scope)
{
# Create a proxy for write-host
$metaData = New-Object System.Management.Automation.CommandMetaData (Get-Command 'Microsoft.PowerShell.Utility\Write-Host')
$proxy = [System.Management.Automation.ProxyCommand]::create($metaData)
# Change its behavior
$content = if($quiet)
{
# In quiet mode, whack the entire function body,
# simply pass input directly to the pipeline
$proxy -replace '(?s)\bbegin\b.+', '$Object'
}
else
{
# In noisy mode, pass input to the pipeline, but allow
# real Write-Host to process as well
$proxy -replace '(\$steppablePipeline\.Process)', '$Object; $1'
}
# Load our version into the specified scope
Invoke-Expression "function ${scope}:Write-Host { $content }"
}
Cleanup
# If we are running at the end of a pipeline, we need
# to immediately inject our version into global
# scope, so that everybody else in the pipeline
# uses it. This works great, but it is dangerous
# if we don't clean up properly.
if($pscmdlet.ParameterSetName -eq 'FromPipeline')
{
ReplaceWriteHost -Quiet:$quiet -Scope 'global'
}
}
process
{
# If a scriptblock was passed to us, then we can declare
# our version as local scope and let the runtime take
# it out of scope for us. It is much safer, but it
# won't work in the pipeline scenario.
#
# The scriptblock will inherit our version automatically
# as it's in a child scope.
if($pscmdlet.ParameterSetName -eq 'FromScriptBlock')
{
. ReplaceWriteHost -Quiet:$quiet -Scope 'local'
& $scriptblock
}
else
{
# In a pipeline scenario, just pass input along
$InputObject
}
}
end
{
Cleanup
}
}
PS: I tried inserting -NoNewLine to the line below (just to see how it would react) however, its producing the exception, "Missing function body in function declaration"
Invoke-Expression "function ${scope}:Write-Host { $content }"
to:
Invoke-Expression "function ${scope}:Write-Host -NoNewLine { $content }"
(Just to recap) Write-Host is meant for host, i.e. display / console output only, and originally couldn't be captured (in-session) at all. In PowerShell 5, the ability to capture Write-Host output was introduced via the information stream, whose number is 6, enabling techniques such as redirection 6>&1 in order to merge Write-Host output into the success (output) stream (whose number is 1), where it can be captured as usual.
However, due to your desire to use the -NoNewLine switch across several calls, 6>&1 by itself is not enough, because the concept of not emitting a newline only applies to display output, not to distinct objects in the pipeline.
E.g., in the following call -NoNewLine is effectively ignored, because there are multiple Write-Host calls producing multiple output objects (strings) that are captured separately:
'a','b','c' | % { Write-Host $_ -NoNewline } 6>&1
Your Select-WriteHost function - necessary in PowerShell 4 and below only - would have the same problem if you adapted it to support the -NoNewLine switch.
An aside re 6>&1: The strings that Write-Host invariably outputs are wrapped in [System.Management.Automation.InformationRecord] instances, due to being re-routed via the information stream. In display output you will not notice the difference, but to get the actual string you need to access the .MessageData.Message property or simply call .ToString().
There is no general solution I am aware of, but situationally the following may work:
If you know that the code of interest uses only Write-Host -NoNewLine calls:
Simply join the resulting strings after the fact without a separator to emulate -NoNewLine behavior:
# -> 'abc'
# Note: Whether or not you use -NoNewLine here makes no difference.
-join ('a','b','c' | % { Write-Host -NoNewLine $_ })
If you know that all instances of Write-Host -NoNewLine calls apply only to their respective pipeline input, you can write a simplified proxy function that collects all input up front and performs separator-less concatenation of the stringified objects:
# -> 'abc'
$test = & {
# Simplified proxy function
function Write-Host {
param([switch] $NoNewLine)
if ($MyInvocation.ExpectingInput) { $allInput = $Input }
else { $allInput = $args }
if ($NoNewLine) { -join $allInput.ForEach({ "$_" }) }
else { $allInput.ForEach({ "$_" }) }
}
# Important: pipe all input directly.
'a','b','c' | Write-Host -NoNewLine
}
I have a script that grabs a series of information from SQL. It then parses the information and passes it to a series of arrays. I want to then pass each array to a separate script.
I've seen Start-job should be able to do this but form my testing it didn't seem to work. This is what I have tried. Each Script individually works, and I am currently just using CVS's to pass the information.
Once the information is in the script I need to be able to call specific properties from each object. I did get it to just print the array as a string, but I couldn't call anything specific.
Invoke-Sqlcmd -Query $Q1 -ServerInstance $I -Database $DB | Export-Csv "$Files\Employees.csv"
$emps = Import-Csv "$Files\Employees.csv"
$newaccounts = #()
$deacaccounts = #()
$changedusers = #()
if(Test-Path -Path "$Files\Employees.csv"){
foreach ($emp in $emps) {
if ($emp.emp_num.trim() -ne $emp.EmpNum) {
$newaccounts += $emp
}
if ($emp.emp_num.trim() -eq $emp.EmpNum) {
if ($emp.fname -ne $emp.GivenName -and $emp.lname -ne $emp.SurName) {
$deacaccounts += $emp
$newaccounts += $emp
}
else ($emp.dept -ne $emp.DepartmentNumber -or $emp.job_title -ne $emp.JobTitle) {
$changedusers += $emp
}
}
}
}
Start-job -path "script" -argumentlist (,$deacaccounts)
Start-job -path "script" -argumentlist (,$changedusers)
Start-job -path "script" -argumentlist (,$newaccounts )
EDIT:
The Information passed to the scripts would be multiple lines of employee data. I need to be able to grab that info in the "Sub" scripts and perform actions based on them.
EX:
Deacaccounts =
fname
Lname
empnum
ted
kaz
1234
sam
cart
245
If you really need background jobs - it turns out that you don't - note that Start-Job doesn't have a -Path parameter; you'd have use -ScriptBlock { & "$script" } instead.
To simply invoke the script in the foreground, in sequence, use the following (script representing your .ps1 file path(s)):
& "script" $deacaccounts
& "script" $changedusers
& "script" $newaccounts
Note: &, the call operator, is only needed if the script / executable path is quoted and/or contains variable references (or subexpresions); e.g., a script with path c:\foo\bar.ps1 may be invoked without &; e.g.
c:\foo\bar.ps1 $deacaccounts
Note that your script(s) will receive a single argument each, containing an array of values.
If instead, you wanted to pass the array elements as individual (positional) arguments, you'd have to use splatting, where you use sigil # instead of $ to pass your variable (e.g.,
& "script" #deaccounts).
If you need to enumerate the arrays and pass each object individually as a parameter, use the following:
foreach ($obj in $deaccounts) { & "script" $obj }
foreach ($obj in $changedusers) { & "script" $obj }
foreach ($obj in $newaccounts) { & "script" $obj }
If each object should be splatted positionally based on its property values:
foreach ($obj in $deaccounts) {
$vals = $obj.psobject.Properties.Value
& "script" #vals
}
# ... ditto for $changeduser and $newaccounts
If each object should be splatted by property names, based on both property names and values, you need to convert each object to a hashtable first:
foreach ($obj in $deaccounts) {
$params = #{}
foreach ($prop in $obj.psobject.Properties) {
$params[$prop.Name] = $prop.Value
}
& "script" #params
}
# ... ditto for $changeduser and $newaccounts
As an aside: Incrementally extending arrays in a loop with += is inefficient, because a new array must be created behind the scenes in every iteration, because arrays are of fixed size.
In general, a much more efficient approach is to use a foreach loop as an expression and let PowerShell itself collect the outputs in an array: [array] $outputs = foreach (...) { ... } - see this answer.
In case you need to create arrays manually, e.g to create multiple ones, such as in your case, use an efficiently extensible list type, such as [System.Collections.Generic.List[object]] - see this answer.
I want to use start-job to run a .ps1 script requiring a parameter. Here's the script file:
#Test-Job.ps1
Param (
[Parameter(Mandatory=$True)][String]$input
)
$output = "$input to output"
return $output
and here is how I am running it:
$input = "input"
Start-Job -FilePath 'C:\PowerShell\test_job.ps1' -ArgumentList $input -Name "TestJob"
Get-Job -name "TestJob" | Wait-Job | Receive-Job
Get-Job -name "TestJob" | Remove-Job
Run like this, it returns " to output", so $input is null in the script run by the job.
I've seen other questions similar to this, but they mostly use -Scriptblock in place of -FilePath. Is there a different method for passing parameters to files through Start-Job?
tl;dr
$input is an automatic variable (value supplied by PowerShell) and shouldn't be used as a custom variable.
Simply renaming $input to, say, $InputObject solves your problem.
As Lee_Dailey notes, $input is an automatic variable and shouldn't be assigned to (it is automatically managed by PowerShell to provide an enumerator of pipeline input in non-advanced scripts and functions).
Regrettably and unexpectedly, several automatic variables, including $input, can be assigned to: see this answer.
$input is a particularly insidious example, because if you use it as a parameter variable, any value you pass to it is quietly discarded, because in the context of a function or script $input invariably is an enumerator for any pipeline input.
Here's a simple example to demonstrate the problem:
PS> & { param($input) "[$input]" } 'hi'
# !! No output - the argument was quietly discarded.
That the built-in definition of $input takes precedence can be demonstrated as follows:
PS> 'ho' | & { param($input) "[$input]" } 'hi'
ho # !! pipeline input took precedence
While you can technically get away with using $input as a regular variable (rather than a parameter variable) as long as you don't cross scope boundaries, custom use of $input should still be avoided:
& {
$input = 'foo' # TO BE AVOIDED
"[$input]" # Technically works: -> '[foo]'
& { "[$input]" } # FAILS, due to child scope: -> '[]'
}
I have following function:
function PipeScript {
param(
[Parameter(ValueFromPipeline)]
[Object] $InputObject,
[Object] $ScriptBlock
)
process {
$value = Invoke-Command -ScriptBlock $ScriptBlock
Write-Host "Script: $value"
}
}
When I define this function directly in script and pipe input into it I get following result which is expected:
#{ Name = 'Test' } | PipeScript -ScriptBlock { $_.Name }
# Outputs: "Script: Test"
But when I define this function inside module and export it with Export-ModuleMember -Function PipeScript then pipeline variable $_ inside script block is always null:
Import-Module PipeModule
#{ Name = 'Test' } | PipeScript -ScriptBlock { $_.Name }
# Outputs: "Script: "
Full repro is available at: https://github.com/lpatalas/DelayBindScriptBlock
Can someone explain this behaviour?
Tip of the hat to PetSerAl for all his help.
Here's a simple solution, but note that it runs the script block directly in the caller's scope, i.e. it effectively "dot-sources", which allows modification of the caller's variables.
By contrast, your use of Invoke-Command runs the script block in a child scope of the caller's scope - if that is truly the intent, see the variant solution below.
"Dot-sourcing" the script block is also what standard cmdlets such as Where-Object and ForEach-Object do.
# Define the function in an (in-memory) module.
# An in-memory module is automatically imported.
$null = New-Module {
function PipeScript {
param(
[Parameter(ValueFromPipeline)]
[Object] $InputObject
,
[scriptblock] $ScriptBlock
)
process {
# Use ForEach-Object to create the automatic $_ variable
# in the script block's origin scope.
$value = ForEach-Object -Process $ScriptBlock -InputObject $InputObject
# Output the value
"Script: $value"
}
}
}
# Test the function:
$var = 42; #{ Name = 'Test' } | PipeScript -ScriptBlock { $_.Name; ++$var }
$var # -> 43 - the script block ran in the caller's scope.
The above outputs string Script: Test and 43 afterwards, proving that the input object was seen as $_ and that dot-sourcing worked ($var was successfully incremented in the caller's scope).
Here's a variant, via the PowerShell SDK, that runs the script block in a child scope of the caller's scope.
This can be helpful if you don't want the execution of the script block to accidentally modify the caller's variables.
It is the same behavior you get with the engine-level delay-bind script-block and calculated-property features - though it's unclear whether that behavior was chosen intentionally.
$null = New-Module {
function PipeScript {
param(
[Parameter(ValueFromPipeline)]
[Object] $InputObject
,
[scriptblock] $ScriptBlock
)
process {
# Use ScriptBlock.InvokeContext() to inject a $_ variable
# into the child scope that the script block runs in:
# Creating a custom version of what is normally an *automatic* variable
# seems hacky, but the docs do state:
# "The list of variables may include the special variables
# $input, $_ and $this." - see https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.scriptblock.invokewithcontext
$value = $ScriptBlock.InvokeWithContext(
$null, # extra functions to define (none here)
[psvariable]::new('_', $InputObject) # actual parameter type is List<PSVariable>
)
# Output the value
"Script: $value"
}
}
}
# Test the function:
$var = 42
#{ Name = 'Test' } | PipeScript -ScriptBlock { $_.Name; ++$var }
$var # -> 42 - unaltered, because the script block ran in a child scope.
The above outputs string Script: Test, followed by 42, proving that the script block saw the input object as $_ and that variable $var - although seen in the script block, was not modified, due to running in a child scope.
The ScriptBlock.InvokeWithContext() method is documented here.
As for why your attempt didn't work:
Generally, script blocks are bound to the scope and scope domain in which they are created (except if they're created expressly as unbound script blocks, with [scriptblock]::Create('...')).
A scope outside of a module is part of the default scope domain. Every module has its own scope domain, and except for the global scope, which all scopes across all scope domains see, scopes in different scope domains do not see one another.
Your script block is created in the default scope domain, and when the module-defined function invokes it, the $_ is looked for in the scope of origin, i.e., in the (non-module) caller scope, where it isn't defined, because the automatic $_ variable is created by PowerShell on demand in the local scope, which is in the enclosing module's scope domain.
By using .InvokeWithContext(), the script block runs in a child scope of the caller's scope (as would be the case with .Invoke() and Invoke-Command by default), into which the above code injects a custom $_ variable so that the script block can reference it.
Providing better SDK support for these scenarios is being discussed in GitHub issue #3581.
I'm basically building my own parallel foreach pipeline function, using runspaces.
My problem is: I call my function like this:
somePipeline | MyNewForeachFunction { scriptBlockHere } | pipelineGoesOn...
How can I pass the $_ parameter correctly into the ScriptBlock? It works when the ScriptBlock contains as first line
param($_)
But as you might have noticed, the powershell built-in ForEach-Object and Where-Object do not need such a parameter declaration in every ScriptBlock that is passed to them.
Thanks for your answers in advance
fjf2002
EDIT:
The goal is: I want comfort for the users of function MyNewForeachFunction - they shoudln't need to write a line param($_) in their script blocks.
Inside MyNewForeachFunction, The ScriptBlock is currently called via
$PSInstance = [powershell]::Create().AddScript($ScriptBlock).AddParameter('_', $_)
$PSInstance.BeginInvoke()
EDIT2:
The point is, how does for example the implementation of the built-in function ForEach-Object achieve that $_ need't be declared as a parameter in its ScriptBlock parameter, and can I use that functionality, too?
(If the answer is, ForEach-Object is a built-in function and uses some magic I can't use, then this would disqualify the language PowerShell as a whole in my opinion)
EDIT3:
Thanks to mklement0, I could finally build my general foreach loop. Here's the code:
function ForEachParallel {
[CmdletBinding()]
Param(
[Parameter(Mandatory)] [ScriptBlock] $ScriptBlock,
[Parameter(Mandatory=$false)] [int] $PoolSize = 20,
[Parameter(ValueFromPipeline)] $PipelineObject
)
Begin {
$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $poolSize)
$RunspacePool.Open()
$Runspaces = #()
}
Process {
$PSInstance = [powershell]::Create().
AddCommand('Set-Variable').AddParameter('Name', '_').AddParameter('Value', $PipelineObject).
AddCommand('Set-Variable').AddParameter('Name', 'ErrorActionPreference').AddParameter('Value', 'Stop').
AddScript($ScriptBlock)
$PSInstance.RunspacePool = $RunspacePool
$Runspaces += New-Object PSObject -Property #{
Instance = $PSInstance
IAResult = $PSInstance.BeginInvoke()
Argument = $PipelineObject
}
}
End {
while($True) {
$completedRunspaces = #($Runspaces | where {$_.IAResult.IsCompleted})
$completedRunspaces | foreach {
Write-Output $_.Instance.EndInvoke($_.IAResult)
$_.Instance.Dispose()
}
if($completedRunspaces.Count -eq $Runspaces.Count) {
break
}
$Runspaces = #($Runspaces | where { $completedRunspaces -notcontains $_ })
Start-Sleep -Milliseconds 250
}
$RunspacePool.Close()
$RunspacePool.Dispose()
}
}
Code partly from MathiasR.Jessen, Why PowerShell workflow is significantly slower than non-workflow script for XML file analysis
The key is to define $_ as a variable that your script block can see, via a call to Set-Variable.
Here's a simple example:
function MyNewForeachFunction {
[CmdletBinding()]
param(
[Parameter(Mandatory)]
[scriptblock] $ScriptBlock
,
[Parameter(ValueFromPipeline)]
$InputObject
)
process {
$PSInstance = [powershell]::Create()
# Add a call to define $_ based on the current pipeline input object
$null = $PSInstance.
AddCommand('Set-Variable').
AddParameter('Name', '_').
AddParameter('Value', $InputObject).
AddScript($ScriptBlock)
$PSInstance.Invoke()
}
}
# Invoke with sample values.
1, (Get-Date) | MyNewForeachFunction { "[$_]" }
The above yields something like:
[1]
[10/26/2018 00:17:37]
What I think you're looking for (and what I was looking for) is to support a "delay-bind" script block, supported in PowerShell 5.1+. The Microsoft documentation tells a bit about what's required, but doesn't provide any user-script examples (currently).
The gist is that PowerShell will implicitly detect that your function can accept a delay-bind script block if it defines an explicitly typed pipeline parameter (either by Value or by PropertyName), as long as it's not of type [scriptblock] or type [object].
function Test-DelayedBinding {
param(
# this is our typed pipeline parameter
# per doc this cannot be of type [scriptblock] or [object],
# but testing shows that type [object] may be permitted
[Parameter(ValueFromPipeline, Mandatory)][string]$string,
# this is our scriptblock parameter
[Parameter(Position=0)][scriptblock]$filter
)
Process {
if (&$filter $string) {
Write-Output $string
}
}
}
# sample invocation
>'foo', 'fi', 'foofoo', 'fib' | Test-DelayedBinding { return $_ -match 'foo' }
foo
foofoo
Note that the delay-bind will only be applied if input is piped into the function, and that the script block must use named parameters (not $args) if additional parameters are desired.
The frustrating part is that there is no way to explicitly specify that delay-bind should be used, and errors resulting from incorrectly structuring your function may be non-obvious.
Maybe this can help.
I'd normally run auto-generated jobs in parallel this way:
Get-Job | Remove-Job
foreach ($param in #(3,4,5)) {
Start-Job -ScriptBlock {param($lag); sleep $lag; Write-Output "slept for $lag seconds" } -ArgumentList #($param)
}
Get-Job | Wait-Job | Receive-Job
If I understand you correctly, you are trying to get rid of param() inside the scriptblock. You may try to wrap that SB with another one. Below is the workaround for my sample:
Get-Job | Remove-Job
#scriptblock with no parameter
$job = { sleep $lag; Write-Output "slept for $lag seconds" }
foreach ($param in #(3,4,5)) {
Start-Job -ScriptBlock {param($param, $job)
$lag = $param
$script = [string]$job
Invoke-Command -ScriptBlock ([Scriptblock]::Create($script))
} -ArgumentList #($param, $job)
}
Get-Job | Wait-Job | Receive-Job
# I was looking for an easy way to do this in a scripted function,
# and the below worked for me in PSVersion 5.1.17134.590
function Test-ScriptBlock {
param(
[string]$Value,
[ScriptBlock]$FilterScript={$_}
)
$_ = $Value
& $FilterScript
}
Test-ScriptBlock -Value 'unimportant/long/path/to/foo.bar' -FilterScript { [Regex]::Replace($_,'unimportant/','') }