In Powershell with the function Start-Job you can call functions if you initialize them beforehand.
$export_functions =
{
Function Log($message) {Write-Host $message}
Function MyFunction($message) {Log($message)}
}
$param1="Hello stack overflow!"
Start-Job -ScriptBlock {MyFunction $using:param1} -InitializationScript $export_functions -ArgumentList #($param1) | Wait-Job | Receive-Job
Would it be possible to use a 'global' function inside the script block as well?
So something like:
Function Log($message) {Write-Host $message}
$export_functions =
{
Function Log($message) {$Function:Log($message)}
Function MyFunction($message) {Log($message)}
}
$param1="Hello stack overflow!"
Start-Job -ScriptBlock {MyFunction $using:param1} -InitializationScript $export_functions -ArgumentList #($param1) | Wait-Job | Receive-Job
Or is that not intended after all?
One way to capture the function definition from the caller's scope (defined outside the initialization script block) is to use an expandable (double-quoted) string ("..."), i.e. string interpolation, and then use [scriptblock]::Create() to create a script block from the result.
You can combine this technique with defining the code you want to place directly inside the script block as usual:
Function Log($message) { Write-Host $message }
# Use string interpolation to capture the body of function Log ($function:Log)
$export_functions = [scriptblock]::Create(
"Function Log { $function:Log }`n" +
{
Function MyFunction($message) { Log $message }
}
)
Start-Job -InitializationScript $export_functions { Log 'hi1'; MyFunction 'hi2' } |
Receive-Job -Wait -AutoRemoveJob
Note the use of $function:Log to refer to the Log function's body, which is an example of namespace variable notation.
Assuming Get-Foo and Get-Foo2 and Deploy-Jobs are 3 functions that are part of a very large module. I would like to use Get-Foo and Get-Foo2 in Deploy-Jobs's Start-ThreadJob (below) without reloading the entire module each time.
Is an working example available for how to do this?
function Deploy-Jobs {
foreach ($Device in $Devices) {
Start-ThreadJob -Name $Device -ThrottleLimit 50 -InitializationScript $initScript -ScriptBlock {
param($Device)
Get-Foo | Get-Foo2 -List
} -ArgumentList $Device | out-null
}
}
The method you can use to pass the function's definition to a different scope is the same for Invoke-Command (when PSRemoting), Start-Job, Start-ThreadJob and ForeEach-Object -Parallel. Since you want to invoke 2 different functions in your job's script block, I don't think -InitializationScript is an option, and even if it is, it might make the code even more complicated than it should be.
You can use this as an example of how you can store 2 function definitions in an array ($def), which is then passed to the scope of each TreadJob, this array is then used to define each function in said scope to be later used by each Job.
function Say-Hello {
"Hello world!"
}
function From-ThreadJob {
param($i)
"From ThreadJob # $i"
}
$def = #(
${function:Say-Hello}.ToString()
${function:From-ThreadJob}.ToString()
)
function Run-Jobs {
param($numerOfJobs, $functionDefinitions)
$jobs = foreach($i in 1..$numerOfJobs) {
Start-ThreadJob -ScriptBlock {
# bring the functions definition to this scope
$helloFunc, $threadJobFunc = $using:functionDefinitions
# define them in this scope
${function:Say-Hello} = $helloFunc
${function:From-ThreadJob} = $threadJobFunc
# sleep random seconds
Start-Sleep (Get-Random -Maximum 10)
# combine the output from both functions
(Say-Hello) + (From-ThreadJob -i $using:i)
}
}
Receive-Job $jobs -AutoRemoveJob -Wait
}
Run-Jobs -numerOfJobs 10 -functionDefinitions $def
I want to increment a variable in a PowerShell job with a number defined before the job starts.
I tried with a global variable but it didn't work, so now I try to write in a file and load it in my job but that didn't work either.
I summarize my loop:
$increment = 1
$Job_Nb = 1..3
foreach ($nb in $Job_Nb) {
$increment > "C:\increment.txt"
Start-Job -Name $nb -ScriptBlock {
$increment_job = Get-Content -Path "C:\increment.txt"
$increment_job
}
$increment++
}
I want my 2 variables $increment_job and $increment to be equal.
I obtain the good result with the command Wait-Job, like that:
$increment = 1
$Job_Nb = 1..3
foreach ($nb in $Job_Nb) {
$increment > "C:\increment.txt"
Start-Job -Name $nb -ScriptBlock {
$increment_job = Get-Content -Path "C:\increment.txt"
$increment_job
} | Wait-Job | Receive-Job
$increment++
}
But I can't wait each job to finish before starting the next, it's too long... I need to execute a lot of jobs in the background.
For me, even $nb, $increment and $increment_job can be equal.
If it can help you to understand, a really simple way to put it:
$nb = 1
$Job_Nb = 1..3
foreach ($nb in $Job_Nb) {
Start-Job -Name $nb -ScriptBlock {$nb}
$nb++
}
If you want the two variables to be equal, you can just pass $increment into your script block as an argument.
# Configure the Jobs
$increment = 1
$Job_Nb = 1..3
Foreach ($nb in $Job_Nb) {
Start-Job -Name $nb -ScriptBlock {
$increment_job = $args[0]
$increment_job
} -ArgumentList $increment
$increment++
}
# Retrieve the Jobs After Waiting For All to Complete
Wait-Job -Name $job_nb | Receive-Job
The problem with your initial approach as you have discovered is that PowerShell processes the entire loop before a single job completes. Therefore, the job doesn't read the increment.txt file until after its contents are set to 3.
Passing values into the -ArgumentList parameter of a script block without a parameter block will automatically assign the arguments to the $args array. Space delimited arguments will each become an element of the array. A value not space-separated can simply be retrieved as $args or $args[0] with the difference being $args will return a type of Object[] and $args[0] will return the type of the data you passed into it.
Obviously, you do not need to wait for all jobs to complete. You can just use Get-Job to retrieve whichever jobs you want at any time.
I'm basically building my own parallel foreach pipeline function, using runspaces.
My problem is: I call my function like this:
somePipeline | MyNewForeachFunction { scriptBlockHere } | pipelineGoesOn...
How can I pass the $_ parameter correctly into the ScriptBlock? It works when the ScriptBlock contains as first line
param($_)
But as you might have noticed, the powershell built-in ForEach-Object and Where-Object do not need such a parameter declaration in every ScriptBlock that is passed to them.
Thanks for your answers in advance
fjf2002
EDIT:
The goal is: I want comfort for the users of function MyNewForeachFunction - they shoudln't need to write a line param($_) in their script blocks.
Inside MyNewForeachFunction, The ScriptBlock is currently called via
$PSInstance = [powershell]::Create().AddScript($ScriptBlock).AddParameter('_', $_)
$PSInstance.BeginInvoke()
EDIT2:
The point is, how does for example the implementation of the built-in function ForEach-Object achieve that $_ need't be declared as a parameter in its ScriptBlock parameter, and can I use that functionality, too?
(If the answer is, ForEach-Object is a built-in function and uses some magic I can't use, then this would disqualify the language PowerShell as a whole in my opinion)
EDIT3:
Thanks to mklement0, I could finally build my general foreach loop. Here's the code:
function ForEachParallel {
[CmdletBinding()]
Param(
[Parameter(Mandatory)] [ScriptBlock] $ScriptBlock,
[Parameter(Mandatory=$false)] [int] $PoolSize = 20,
[Parameter(ValueFromPipeline)] $PipelineObject
)
Begin {
$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $poolSize)
$RunspacePool.Open()
$Runspaces = #()
}
Process {
$PSInstance = [powershell]::Create().
AddCommand('Set-Variable').AddParameter('Name', '_').AddParameter('Value', $PipelineObject).
AddCommand('Set-Variable').AddParameter('Name', 'ErrorActionPreference').AddParameter('Value', 'Stop').
AddScript($ScriptBlock)
$PSInstance.RunspacePool = $RunspacePool
$Runspaces += New-Object PSObject -Property #{
Instance = $PSInstance
IAResult = $PSInstance.BeginInvoke()
Argument = $PipelineObject
}
}
End {
while($True) {
$completedRunspaces = #($Runspaces | where {$_.IAResult.IsCompleted})
$completedRunspaces | foreach {
Write-Output $_.Instance.EndInvoke($_.IAResult)
$_.Instance.Dispose()
}
if($completedRunspaces.Count -eq $Runspaces.Count) {
break
}
$Runspaces = #($Runspaces | where { $completedRunspaces -notcontains $_ })
Start-Sleep -Milliseconds 250
}
$RunspacePool.Close()
$RunspacePool.Dispose()
}
}
Code partly from MathiasR.Jessen, Why PowerShell workflow is significantly slower than non-workflow script for XML file analysis
The key is to define $_ as a variable that your script block can see, via a call to Set-Variable.
Here's a simple example:
function MyNewForeachFunction {
[CmdletBinding()]
param(
[Parameter(Mandatory)]
[scriptblock] $ScriptBlock
,
[Parameter(ValueFromPipeline)]
$InputObject
)
process {
$PSInstance = [powershell]::Create()
# Add a call to define $_ based on the current pipeline input object
$null = $PSInstance.
AddCommand('Set-Variable').
AddParameter('Name', '_').
AddParameter('Value', $InputObject).
AddScript($ScriptBlock)
$PSInstance.Invoke()
}
}
# Invoke with sample values.
1, (Get-Date) | MyNewForeachFunction { "[$_]" }
The above yields something like:
[1]
[10/26/2018 00:17:37]
What I think you're looking for (and what I was looking for) is to support a "delay-bind" script block, supported in PowerShell 5.1+. The Microsoft documentation tells a bit about what's required, but doesn't provide any user-script examples (currently).
The gist is that PowerShell will implicitly detect that your function can accept a delay-bind script block if it defines an explicitly typed pipeline parameter (either by Value or by PropertyName), as long as it's not of type [scriptblock] or type [object].
function Test-DelayedBinding {
param(
# this is our typed pipeline parameter
# per doc this cannot be of type [scriptblock] or [object],
# but testing shows that type [object] may be permitted
[Parameter(ValueFromPipeline, Mandatory)][string]$string,
# this is our scriptblock parameter
[Parameter(Position=0)][scriptblock]$filter
)
Process {
if (&$filter $string) {
Write-Output $string
}
}
}
# sample invocation
>'foo', 'fi', 'foofoo', 'fib' | Test-DelayedBinding { return $_ -match 'foo' }
foo
foofoo
Note that the delay-bind will only be applied if input is piped into the function, and that the script block must use named parameters (not $args) if additional parameters are desired.
The frustrating part is that there is no way to explicitly specify that delay-bind should be used, and errors resulting from incorrectly structuring your function may be non-obvious.
Maybe this can help.
I'd normally run auto-generated jobs in parallel this way:
Get-Job | Remove-Job
foreach ($param in #(3,4,5)) {
Start-Job -ScriptBlock {param($lag); sleep $lag; Write-Output "slept for $lag seconds" } -ArgumentList #($param)
}
Get-Job | Wait-Job | Receive-Job
If I understand you correctly, you are trying to get rid of param() inside the scriptblock. You may try to wrap that SB with another one. Below is the workaround for my sample:
Get-Job | Remove-Job
#scriptblock with no parameter
$job = { sleep $lag; Write-Output "slept for $lag seconds" }
foreach ($param in #(3,4,5)) {
Start-Job -ScriptBlock {param($param, $job)
$lag = $param
$script = [string]$job
Invoke-Command -ScriptBlock ([Scriptblock]::Create($script))
} -ArgumentList #($param, $job)
}
Get-Job | Wait-Job | Receive-Job
# I was looking for an easy way to do this in a scripted function,
# and the below worked for me in PSVersion 5.1.17134.590
function Test-ScriptBlock {
param(
[string]$Value,
[ScriptBlock]$FilterScript={$_}
)
$_ = $Value
& $FilterScript
}
Test-ScriptBlock -Value 'unimportant/long/path/to/foo.bar' -FilterScript { [Regex]::Replace($_,'unimportant/','') }
Is there any way to run parallel programmed functions in PowerShell?
Something like:
Function BuildParallel($configuration)
{
$buildJob = {
param($configuration)
Write-Host "Building with configuration $configuration."
RunBuilder $configuration;
}
$unitJob = {
param()
Write-Host "Running unit."
RunUnitTests;
}
Start-Job $buildJob -ArgumentList $configuration
Start-Job $unitJob
While (Get-Job -State "Running")
{
Start-Sleep 1
}
Get-Job | Receive-Job
Get-Job | Remove-Job
}
Does not work because it complains about not recognizing "RunUnitTests" and "RunBuilder", which are functions declared in the same script file. Apparently this happens because the script block is a new context and does not know anything about the scripts declared in the same file.
I could try to use -InitializationScript in Start-Job, but both RunUnitTests and RunBuilder call more functions declared in the same file or referred from other files, so...
I'm sure there's a way to do this, since it's just modular programming (functions, routines and all that stuff).
You could have the functions in a separate file and import them into the current context wherever needed via dot sourcing. I do this in my Powershell profile, so some of my custom functions are available.
$items = Get-ChildItem "$PSprofilePath\functions"
$items | ForEach-Object {
. $_.FullName
}
If you wanted import one file, it would just be:
. C:\some\path\RunUnitTests.ps1