This lambda function executes as expected:
$WriteServerName = {
param($server)
Write-Host $server
}
$server = "servername"
$WriteServerName.invoke($server)
servername
However, using the same syntax, the following script prompts for credentials and then exits to the command line (running like this: .\ScriptName.ps1 -ConfigFile Chef.config), implying that the lambda functions aren't executing properly (for testing, each should just output the server name).
Why does the former lambda function return the server name, but the ones in the script don't?
Param(
$ConfigFile
)
Function Main {
#Pre-reqs: get credential, load config from file, and define lambda functions.
$jobs = #()
$Credential = Get-Credential
$Username = $Credential.username
$ConvertedPassword = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($Credential.password)
$Password = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($ConvertedPassword)
$Config = Get-Content $ConfigFile -Raw | Out-String | Invoke-Expression
#Define lambda functions
$BootStrap = {
param($Server)
write-host $server
}
$RunChefClient = {
param($Server)
write-host $server
}
$SetEnvironment = {
param($Server)
write-host $server
}
#Create bootstrap job for each server and pass lambda functions to Scriptblock for execution.
if(($Username -ne $null) -and ($Password -ne $null))
{
ForEach($HashTable in $Config)
{
$Server = $HashTable.Server
$Roles = $HashTable.Roles
$Tags = $HashTable.Tags
$Environment = $HashTable.Environment
$ScriptBlock = {
param ($Server,$BootStrap,$RunChefClient,$SetEnvironment)
$BootStrap.invoke($Server)
$RunChefClient.invoke($Server)
$SetEnvironment.invoke($Server)
}
$Jobs += Start-Job -ScriptBlock $ScriptBlock -ArgumentList #($Server,$BootStrap,$RunChefClient,$SetEnvironment)
}
}
else {Write-Host "Username or password is missing, exiting..." -ForegroundColor Red; exit}
}
Main
Without testing, I am going to go ahead and say it's because you are putting your scriptblock executions in PowerShell Jobs and then not doing anything with them. When you start a job, it starts a new PowerShell instance and executes the code you give it along with the parameters you give it. Once it completes, the completed PSRemotingJob object sits there and does nothing until you actually do something with it.
In your code, all the jobs you start are assigned to the $Jobs variable. You can also get all your running jobs with Get-Job:
Get-Job -State Running
If you want to get any of the data returned from your jobs, you'll have to use Receive-Job
# Either
$Jobs | Receive-Job
# Or
Get-Job -State Running | Receive-Job
Related
At [Asynchronous start][1] I had a question about starting a power-shell script asynchronously which creates a form. As answered in that question this can be solved using start-job
eg
Start-Job -ScriptBlock { test NW -NoWarning -Paranoia:2 }
So I have tried to write the Test.PS1 script routine so it re-calls itself with "Start-Job Test -NoSpawn" The switch nospawn then means it runs without a second call. I have tested this with the example code the above line now has to be and it works
Start-Job -ScriptBlock { test NW -NoSpawn -NoWarning -Paranoia:2 }
However I'm struggling to get the parameters from the original command line to passthrough to the job
I have tried creating a string in the correct format , an array , list the arguments manually , I either get repeated arguments being passed or all of the string ending up in the first Parameter $ComputerList -
A summary of the parameters and the attempts are
Param ([string]$ComputerList = 'status\edi.csv',[switch]$NoSpawn,[switch]$NoWarning,[switch]$Debug,[INT]$Paranoia=6)
...... <Snip>
Start-Job -ScriptBlock { test $ComputerList -NoSpawn -NoWarning:$NoWarning -Paranoia:$Paranoia }
Doesn't work due to scope - also switches are wrong way to do this
Start-Job -ScriptBlock { test -NoSpawn $Args } -argumentlist $ComputerList
Insufficent arguments but works - But I think One Argument is possible ?
Start-Job -ScriptBlock { test $Args -NoSpawn } -argumentlist #("-NoWarning:$NoWarning","-ComputerList:$ComputerList","-Paranoia:$Paranoia")
Everything ends up in $ComputerList
Start-Job -ScriptBlock { test $Args -NoSpawn } -argumentlist "-NoWarning:$NoWarning -ComputerList:$ComputerList -Paranoia:$Paranoia"
Everything ends up in $ComputerList
Full code follows
Param ([string]$ComputerList = 'status\edi.csv',[switch]$NoSpawn,[switch]$NoWarning,[switch]$Debug,[INT]$Paranoia=6)
$Log_Paranoia=$Paranoia
If ($Debug) { $debugPreference="Continue"} #enable debug messages if -debug is specified
If ($NoWarning) { $WarningPreference="SilentlyContinue"} #turn off warning messages
function Write-Paranoid($Level, $message) {
$CS=Get-PSCallStack
$Caller = $CS[1]
$Module = "$($Caller.FunctionName)[$($Caller.ScriptLineNumber)]"
$Diff=$level - $Log_Paranoia
$MSG= "$Module($($Level),$($Log_Paranoia)):$message"
if ($level - $Log_Paranoia -le 0 ) {
Write-host $MSG
}
if($Error.Count -gt 0 ) {
$MSG= "$Module($Level)ERROR:$Error[0]"
Write-Error $MSG
}
$error.clear()
}
Function AddStatusBar($form , $Txt) {
Write-Paranoid 10 "Enter"
$statusBar = New-Object System.Windows.Forms.StatusBar
$statusBar.DataBindings.DefaultDataSourceUpdateMode = 0
$statusBar.TabIndex = 4
$statusBar.Size = SDS 428 22
$statusBar.Location = SDP 0 337
$statusBar.Text = $Txt
$form.Controls.Add($statusBar)
$statusBar
Write-Paranoid 10 "Exit"
}
Function Create-Form ($Title)
{
Write-Paranoid 10 "Enter"
$form1 = New-Object System.Windows.Forms.Form
$form1.Text = $Title
$form1.DataBindings.DefaultDataSourceUpdateMode = 0
$form1.ClientSize = SDS 890 359
$form1.StartPosition = 0
$form1.BackColor = [System.Drawing.Color]::FromArgb(255,185,209,234)
$form1
Write-Paranoid 10 "Exit"
}
Function GenerateTestForm
{
Write-Paranoid 10 "Enter"
[reflection.assembly]::loadwithpartialname("System.Drawing") | Out-Null
[reflection.assembly]::loadwithpartialname("System.Windows.Forms") | Out-Null
$Form1 = Create-Form "Test Form"
$Alist = Get-CommandLine
$StatusBar = AddStatusBar $form1 $AList
$form1.ShowDialog() | Out-Null # Suspends calller
Write-Paranoid 10 "Exit"
}
if ($NoSpawn )
{
Write-Paranoid 3 " NoSpawn "
Write-Paranoid 5 "Call GenerateForm"
if ($Test) {
GenerateTestForm
} else {
GenerateTestForm
}
} else {
Write-Paranoid 3 "NOT NoSpawn restarting as job"
# note that test.ps1 is in the path so it will restart this script
# Start-Job -ScriptBlock { test $ComputerList -NoSpawn -NoWarning:$NoWarning -Paranoia:$Paranoia } #Wrong scope
# Start-Job -ScriptBlock { test -NoSpawn $Args } -argumentlist $ComputerList # Insufficent aruments but works - ONLY One Argument possible -
# Start-Job -ScriptBlock { test $Args -NoSpawn } -argumentlist #("-NoWarning:$NoWarning","-ComputerList:$ComputerList","-Paranoia:$Paranoia") # Everything ends up in $ComputerList
# Start-Job -ScriptBlock { test $Args -NoSpawn } -argumentlist "-NoWarning:$NoWarning -ComputerList:$ComputerList -Paranoia:$Paranoia" # Everything ends up in $ComputerList
}
Your problem can be reduced to this:
How can I re-invoke the script at hand as a background job, passing all original arguments (parameter values, including default parameter values) through?
A simplified example:
param (
[string] $ComputerList = 'status\edi.csv',
[switch] $NoSpawn,
[switch] $NoWarning,
[switch] $Debug,
[int] $Paranoia=6
)
if ($NoSpawn) { # already running as a background job.
"I'm now running in the background with the following arguments:"
$PSBoundParameters
} else { # must re-invoke via a background job
# Add *default* parameter values, if necessary, given that
# they're *not* reflected in $PSBoundParameters.
foreach ($paramName in $MyInvocation.MyCommand.Parameters.Keys) {
if (-not $PSBoundParameters.ContainsKey($paramName)) {
$defaultValue = Get-Variable -Scope Local -ValueOnly $paramName
if (-not ($null -eq $defaultValue -or ($defaultValue -is [switch] -and -not $defaultValue))) {
$PSBoundParameters[$paramName] = $defaultValue
}
}
}
# Start a background job that reinvokes this script with the original
# arguments / default values.
Start-Job {
$argsHash = $using:PSBoundParameters
& $using:PSCommandPath -NoSpawn #argsHash
} |
Receive-Job -Wait -AutoRemoveJob
}
Note:
For demonstration purposes, the initial call waits for the re-invocation via a background job to finish, using Receive-Job -Wait -AutoRemoveJob
In your real code, you can simply discard Start-Job's output (a job-information object) with $null = Start-Job { ... }, and then rely on the job getting cleaned up when the caller's session as a whole exits.
The extra code needed to propagate parameter default values is somewhat cumbersome, but necessary, given that the automatic $PSBoundParameters variable does not reflect default values.
GitHub issue #3285 discusses this limitation, and suggests a potential future solution.
I have developed a script which does a lot of processing for a front end tool, now I am attempting to have the script run with multiple threads. It interacts a lot with SQL databases, this should not be a problem for multithreading as the database transactions are very short lived, and the queries well optimised.
what is the issue ?
#.\tester.ps1 -servers (1,'Server1',3,1),(2,'Server2',3,1) -output_folder 'C:\temp'
param ([array[]]$servers),$output_folder
for ($i = 0; $i -lt $servers.Count; $i++)
{
$myserverid = $servers[$i][0]
$myservername = $servers[$i][1]
$mylocationid = $servers[$i][2]
$myappid = $servers[$i][3]
write-output " $myserverid and $myservername and $mylocationid and $myappid"
invoke-sqlcmd -ServerInstance "$myservername" -query "select top 10 name from sysobjects" -Database "master"
}
The script file above will gets passed an array of servers and currently it will loop through the array one by one. A way for me to make the process faster is to run the script in parallel /run the script with multiple threads.
Research
I have looked at a technet script on https://gallery.technet.microsoft.com/scriptcenter/Run-a-PowerShell-script-991c8a42
Its not quite the same as my array is not just a list of servers, there will be other parameters sent with it.
What am I after
A way or pointer to make the script be able to run in parallel or an example using the provided script above.
Thanks in advance.
Extending my comment. In PowerShell v5, use Jobs and Workflows for Parallel use cases.
# Example using parallel jobs
$start = Get-Date
# get all hotfixes
$task1 = { Get-Hotfix }
# get all scripts in your profile
$task2 = { Get-Service | Where-Object Status -eq Running }
# parse log file
$task3 = { Get-Content -Path $env:windir\windowsupdate.log | Where-Object { $_ -like '*successfully installed*' } }
# run 2 tasks in the background, and 1 in the foreground task
$job1 = Start-Job -ScriptBlock $task1
$job2 = Start-Job -ScriptBlock $task2
$result3 = Invoke-Command -ScriptBlock $task3
# wait for the remaining tasks to complete (if not done yet)
$null = Wait-Job -Job $job1, $job2
# now they are done, get the results
$result1 = Receive-Job -Job $job1
$result2 = Receive-Job -Job $job2
# discard the jobs
Remove-Job -Job $job1, $job2
$end = Get-Date
# Example, using WorkFlow
workflow Test-WFConnection
{
param
(
[string[]]$Computers
)
foreach -parallel ($computer in $computers)
{
Test-Connection -ComputerName $computer -Count 1 -ErrorAction SilentlyContinue
}
}
Trying to user start-parallel module but receiving not output. While debugging I could see that it goes to that line but never enter function that I have provided
I have tried using a script block and as a function as well. Went through example provided for Start-parallel module but no result. Be is a script block or a function it showcases a bar that executing two tasks but nothing gets executed
Function avolume{
param ($SCID_server_profile)
#foreach ($SCID_server_profile in $SCID_JSON.serverProfile)
#{
if ($SCID_server_profile.spdefName -like "spdef_workload*")
{
#---------------------------------
#Getting Server Profile Details
#---------------------------------
$SCID_frame_number = $SCID_server_profile.synergyFrame
$SCID_bay_number = $SCID_server_profile.bayNumber
$OV_server_profile_uri = (Get-HPOVServer -Name "$SCID_frame_number, Bay $SCID_bay_number").serverProfileUri
if ($OV_server_profile_uri)
{
$OV_server_profile = Send-HPOVRequest -uri $OV_server_profile_uri
$result = Add-SanVolume -ServerProfile $SCID_server_profile -serverprofileobj $OV_server_profile
Report-Row -reportwriter $reportwriter -category $result.category -status $result.status -description $result.description
if($result.status -eq "fail")
{
$Script:status = $false
}
else{$Successful_Configs += 1}
}
else
{
Report-Row -reportwriter $reportwriter -category "Update Server Profile" -status "Fail" -description "Server Profile does not exist for $SCID_frame_number, $SCID_frame_number"
$Script:status = $false
}
}
#}
}
#$SCID_JSON.serverProfile | Start-Parallel -ScriptBlock $Script
#Start-Parallel -InputObject $SCID_JSON.serverProfile -ScriptBlock $Script
$SCID_JSON.serverProfile | Start-Parallel -Scriptblock ${Function:\avolume}
I am expecting that whatever is in function should get executed in parallel for values that I am providing
I was reading this post about getting functions passed into a scriptblock for use with jobs:
Powershell start-job -scriptblock cannot recognize the function defined in the same file?
I get how that works by passing the function in as variable and it works for the simple example. What about a real world solution though, is there a more elegant way of handling this?
I have script I'm using to deploy changes to vendor software. It reads an xml that tells it how to navigate the environment and performs the various tasks, ie: map drives, stop services, call a perl installation script. I would like to provide a parameter to the script to allow it to run concurrently, this way if the perl script takes 5 minutes (not uncommon) and you're rolling out to 11 servers you're not waiting for the script to run for an hour.
I'm just going to post some snippets since the full script is a little lengthy. A log function:
function Log
{
Param(
[parameter(ValueFromPipeline=$true)]
$InputObject,
[parameter()]
[alias("v")]
$verbosity = $debug
)
$messageIndex = [array]::IndexOf($verbosityArray, $verbosity)
$verbosityIndex = [array]::IndexOf($verbosityArray, $loggingVerbosity)
if($messageIndex -ge $VerbosityIndex)
{
switch($verbosity)
{
$debug {Write-Host $verbosity ": " $InputObject}
$info {Write-Host $verbosity ": " $InputObject}
$warn {Write-Host $verbosity ": " $InputObject -ForegroundColor yellow}
$error {Write-Host $verbosity ": " $InputObject -ForegroundColor red}
}
}
}
Here's another function that calls the log function:
function ExecuteRollout
{
param(
[parameter(Mandatory=$true)]
[alias("ses")]
$session,
[parameter(Mandatory=$true)]
$command
)
#invoke command
Invoke-Command -session $session -ScriptBlock {$res = cmd /v /k `"$args[0]`"} -args $command
#get the return code from the remote session
$res = Invoke-Command -session $session {$res}
Log ("Command Output: "+$res)
$res = [string] $res
$exitCode = $res.substring($res.IndexOf("ExitCode:"), 10)
$exitCode = $exitCode.substring(9,1)
Log ("Exit code: "+$exitCode)
return $exitCode
}
And lastly a snippet from my main so you can get an idea of what's going on. $target.Destinations.Destination will contain all the servers and relevant information about them that the deployment will go to. I removed some variable setup and logging to make this more compact so yes you'll see variables referenced that are never defined:
#Execute concurrently
$target.Destinations.Destination | %{
$ScriptBlock = {
$destination = $args[0]
Log -v $info ("Starting remote session on: "+$destination.Server)
$session = New-PSSession -computerName $destination.Server
$InitializeRemote -session $session -destination $destination
#Gets a little tricky here, we need to keep the cmd session so it doesn't lose the sys vars set by env.bat
#String everything together with &'s
$cmdString = $destDrive + ": & call "+$lesDestDir+"data\env.bat & cd "+$rolloutDir+" & perl ..\JDH-rollout-2010.pl "+$rollout+" NC,r:\les & echo ExitCode:!errorlevel!"
Log ("cmdString: "+$cmdString)
Log -v $info ("Please wait, executing the rollout now...")
$exitCode = $ExecuteRollout -session $session -command $cmdString
Log ("ExitCode: "+$exitCode)
#respond to return code from rollout script
$HandleExitCode -session $session -destination $destination -exitCode $exitCode
$CleanUpRemote -session $session -destination $destination
}
Start-Job $ScriptBlock -Args $_
}
So if i go with the approach in the link I'd be converting all my functions to variables and passing them in to the script block. Currently, my log function will by default log in DEBUG unless the verbosity parameter is explicitly passed as a different verbosity. If I convert my functins to variables however powershell doesn't seem to like this syntax:
$Log ("Print this to the log")
So I think I'd need to use the parameter all the time now:
$Log ("Print this to the log" -v $debug
So bottom line it looks like I just need to pass all my functions as variables to the script block and change some formatting when I call them. It's not a huge effort, but I'd like to know if there's a better way before I start hacking my script up. Thanks for the input and for looking, I know this is quite a long post.
I started another post about passing parameters to functions stored as variables, the answer to that also resolves this issue. That post can be found here:
Powershell: passing parameters to functions stored in variables
The short answer is you can use the initializationscript parameter of Start-Job to feed all your functions in if you wrap them in a block and store that in a variable.
Example:
# concurrency
$func = {
function Logx
{
param(
[parameter(ValueFromPipeline=$true)]
$msg
)
Write-Host ("OUT:"+$msg)
}
}
# Execution starts here
cls
$colors = #("red","blue","green")
$colors | %{
$scriptBlock =
{
Logx $args[0]
Start-Sleep 9
}
Write-Host "Processing: " $_
Start-Job -InitializationScript $func -scriptblock $scriptBlock -args $_
}
Get-Job
while(Get-Job -State "Running")
{
write-host "Running..."
Start-Sleep 2
}
# Output
Get-Job | Receive-Job
# Cleanup jobs
Remove-Job *
I'm trying to get a simple working example of using functions inside of jobs. I've managed to pass my function into the scriptblock used for my job, but I can't seem to get parameters to the function.
# concurrency
$Logx =
{
param(
[parameter(ValueFromPipeline=$true)]
$msg
)
Write-Host ("OUT:"+$msg)
}
# Execution starts here
cls
$colors = #("red","blue","green")
$colors | %{
$scriptBlock =
{
Invoke-Expression -Command $args[1]
Start-Sleep 3
}
Write-Host "Processing: " $_
Start-Job -scriptblock $scriptBlock -args $_, $Logx
}
Get-Job
while(Get-Job -State "Running")
{
write-host "Running..."
Start-Sleep 2
}
# Output
Get-Job | Receive-Job
# Cleanup jobs
Remove-Job *
Here's the output:
Processing: red
Id Name State HasMoreData Location Command
-- ---- ----- ----------- -------- -------
175 Job175 Running True localhost ...
Processing: blue
177 Job177 Running True localhost ...
Processing: green
179 Job179 Running True localhost ...
179 Job179 Running True localhost ...
177 Job177 Running True localhost ...
175 Job175 Running True localhost ...
Running...
Running...
OUT:
OUT:
OUT:
So as evidenced by the OUT: x3 in the output my function is getting called, but I haven't found any syntax that allows me to get the parameter to the function. Thoughts?
EDIT:
Note in Shawn's observation below and my response I tried using functions as variables because using a traditional function does not seem to work. If there is a way to get that working I'd be more than happy to not have to pass my functions around as variables.
The answer is to use the initializationscript parameter of Start-Job. If you define all your functions in a block and pass the block they become available.
Solution was found in this post:
How do I Start a job of a function i just defined?
Here is my example from before, now working:
# concurrency
$func = {
function Logx
{
param(
[parameter(ValueFromPipeline=$true)]
$msg
)
Write-Host ("OUT:"+$msg)
}
}
# Execution starts here
cls
$colors = #("red","blue","green")
$colors | %{
$scriptBlock =
{
Logx $args[0]
Start-Sleep 9
}
Write-Host "Processing: " $_
Start-Job -InitializationScript $func -scriptblock $scriptBlock -args $_
}
Get-Job
while(Get-Job -State "Running")
{
write-host "Running..."
Start-Sleep 2
}
# Output
Get-Job | Receive-Job
# Cleanup jobs
Remove-Job *
If you do not prefix your function name with keyword function, PowerShell does not know to treat it as such. As you have written your script it is basically a variable with some special text in it. Which as your output shows it is only executing the commands it recognizes within that variable's content: Write-Host "OUT:".
Using the correct syntax will tell PowerShell it is a function and that you have variables to pass into it that you need executed:
function Logx
{
param(
[parameter(ValueFromPipeline=$true)]
$msg
)
Write-Host ("OUT:"+$msg)
}
Then when you call it within your script you will just use Logx
Got this far. Have to run out, will try back later.
PS: What is getting passed at args[1], I am getting a lot of red,
CategoryInfo : InvalidData: (:) [Invoke-Expression], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.InvokeExpressionCommand
here is what I've managed so far.
# concurrency
$Logx =
{
param(
[parameter(ValueFromPipeline=$true)]
$msg
)
Write-Host ("OUT:"+$msg)
}
# Execution starts here
cls
$colors = #("red","blue","green")
$colors | %{
& $scriptBlock =
{ Invoke-Expression -Command $args[1]
Start-Sleep 3
}
Write-Host "Processing: " $_
Start-Job -scriptblock $scriptBlock -ArgumentList #($_, $Logx)
}
# Get-Job
while(Get-Job -State "Running")
{
write-host "Running..."
Start-Sleep 2
}
# Output
Get-Job | Receive-Job
# Cleanup jobs
Remove-Job *