Passing "native" object to background jobs - powershell

Here is what I'd like to achieve in one way or another.
I have a custom assembly defining some objects. In my script, I create a custom object that I'd like to pass to a script block, keeping that object behavior.
Add-Type -AssemblyName MyCustomDLL
$global:object = new-object MyCustomDLL.MyCustomObject()
$object | gm
$jobWork = { param ($object) $object | gm } # I'd like to keep my object behavior in that block
$job = Start-Job -ScriptBlock $jobWork -ArgumentList $object
Wait-Job $job
Receive-Job $job
How can I do that or achieve the same effect? Thanks for your help

Instead of background jobs you may use PowerShell with BeginInvoke, EndInvoke. Here is the simple but working example of passing a live object in a "job", changing it there, getting the results:
# live object to be passed in a job and changed there
$liveObject = #{ data = 42}
# job script
$script = {
param($p1)
$p1.data # some output (42)
$p1.data = 3.14 # change the live object data
}
# create and start the job
$p = [PowerShell]::Create()
$null = $p.AddScript($script).AddArgument($liveObject)
$job = $p.BeginInvoke()
# wait for it to complete
$done = $job.AsyncWaitHandle.WaitOne()
# get the output, this line prints 42
$p.EndInvoke($job)
# show the changed live object (data = 3.14)
$liveObject

Background jobs are built on top of PowerShell remoting and as such, perform similar actions when passing objects around. They would serialize/ deserialize them rather than pass them with all their complexity.
My guess is that the only way to get complex object is just to pass constructor arguments and/ or operations as -ArgumentList and create object inside job.
In such a case also adding assembly would have to be part of the job:
Start-Job {
param ($ConstructorArguments)
Add-Type -AssemblyName MyCustomDll
$object = New-Object MyCustomDll.MyCustomObject $ConstructorArguments
$object | Get-Member
} -ArgumentList Foo, Bar | Wait-Job | Receive-Job

Related

Save to array/list in parallel in powershell

I want to parallelize information gathering in my PS scripts.
My scripts usually do something along the lines of
foreach ($system in $systemlist) {
$system = Add-InformationToServerObj $system
}
and thus the $systemlist gets populated with more information which later gets used.
How can such a task which requires saving output to one shared list/array be parallelized?
Start-Job is an option, but it's quite high overhead in processing time since each job kicks of a new process and data will have to be serialized between the parent process and the job process.
Use Runspaces instead:
# Create initial sessionstate object for the runspaces
$InitialSessionState = [initialsessionstate]::Create()
# Import module that contains Add-InformationToServerObj
$InitialSessionState.ImportPSModule("InformationModule")
# Create and open the runspacepool
$RunspacePool = [runspacefactory]::CreateRunspacePool($InitialSessionState)
$RunspacePool.Open()
# Create a new PowerShell instance per "job", collect these along with the IAsyncResult handle (we'll need it later)
$Jobs = foreach($system in $systemlist)
{
$PSInstance = [powershell]::Create()
[void]$PSInstance.AddCommand('Add-InformationToServerObj').AddArgument($system)
New-Object psobject -Property #{
Instance = $PSInstance
IAResult = $PSInstance.BeginInvoke()
}
}
# Wait for runspaces to complete
while($InProgress = #($Jobs |Where-Object {-not $_.IAResult.IsCompleted})){
# Here you could also use Write-Progress
Write-Host "$($InProgress.Count) jobs still in progress..."
Start-Sleep -Milliseconds 500
}
# Collect the output
$systemlist = foreach($Job in $Jobs)
{
$Job.Instance.EndInvoke($Job.IAResult)
}
# Dispose of the runspacepool
$RunspacePool.Dispose()
The above is a very basic example and has zero error handling - consider using something like Invoke-Parallel or PoshRSJobs instead (PoshRSJobs can also be found on the gallery)
use this :
foreach ($system in $systemlist) {
start-job -scriptblock{$system = Add-InformationToServerObj $system }
}

Use hashtable as parameter when splatting?

I am trying to use Start-Job to launch a new Powershell script. The new script has several parameters (some optional, some not), so I want to make a hashtable and splat them. However, one of these parameters is itself a hash table. I am trying to start the job like this:
$MyStringParam = "string1"
$MyHashParam = #{}
$MyHashParam.Add("Key1", "hashvalue1")
$arguments = #{MyStringParam=$MyStringParam;MyHashParam=$MyHashParam}
Start-Job -Name "MyJob" -ScriptBlock ([scriptblock]::create("C:\myscript.ps1 $(&{$args} #arguments)"))
Whereupon I get this error in the new job:
Cannot process argument transformation on parameter 'arguments'.
Cannot convert the "System.Collections.Hashtable" value of
type "System.String" to type "System.Collections.Hashtable".
Looks like it's treating the value I want to pass in as a hash table as a string. For the life of me I can't figure out how to get around this. Can anyone help?
You will need to pass the variable into the scriptblock as a parameter of the scriptblock, and then splat that parameter to your second script. Something like this should work for you:
Start-Job -Name "MyJob" -ScriptBlock {Param($PassedArgs);& "C:\myscript.ps1" #PassedArgs} -ArgumentList $Arguments
I created the following script and saved it to C:\Temp\TestScript.ps1
Param(
[String]$InString,
[HashTable]$InHash
)
ForEach($Key in $InHash.keys){
[pscustomobject]#{'String'=$InString;'HashKey'=$Key;'HashValue'=$InHash[$Key]}
}
I then ran the following:
$MyString = "Hello World"
$MyHash = #{}
$MyHash.Add("Green","Apple")
$MyHash.Add("Yellow","Banana")
$MyHash.Add("Purple","Grapes")
$Arguments = #{'InString'=$MyString;'InHash'=$MyHash}
$MyJob = Start-Job -scriptblock {Param($MyArgs);& "C:\Temp\testscript.ps1" #MyArgs} -Name "MyJob" -ArgumentList $Arguments | Wait-Job | Receive-Job
Remove-Job -Name 'MyJob'
$MyJob | Select * -ExcludeProperty RunspaceId | Format-Table
It produced the expected results:
String HashKey HashValue
------ ------- ---------
Hello World Yellow Banana
Hello World Green Apple
Hello World Purple Grapes
The process of running the job will add a RunspaceId property to any objects returned, that is why I had to exclude that.
Rather than
[scriptblock]::create("C:\myscript.ps1 $(&{$args} #arguments)")
does this work?
[scriptblock]::create("C:\myscript.ps1 $(&{$args}) #arguments")
I.e. move the splat outside of the $()

How to run a command against multiple servers simultaneously in Powershell

I am looking for a way to restart three services on multiple servers simultaneously. I know how to restart services against a list of servers by using a loop but as I have many servers it would take a long time to wait for each service on each server to restart in a sequential order. Is there a way to send restart service command to all servers at once instead of waiting for each server?
You could try to work with jobs. Jobs are run in the background and you have to retrieve them with Get-Job to see their status. Please read the information to Powershell jobs on these two sites:
http://msdn.microsoft.com/en-us/library/dd878288%28v=vs.85%29.aspx
http://technet.microsoft.com/de-DE/library/hh847783.aspx
Your code would look something like this:
$servernames | ForEach-Object {Start-Job -Name "Job-$_" -Scriptblock {"Enter your code here -Computername $_"}}
This will create a background job for each servername. As already mentioned you can see the status using the cmdlet Get-Job. To get the result use the cmdlet Receive-Job.
you can use the invoke-command cmdlet
invoke-command -computername computer1,computer2,computer3 {restart-service servicename}
I use and improove a multi-thread Function, you can use it like :
$Script = {
param($Computername)
restart-service servicename -Computername $Computername
}
#('Srv1','Srv2') | Run-Parallel -ScriptBlock $Script
include this code in your script
function Run-Parallel {
<#
.Synopsis
This is a quick and open-ended script multi-threader searcher
http://www.get-blog.com/?p=189#comment-28834
Improove by Alban LOPEZ 2016
.Description
This script will allow any general, external script to be multithreaded by providing a single
argument to that script and opening it in a seperate thread. It works as a filter in the
pipeline, or as a standalone script. It will read the argument either from the pipeline
or from a filename provided. It will send the results of the child script down the pipeline,
so it is best to use a script that returns some sort of object.
.PARAMETER ScriptBlock
This is where you provide the PowerShell ScriptBlock that you want to multithread.
.PARAMETER ItemObj
The ItemObj represents the arguments that are provided to the child script. This is an open ended
argument and can take a single object from the pipeline, an array, a collection, or a file name. The
multithreading script does it's best to find out which you have provided and handle it as such.
If you would like to provide a file, then the file is read with one object on each line and will
be provided as is to the script you are running as a string. If this is not desired, then use an array.
.PARAMETER InputParam
This allows you to specify the parameter for which your input objects are to be evaluated. As an example,
if you were to provide a computer name to the Get-Process cmdlet as just an argument, it would attempt to
find all processes where the name was the provided computername and fail. You need to specify that the
parameter that you are providing is the "ComputerName".
.PARAMETER AddParam
This allows you to specify additional parameters to the running command. For instance, if you are trying
to find the status of the "BITS" service on all servers in your list, you will need to specify the "Name"
parameter. This command takes a hash pair formatted as follows:
#{"key" = "Value"}
#{"key1" = "Value"; "key2" = 321; "key3" = 1..9}
.PARAMETER AddSwitch
This allows you to add additional switches to the command you are running. For instance, you may want
to include "RequiredServices" to the "Get-Service" cmdlet. This parameter will take a single string, or
an aray of strings as follows:
"RequiredServices"
#("RequiredServices", "DependentServices")
.PARAMETER MaxThreads
This is the maximum number of threads to run at any given time. If ressources are too congested try lowering
this number. The default value is 20.
.PARAMETER SleepTimer_ms
This is the time between cycles of the child process detection cycle. The default value is 200ms. If CPU
utilization is high then you can consider increasing this delay. If the child script takes a long time to
run, then you might increase this value to around 1000 (or 1 second in the detection cycle).
.PARAMETER TimeOutGlobal
this is the TimeOut in second for listen the last thread, after this timeOut All thread are closed, only each other are returned
.PARAMETER TimeOutThread
this is the TimeOut in second for each thread, the thread are aborted at this time
.PARAMETER PSModules
List of PSModule name to include for use in ScriptBlock
.PARAMETER PSSapins
List of PSSapin name to include for use in ScriptBlock
.EXAMPLE
1..20 | Run-Parallel -ScriptBlock {param($i) Start-Sleep $i; "> $i sec <"} -TimeOutGlobal 15 -TimeOutThread 5
.EXAMPLE
Both of these will execute the scriptBlock and provide each of the server names in AllServers.txt
while providing the results to GridView. The results will be the output of the child script.
gc AllServers.txt | Run-Parallel $ScriptBlock_GetTSUsers -MaxThreads $findOut_AD.ActiveDirectory.Servers.count -PSModules 'PSTerminalServices' | out-gridview
#>
Param(
[Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]
$ItemObj,
[ScriptBlock]$ScriptBlock = $null,
$InputParam = $Null,
[HashTable] $AddParam = #{},
[Array] $AddSwitch = #(),
$MaxThreads = 20,
$SleepTimer_ms = 100,
$TimeOutGlobal = 300,
$TimeOutThread = 100,
[string[]]$PSSapins = $null,
[string[]]$PSModules = $null,
$Modedebug = $true
)
Begin{
$ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
ForEach ($Snapin in $PSSapins){
[void]$ISS.ImportPSSnapIn($Snapin, [ref]$null)
}
ForEach ($Module in $PSModules){
[void]$ISS.ImportPSModule($Module)
}
$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)
$RunspacePool.CleanupInterval=1000
$RunspacePool.Open()
$Jobs = #()
}
Process{
#ForEach ($Object in $ItemObj){
if ($ItemObj){
Write-Host $ItemObj -ForegroundColor Yellow
$PowershellThread = [powershell]::Create().AddScript($ScriptBlock)
If ($InputParam -ne $Null){
$PowershellThread.AddParameter($InputParam, $ItemObj.ToString()) | out-null
}Else{
$PowershellThread.AddArgument($ItemObj.ToString()) | out-null
}
ForEach($Key in $AddParam.Keys){
$PowershellThread.AddParameter($Key, $AddParam.$key) | out-null
}
ForEach($Switch in $AddSwitch){
$PowershellThread.AddParameter($Switch) | out-null
}
$PowershellThread.RunspacePool = $RunspacePool
$Handle = $PowershellThread.BeginInvoke()
$Job = [pscustomobject][ordered]#{
Handle = $Handle
Thread = $PowershellThread
object = $ItemObj.ToString()
Started = Get-Date
}
$Jobs += $Job
}
#}
}
End{
$GlobalStartTime = Get-Date
$continue = $true
While (#($Jobs | Where-Object {$_.Handle -ne $Null}).count -gt 0 -and $continue) {
ForEach ($Job in $($Jobs | Where-Object {$_.Handle.IsCompleted -eq $True})){
$out = $Job.Thread.EndInvoke($Job.Handle)
$out # return vers la sortie srandard
#Write-Host $out -ForegroundColor green
$Job.Thread.Dispose() | Out-Null
$Job.Thread = $Null
$Job.Handle = $Null
}
foreach ($InProgress in $($Jobs | Where-Object {$_.Handle})) {
if ($TimeOutGlobal -and (($(Get-Date) - $GlobalStartTime).totalseconds -gt $TimeOutGlobal)){
$Continue = $false
#Write-Host $InProgress -ForegroundColor magenta
}
if (!$Continue -or ($TimeOutThread -and (($(Get-Date) - $InProgress.Started).totalseconds -gt $TimeOutThread))) {
$InProgress.thread.Stop() | Out-Null
$InProgress.thread.Dispose() | Out-Null
$InProgress.Thread = $Null
$InProgress.Handle = $Null
#Write-Host $InProgress -ForegroundColor red
}
}
Start-Sleep -Milliseconds $SleepTimer_ms
}
$RunspacePool.Close() | Out-Null
$RunspacePool.Dispose() | Out-Null
}
}

Powershell command timeout

I am trying to execute a function or a scriptblock in powershell and set a timeout for the execution.
Basically I have the following (translated into pseudocode):
function query{
#query remote system for something
}
$computerList = Get-Content "C:\scripts\computers.txt"
foreach ($computer in $computerList){
$result = query
#do something with $result
}
The query can range from a WMI query using Get-WmiObject to a HTTP request and the script has to run in a mixed environment, which includes Windows and Unix machines which do not all have a HTTP interface.
Some of the queries will therefore necessarily hang or take a VERY long time to return.
In my quest for optimization I have written the following:
$blockofcode = {
#query remote system for something
}
foreach ($computer in $computerList){
$Job = Start-Job -ScriptBlock $blockofcode -ArgumentList $computer
Wait-Job $Job.ID -Timeout 10 | out-null
$result = Receive-Job $Job.ID
#do something with result
}
But unfortunately jobs seem to carry a LOT of overhead. In my tests a query that executes in 1.066 seconds (according to timers inside $blockofcode) took 6.964 seconds to return a result when executed as a Job. Of course it works, but I would really like to reduce that overhead. I could also start all jobs together and then wait for them to finish, but the jobs can still hang or take ridiculous amounts to time to complete.
So, on to the question: is there any way to execute a statement, function, scriptblock or even a script with a timeout that does not comprise the kind of overhead that comes with jobs? If possible I would like to run the commands in parallel, but that is not a deal-breaker.
Any help or hints would be greatly appreciated!
EDIT: running powershell V3 in a mixed windows/unix environment
Today, I ran across a similar question, and noticed that there wasn't an actual answer to this question. I created a simple PowerShell class, called TimedScript. This class provides the following functionality:
Method: Start() method to kick off the job, when you're ready
Method:GetResult() method, to retrieve the output of the script
Constructor: A constructor that takes two parameters:
ScriptBlock to execute
[int] timeout period, in milliseconds
It currently lacks:
Passing in arguments to the PowerShell ScriptBlock
Other useful features you think up
Class: TimedScript
class TimedScript {
[System.Timers.Timer] $Timer = [System.Timers.Timer]::new()
[powershell] $PowerShell
[runspace] $Runspace = [runspacefactory]::CreateRunspace()
[System.IAsyncResult] $IAsyncResult
TimedScript([ScriptBlock] $ScriptBlock, [int] $Timeout) {
$this.PowerShell = [powershell]::Create()
$this.PowerShell.AddScript($ScriptBlock)
$this.PowerShell.Runspace = $this.Runspace
$this.Timer.Interval = $Timeout
Register-ObjectEvent -InputObject $this.Timer -EventName Elapsed -MessageData $this -Action ({
$Job = $event.MessageData
$Job.PowerShell.Stop()
$Job.Runspace.Close()
$Job.Timer.Enabled = $False
})
}
### Method: Call this when you want to start the job.
[void] Start() {
$this.Runspace.Open()
$this.Timer.Start()
$this.IAsyncResult = $this.PowerShell.BeginInvoke()
}
### Method: Once the job has finished, call this to get the results
[object[]] GetResult() {
return $this.PowerShell.EndInvoke($this.IAsyncResult)
}
}
Example Usage of TimedScript Class
# EXAMPLE: The timeout period is set longer than the execution time of the script, so this will succeed
$Job1 = [TimedScript]::new({ Start-Sleep -Seconds 2 }, 4000)
# EXAMPLE: This script will fail. Even though Get-Process returns quickly, the Start-Sleep call will cause it to be terminated by its Timer.
$Job2 = [TimedScript]::new({ Get-Process -Name s*; Start-Sleep -Seconds 3 }, 2000)
# EXAMPLE: This job will fail, because the timeout is less than the script execution time.
$Job3 = [TimedScript]::new({ Start-Sleep -Seconds 3 }, 1000)
$Job1.Start()
$Job2.Start()
$Job3.Start()
Code is also hosted on GitHub Gist.
I think you might want to investigate using Powershell runspaces:
http://learn-powershell.net/2012/05/13/using-background-runspaces-instead-of-psjobs-for-better-performance/

Powershell module initialization

Does PowerShell call any initialization code when a module is loaded?
I am looking for something like a Perl BEGIN block, or a constructor.
Both NEW-MODULE and IMPORT-MODULE will return a PSCustomObject. I am trying to encapsulate a custom object in a module to avoid lengthy code in scripts. One method that tests well in open code is:
$m = new-module -scriptblock {
New-Object PSCustomObject |
Add-Member NoteProperty -name person -value Frodo -passthru |
Add-Member ScriptMethod Who { $this.person } -passthru |
Add-Member ScriptMethod Mod {
param($x)
$this.person = $x
} -passthru
} -ascustomobject -returnresult
Ideally I would like to drop this code into a module and use something like:
$MyObj = Import-Module -Name ".\MyPackage" -AsCustomObject
and have MyObj be a handle to an object the same as the first snippet provides.
Suggestions appreciated.
It's not clear if you want to run initialization code when a module is loaded (like Perl's BEGIN block) or if you want to create a custom class (which is what "constructor" suggests).
Initialization code in a module is easy. Any code in a module not embedded in a function is executed when the module is imported.
Creating a custom class isn't supported natively in PS. But see: http://psclass.codeplex.com/. You can also write C#, VBScript, etc. and use Add-Type.
Import-module won't work to simulate a class, because you can only have 1 instance of a module with a given name - at best you'd have a singleton class. (BTW, import-module does have a -passthru parameter, which would more or less make your last line of code work - as a singleton. You'd also have to add export-module -variable * -function * to your module code) You could use New-Module to simulate a class though. And you could wrap it in a function named, new-myClass for example.
BTW, if you use the -ASCustomObject parameter you end up with a hashtable, which doesn't support "this" (in words hash table values that are script blocks don't have a built-in way to refer to the hashtable itself). If you use new-module without -AsCustomObject (and potentially use a factory function, for example new-myclass) then you could simulate "this.varInMyModule" with & $myModule $varInMyModule. However if you create a PSCustomObject, using Add-Member, then script method have access to $this and it in general acts a lot more like a typical object with properties and methods.
Modules are really supposed to output cmdlets, not objects. A module should provide a set of related cmdlets. There is a way to send data into the module using Import-Modules's -ArgumentList parameter as show here. You could use the technique to provide a server name for your cmdlets to connect to for example. The PowerCLI module handles that differently using a cmdlet that creates a script scope connection object ($script:connection) that the other cmdlets check for and re-use if it exists similar to this:
#test.psm1
$script:myvar = "hi"
function Show-MyVar {Write-Host $script:myvar}
function Set-MyVar ($Value) {$script:myvar = $Value}
#end test.psm1
Using Modules you can export both innate properties and functions, and don't need to run them through add-member or do much acrobatics. Note however that it has some issues if you don't want to export all properties and methods, and while you can initialize properties to an initial value, you CAN'T call an internal function during initialization without doing some akward acrobatics.
But I think what you really want to do is use Classes which are now available in Powershell 5 (they weren't when you posted). I've provided examples of each.
Sysops has a decent tutorial on the new classes in 4 parts
Here's the older way before powershell 5.0
# powershell 3.0 and above (I think)
$m = new-module -ascustomobject -scriptblock `
{
$person = "Frodo"
function Who
()
{return $this.person}
function Rename
($name)
{$this.person = $name}
Export-ModuleMember -Function * -Variable *
}
write-host "direct property access: $($m.person)"
write-host "method call: $($m.who())"
$m.Rename("Shelob")
write-host "change test: $($m.who())"
Also you can replicate multiple objects from a template like this:
# powershell 3.0 and above (I think)
$template = `
{
$person = "Frodo"
function Who
()
{return $this.person}
function Rename
($name)
{$this.person = $name}
Export-ModuleMember -Function * -Variable *
}
$me = new-module -ascustomobject -scriptblock $template; $me.Rename("Shelob")
$you = new-module -ascustomobject -scriptblock $template
"Me: $($me.Who())"
"You: $($you.Who())"
And in powershell 5 you have actual classes (mostly)
#requires -version 5
Class Person
{
hidden [String] $name #not actually private
[string] Who ()
{return $this.name}
[void] Rename ([string] $name)
{$this.name = $name}
# constructors are weird though, you don't specify return type OR explicitly return value.
Person ([String]$name)
{$this.name = $name}
<#
# The above constructor code is secretly converted to this
[Person] New ([string]$name) #note the added return type and renamed to New
{
$this.name = $name
return $this #note that we are returning ourself, you can exploit this to create chained constructors like [person]::New("gandalf").withWizardLevel("White") but I haven't done so here
}
#>
}
$me = [Person]::new("Shelob")
$you = [Person]::new("Frodo")
# $me|gm # Note that Name doesn't show here
# $me.name # But we can still access it...
# $me|gm -Force # Note that Name DOES show here
"`n"
"Me: $($me.who())"
"You: $($you.who())"
$you.Rename("Dinner")
"You after we meet: $($you.who())"