multithread with runspaces instead of foreach cycle - powershell

I have a script with a foreach cycle. It has about a dozen functions, each collecting information from remote machines' C$ share (cutting text files, checking file version, etc.)
This is however taking some time, since each machine's data collected after one by one. (sometimes it runs with 500+ input)
Wish to put this into runspaces with parallel execution, but so far no examples worked. I am quite new to the concept.
Current script's outline
$inputfile = c:\temp\computerlist.txt
function 1
function 2
function 3, etc
foreach cycle
function 1
function 2
function 3
All results written to screen with write-host for now.

This example pings a number of server in parallel, so you easily can modify it for your demands:
Add-Type -AssemblyName System.Collections
$GH = [hashtable]::Synchronized(#{})
[System.Collections.Generic.List[PSObject]]$GH.results = #()
[System.Collections.Generic.List[string]]$GH.servers = #('server1','server2','server3');
[System.Collections.Generic.List[string]]$GH.functions = #('Check-Server');
[System.Collections.Generic.List[PSObject]]$jobs = #()
#-----------------------------------------------------------------
function Check-Server {
#-----------------------------------------------------------------
# a function which runs parallel
param(
[string]$server
)
$result = Test-Connection $server -Count 1 -Quiet
$GH.results.Add( [PSObject]#{ 'Server' = $server; 'Result' = $result } )
}
#-----------------------------------------------------------------
function Create-InitialSessionState {
#-----------------------------------------------------------------
param(
[System.Collections.Generic.List[string]]$functionNameList
)
# Setting up an initial session state object
$initialSessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
foreach( $functionName in $functionNameList ) {
# Getting the function definition for the functions to add
$functionDefinition = Get-Content function:\$functionName
$functionEntry = New-Object System.Management.Automation.Runspaces.SessionStateFunctionEntry -ArgumentList $functionName, $functionDefinition
# And add it to the iss object
[void]$initialSessionState.Commands.Add($functionEntry)
}
return $initialSessionState
}
#-----------------------------------------------------------------
function Create-RunspacePool {
#-----------------------------------------------------------------
param(
[InitialSessionState]$initialSessionState
)
$runspacePool = [RunspaceFactory]::CreateRunspacePool(1, ([int]$env:NUMBER_OF_PROCESSORS + 1), $initialSessionState, $Host)
$runspacePool.ApartmentState = 'MTA'
$runspacePool.ThreadOptions = "ReuseThread"
[void]$runspacePool.Open()
return $runspacePool
}
#-----------------------------------------------------------------
function Release-Runspaces {
#-----------------------------------------------------------------
$runspaces = Get-Runspace | Where { $_.Id -gt 1 }
foreach( $runspace in $runspaces ) {
try{
[void]$runspace.Close()
[void]$runspace.Dispose()
}
catch {
}
}
}
$initialSessionState = Create-InitialSessionState -functionNameList $GH.functions
$runspacePool = Create-RunspacePool -initialSessionState $initialSessionState
foreach ($server in $GH.servers)
{
Write-Host $server
$job = [System.Management.Automation.PowerShell]::Create($initialSessionState)
$job.RunspacePool = $runspacePool
$scriptBlock = { param ( [hashtable]$GH, [string]$server ); Check-Server -server $server }
[void]$job.AddScript( $scriptBlock ).AddArgument( $GH ).AddArgument( $server )
$jobs += New-Object PSObject -Property #{
RunNum = $jobCounter++
JobObj = $job
Result = $job.BeginInvoke() }
do {
Sleep -Seconds 1
} while( $runspacePool.GetAvailableRunspaces() -lt 1 )
}
Do {
Sleep -Seconds 1
} While( $jobs.Result.IsCompleted -contains $false)
$GH.results
Release-Runspaces | Out-Null
[void]$runspacePool.Close()
[void]$runspacePool.Dispose()

This would be concurrent and run in about 10 seconds total. The computer could be localhost three times if you got it working.
invoke-command comp1,comp2,comp3 { sleep 10; 'done' }
Simple attempt at api (threads):
$a = [PowerShell]::Create().AddScript{sleep 5;'a done'}
$b = [PowerShell]::Create().AddScript{sleep 5;'b done'}
$c = [PowerShell]::Create().AddScript{sleep 5;'c done'}
$r1 = $a.BeginInvoke(); $r2 = $b.BeginInvoke() $r3 = $c.BeginInvoke()
$a.EndInvoke($r1); $b.EndInvoke($r2); $c.EndInvoke($r3)
a done
b done
c done

Related

How to get current runspace of RunspacePool

I have multiple commands run in parallel in separate runspaces, managed by a RunspacePool.
How can I determine the runspace each command is running in?
param (
$MaxRunspaces = 4,
$JobCount = 20
)
try {
$pool = [RunspaceFactory]::CreateRunspacePool(1, $MaxRunspaces)
$pool.Open()
$jobs = 1..$JobCount | foreach {
$ps = [PowerShell]::Create()
$ps.RunspacePool = $pool
[void]$ps.AddScript({
# get current runspace here ???
# $runspace = ...
})
[PSCustomObject]#{
PowerShell = $ps
AsyncResult = $ps.BeginInvoke()
}
}
$jobs | foreach {
$_.PowerShell.EndInvoke($_.AsyncResult)
$_.PowerShell.Dispose()
}
}
finally {
$pool.Dispose()
}

Coding with powershell

I'm new with powershell and i would like to use a loop to ping several Printers on my network.
My problem is : once i'm in the loop of pinging , i can't go out of the loop ...
I tried several things from google but without success ( start-stop , Timer ) . Does anybody have any idea?
Here is the code :
$BtnStartPingClicked = {
if ($LblFileSelectPing.Text -eq "*.txt") {
Add-Type -AssemblyName PresentationCore,PresentationFramework
$ButtonType = [System.Windows.MessageBoxButton]::OK
$MessageIcon = [System.Windows.MessageBoxImage]::Error
$MessageBody = "Please select a list of printer first"
$MessageTitle = "Error"
$Result = [System.Windows.MessageBox]::Show($MessageBody,$MessageTitle,$ButtonType,$MessageIcon)
Write-Host "Your choice is $Result"
}
else {
do {
$IPList = Get-Content ($LblFileSelectPing.Text)
$snmp = New-Object -ComObject olePrn.OleSNMP
$ping = New-Object System.Net.NetworkInformation.Ping
$i = 11
$j = 1
foreach ($Printer in $IPList) {
try {
$result = $ping.Send($Printer)
} catch {
$result = $null
}
if ($result.Status -eq 'Success') {
$((Get-Variable -name ("GBMachine"+$j+"Ping")).value).Visible = $True
$j++
test-Connection -ComputerName $Printer -Count 1 -Quiet
$printerip = $result.Address.ToString()
# OPEN SNMP CONNECTION TO PRINTER
$snmp.open($Printer, 'public', 2, 3000)
# MODEL
try {
$model = $snmp.Get('.1.3.6.1.2.1.25.3.2.1.3.1')
} catch {
$model = $null
}
# Serial
try {
$serial = $snmp.Get('.1.3.6.1.4.1.1602.1.2.1.8.1.3.1.1').toupper()
} catch {
$Dns = $null
}
# progress
$TBMonitoringPing.SelectionColor = "green"
$TBMonitoringPing.AppendText("$Printer is Pinging")
$TBMonitoringPing.AppendText("`n")
$mac = (arp -a $Printer | Select-String '([0-9a-f]{2}-){5}[0-9a-f]{2}').Matches.Value
# OPEN SNMP CONNECTION TO PRINTER
$((Get-Variable -name ('LblMach' + $i)).value).Text = "IP : $Printerip"
$i++
$((Get-Variable -name ('LblMach' + $i)).value).Text = "Model : $Model"
$i++
$((Get-Variable -name ('LblMach' + $i)).value).Text = "MAC : $mac"
$i++
$((Get-Variable -name ('LblMach' + $i)).value).Text = "Serial : $serial"
$TBAnswerMachine.AppendText("$Model")
$TBAnswerMachine.AppendText("`n")
$TBAnswerMachine.AppendText("$Printer - $Serial")
$TBAnswerMachine.AppendText("`n")
$TBAnswerMachine.AppendText("$Mac")
$TBAnswerMachine.AppendText("`n")
$TBAnswerMachine.AppendText("`n")
Get-Content ($LblFileSelectPing.Text) | Where-Object {$_ -notmatch $Printer} | Set-Content ("C:\_canonsoftware\out.txt")
$i = $i+7
$snmp.Close()
Start-Sleep -milliseconds 1000 # Take a breather!
}
else {
$TBMonitoringPing.selectioncolor = "red"
$TBMonitoringPing.AppendText("$Printer not pinging")
$TBMonitoringPing.AppendText("`n")
Start-Sleep -milliseconds 1000 # Take a breather!
}
}
$LblFileSelectPing.Text = "C:\_canonsoftware\out.txt"
} until($infinity)
}
}
thanks for your answers...
1 - part of my object are indeed not declared in the code because they are in my other PS1 file....
2 - I do a do until infinity because i don't want to stop the code before i decide it...
3 - I didn't explain my problem correctly ( excuse my poor english ) ... i would like to be able to go out of the loop do until at the moment i click on a stop button ... but apprently the windows doens't respond while in the loop ... i have to stop the script with powershell ... which is annoying because i'd like to make an executable with it ... and not have to go out of my program ...
thank you for your ideas

ping computers and list logged in user

i got a question to my existing script.
the script does exactly what it should to do
This is what i got.
<#
Ping mass huge amount of hosts in parallel multithreading
#>
function Parallel-Ping {
[CmdletBinding()]
param(
[parameter(mandatory=$true)][ValidateNotNullOrEmpty()][string[]]$hosts,
[parameter(mandatory=$false)][ValidateNotNullOrEmpty()][int]$timeout = 2000,
[parameter(mandatory=$false)][ValidateNotNullOrEmpty()][ValidateRange(1,255)][int]$ttl = 128,
[parameter(mandatory=$false)][ValidateNotNullOrEmpty()][bool]$DontFragment = $false,
[parameter(mandatory=$false)][ValidateNotNullOrEmpty()][byte[]]$data = ([byte[]]0),
[parameter(mandatory=$false)][ValidateNotNullOrEmpty()][int]$MaxConcurrent = [System.Environment]::ProcessorCount,
[parameter(mandatory=$false)][ValidateNotNullOrEmpty()][switch]$showprogress
)
begin{
$rspool = [runspacefactory]::CreateRunspacePool(1,$MaxConcurrent)
$rspool.ApartmentState = 'STA'
$rspool.Open()
$jobs = New-Object System.Collections.ArrayList
}
process{
foreach ($hostname in $hosts){
$ps = [Powershell]::Create()
$ps.RunspacePool = $rspool
[void]$ps.AddScript({
param($hostname,$timeout,$ttl,$DontFragment,$data)
$result = [ordered]#{Host=$hostname;Address=$null;Status=[System.Net.NetworkInformation.IPStatus]::Unknown;RoundtripTime = -1;TTL=$ttl;DontFragment=$DontFragment;ResponseBuffer='';Time=(get-date)}
$ping = New-Object System.Net.NetworkInformation.Ping
try{
$reply = $ping.Send($hostname,$timeout,$data,(New-Object System.Net.NetworkInformation.PingOptions($ttl,$DontFragment)))
$result.RoundtripTime = $reply.RoundtripTime
$result.Status = $reply.Status
$result.Address = $reply.Address
$result.ResponseBuffer = $reply.Buffer
}catch [System.Net.NetworkInformation.PingException] {
$result.Status = [System.Net.NetworkInformation.IPStatus]::DestinationHostUnreachable
}finally{
$ping.Dispose()
}
[pscustomobject]$result
}).AddParameters(#($hostname,$timeout,$ttl,$DontFragment,$data))
$job = $ps.BeginInvoke()
[void]$jobs.Add(([pscustomobject]#{Handle = $job; Powershell = $ps}))
}
}
end{
write-verbose "Waiting for all jobs to complete."
while(($jobs | ?{!$_.Handle.IsCompleted})){
if ($showprogress.IsPresent){
$completed = ($jobs | ?{$_.Handle.IsCompleted}).Count
Write-Progress -Activity $PSCmdlet.MyInvocation.InvocationName -Status "Pinging a total of $($hosts.Count) hosts." -PercentComplete (($completed / $hosts.Count) * 100) -CurrentOperation "Completed $completed from $($hosts.Count)."
}
}
# get results of jobs
$results = $jobs | %{
$_.Powershell.EndInvoke($_.handle)
$_.Powershell.Dispose()
}
# cleanup
$rspool.Close();$rspool.Dispose()
$results
}
}
$hosts = Get-Content 'E:\ips.txt'
Parallel-Ping -hosts $hosts -timeout 500 -MaxConcurrent 50 -showprogress | ogv
I would like to mention another function in my script.
My goal is it, to add the function that i can see the actual logged in user to the pinged machines?
Unfortunately i dont know where to add.

How to download a webstring from multiple web-servers in parallel via Powershell?

I need to download some webcontent from many servers in parallel as part of a scheduled job, but I cannot find a correct way to run the download in parallel/async. How can this be done?
Without any parallelism I can do it this way, but it is very slow:
$web = [System.Net.WebClient]::new()
[Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
# $srvList is a list of servers of viariable length
$allData = ""
foreach ($srv in $srvList) {
$url = "https:\\$srv\MyWebPage"
$data = $web.DownloadString($url)
$allData += $data
}
But how to do this in parallel via "$web.DownloadStringAsync"?
I found this snippet, but I dont see how to get the result of each call and how to concatenate it:
$job = Register-ObjectEvent -InputObject $web -EventName DownloadStringCompleted -Action {
Write-Host 'Download completed'
write-host $EventArgs.Result
}
$web.DownloadString($url)
Does someone know, how to get this solved in a short & smart way?
The best and fastest way is using runspaces:
Add-Type -AssemblyName System.Collections
$GH = [hashtable]::Synchronized(#{})
[System.Collections.Generic.List[PSObject]]$GH.results = [System.Collections.Generic.List[string]]::new()
[System.Collections.Generic.List[string]]$GH.servers = #('server1','server2');
[System.Collections.Generic.List[string]]$GH.functions = #('Download-Content');
[System.Collections.Generic.List[PSObject]]$jobs = #()
#-----------------------------------------------------------------
function Download-Content {
#-----------------------------------------------------------------
# a function which runs parallel
param(
[string]$server
)
$web = [System.Net.WebClient]::new()
[Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$url = "https:\\$server\MyWebPage"
$data = $web.DownloadString($url)
$GH.results.Add( $data )
}
#-----------------------------------------------------------------
function Create-InitialSessionState {
#-----------------------------------------------------------------
param(
[System.Collections.Generic.List[string]]$functionNameList
)
# Setting up an initial session state object
$initialSessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
foreach( $functionName in $functionNameList ) {
# Getting the function definition for the functions to add
$functionDefinition = Get-Content function:\$functionName
$functionEntry = New-Object System.Management.Automation.Runspaces.SessionStateFunctionEntry -ArgumentList $functionName, $functionDefinition
# And add it to the iss object
[void]$initialSessionState.Commands.Add($functionEntry)
}
return $initialSessionState
}
#-----------------------------------------------------------------
function Create-RunspacePool {
#-----------------------------------------------------------------
param(
[InitialSessionState]$initialSessionState
)
$runspacePool = [RunspaceFactory]::CreateRunspacePool(1, ([int]$env:NUMBER_OF_PROCESSORS + 1), $initialSessionState, $Host)
$runspacePool.ApartmentState = 'MTA'
$runspacePool.ThreadOptions = "ReuseThread"
[void]$runspacePool.Open()
return $runspacePool
}
#-----------------------------------------------------------------
function Release-Runspaces {
#-----------------------------------------------------------------
$runspaces = Get-Runspace | Where { $_.Id -gt 1 }
foreach( $runspace in $runspaces ) {
try{
[void]$runspace.Close()
[void]$runspace.Dispose()
}
catch {
}
}
}
$initialSessionState = Create-InitialSessionState -functionNameList $GH.functions
$runspacePool = Create-RunspacePool -initialSessionState $initialSessionState
foreach ($server in $GH.servers)
{
Write-Host $server
$job = [System.Management.Automation.PowerShell]::Create($initialSessionState)
$job.RunspacePool = $runspacePool
$scriptBlock = { param ( [hashtable]$GH, [string]$server ); Download-Content -server $server }
[void]$job.AddScript( $scriptBlock ).AddArgument( $GH ).AddArgument( $server )
$jobs += New-Object PSObject -Property #{
RunNum = $jobCounter++
JobObj = $job
Result = $job.BeginInvoke() }
do {
Sleep -Seconds 1
} while( $runspacePool.GetAvailableRunspaces() -lt 1 )
}
Do {
Sleep -Seconds 1
} While( $jobs.Result.IsCompleted -contains $false)
$GH.results
Release-Runspaces | Out-Null
[void]$runspacePool.Close()
[void]$runspacePool.Dispose()
Finally I found a simple solution via events. Here is my code-snippet:
cls
Remove-Variable * -ea 0
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = $null
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$srvList = #('srv1','srv2','srv3')
$webObjList = [System.Collections.ArrayList]::new()
$eventList = [System.Collections.ArrayList]::new()
$resultList = [System.Collections.ArrayList]::new()
$i=0
foreach ($srv in $srvList) {
$null = $webObjList.add([System.Net.WebClient]::new())
$null = $eventList.add($(Register-ObjectEvent -InputObject $webObjList[$i] -EventName DownloadStringCompleted -SourceIdentifier $srv))
$null = $resultList.add($webObjList[$i].DownloadStringTaskAsync("https://$srv/MyWebPage"))
$i++
}
do {sleep -Milliseconds 10} until ($resultList.IsCompleted -notcontains $false)
foreach ($srv in $srvList) {Unregister-Event $srv}
# show all Results:
$resultList.result

Powershell: Runspace/Pool how do i pass a function and shared variable?

I am trying to create a thread that will use a shared variable (between main session and the thread)
plus the give the thread ability to use outside function from the main code
I have managed to pass the function for the thread to use
and i have managed to pass read only variable.
my problem is that if i am changing the value of the variable inside the thread and then i try to read it from the main session - i can not see the change in the value, therefore its not shared.
How do i do that?
My goal is to have one thread in the end.
this is my code:
$x = [Hashtable]::Synchronized(#{})
$global:yo = "START"
Function ConvertTo-Hex {
#Write-Output "Function Ran"
write-host "hi"
$x.host.ui.WriteVerboseLine("===========")
write-host $yo
$yo="TEST"
write-host $yo
}
#endregion
$rs = [RunspaceFactory]::CreateRunspace()
$rs.ApartmentState,$rs.ThreadOptions = "STA","ReUseThread"
$rs.Open()
$rs.SessionStateProxy.SetVariable("x",$x)
ls
# create an array and add it to session state
$arrayList = New-Object System.Collections.ArrayList
$arrayList.AddRange(('a','b','c','d','e'))
$x.host = $host
$sessionstate = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
$sessionstate.Variables.Add((New-Object System.Management.Automation.Runspaces.SessionStateVariableEntry('arrayList', $arrayList, $null)))
$sessionstate.Variables.Add((New-Object System.Management.Automation.Runspaces.SessionStateVariableEntry('x', $x, $null)))
#$sessionstate.Variables.Add((New-Object System.Management.Automation.Runspaces.SessionStateVariableEntry('yo', $yo, $true)))
$sessionstate.Commands.Add((New-Object System.Management.Automation.Runspaces.SessionStateFunctionEntry -ArgumentList 'ConvertTo-Hex', (Get-Content Function:\ConvertTo-Hex -ErrorAction Stop)))
$runspacepool = [runspacefactory]::CreateRunspacePool(1, 2, $sessionstate, $Host)
$runspacepool.Open()
$ps1 = [powershell]::Create()
$ps1.RunspacePool = $runspacepool
$ps1.AddScript({
for ($i = 1; $i -le 15; $i++)
{
$letter = Get-Random -InputObject (97..122) | % {[char]$_} # a random lowercase letter
$null = $arrayList.Add($letter)
start-sleep -s 1
}
})
$ps1.AddParameter(#($yo))
# on the first thread start a process that adds values to $arrayList every second
$handle1 = $ps1.BeginInvoke()
# now on the second thread, output the value of $arrayList every 1.5 seconds
$ps2 = [powershell]::Create()
$ps2.RunspacePool = $runspacepool
$ps2.AddScript({
Write-Host "ArrayList contents is "
foreach ($i in $arrayList)
{
Write-Host $i -NoNewline
Write-Host " " -NoNewline
}
Write-Host ""
$yo = "BAH"
ConvertTo-Hex
})
$ps2.AddParameter(#($yo))
1..10 | % {
$handle2 = $ps2.BeginInvoke()
if ($handle2.AsyncWaitHandle.WaitOne())
{
$ps2.EndInvoke($handle2)
}
start-sleep -s 1.5
write-host "====================" + $yo
}