Related
Abstract
So I work for a company that has roughly 10k computer assets on my domain. My issue is the time it takes to query if a user exists on a computer to see if they've ever logged into said computer. We need this functionality for audits in case they've done something they shouldn't have.
I have two methods in mind I've researched to complete this task, and a third alternative solution I have not thought of;
-Method A: Querying every computer for the "C:\Users<USER>" to see if LocalPath exists
-Method B: Checking every computer registry for the "HKU:<SID>" to see if the SID exists
-Method C: You are all smarter than me and have a better way? XD
Method A Function
$AllCompFound = #()
$AllADComputer = Get-ADComputer -Properties Name -SearchBase "WhatsItToYa" -filter 'Name -like "*"' | Select-Object Name
ForEach($Computer in $AllADComputers) {
$CName = $Computer.Name
if (Get-CimInstance -ComputerName "$CName" -ClassName Win32_Profile | ? {"C:\Users\'$EDIPI'" -contains $_.LocalPath}) {
$AllCompFound += $CName
} else {
#DOOTHERSTUFF
}
}
NOTE: I have another function that prompts me to enter a username to check for. Where I work they are numbers so case sensitivity is not an issue. My issue with this function is I believe it is the 'if' statement returns true every time because it ran rather than because it matched the username.
Method B Function
$AllCompFound = #()
$AllADComputer = Get-ADComputer -Properties Name -SearchBase "WhatsItToYa" -filter 'Name -like "*"' | Select-Object Name
$hive = [Microsoft:Win32.RegistryHive]::Users
ForEach($Computer in $AllADComputers) {
try {
$base = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey($hive, $Computer.Name)
$key = &base.OpenSubKey($strSID)
if ($!key) {
#DOSTUFF
} else {
$AllCompFound += $Computer.Name
#DOOTHERSTUFF
}
} catch {
#IDONTTHROWBECAUSEIWANTITTOCONTINUE
} finally {
if($key) {
$key.Close()
}
if ($base) {
$base.Close()
}
}
}
NOTE: I have another function that converts the username into a SID prior to this function. It works.
Where my eyes start to glaze over is using Invoke-Command and actually return a value back, and whether or not to run all of these queries as their own PS-Session or not. My Method A returns false positives and my Method B seems to hang up on some computers.
Neither of these methods are really fast enough to get through 10k results, I've been using smaller pools of computers in order to get test these results when requested. I'm by no means an expert, but I think I have a good understanding, so any help is appreciated!
First, use WMI Win32_UserProfile, not C:\Users or registry.
Second, use reports from pc to some database, not from server to pc. This is much better usually.
About GPO: If you get access, you can Add\Remove scheduled task for such reports through GPP (not GPO) from time to time.
Third: Use PoshRSJob to make parallel queries.
Get-WmiObject -Class 'Win32_USerProfile' |
Select #(
'SID',
#{
Name = 'LastUseTime';
Expression = {$_.ConvertToDateTime($_.LastUseTime)}}
#{
Name = 'NTAccount';
Expression = { [System.Security.Principal.SecurityIdentifier]::new($_.SID).Translate([System.Security.Principal.NTAccount])}}
)
Be careful with translating to NTAccount: if SID does not translates, it will cause error, so, maybe, it's better not to collect NTAccount from user space.
If you have no other variants, parallel jobs using PoshRSJob
Example for paralleling ( maybe there are some typos )
$ToDo = [System.Collections.Concurrent.ConcurrentQueue[string]]::new() # This is Queue (list) of computers that SHOULD be processed
<# Some loop through your computers #>
<#...#> $ToDo.Enqueue($computerName)
<#LoopEnd#>
$result = [System.Collections.Concurrent.ConcurrentBag[Object]]::new() # This is Bag (list) of processing results
# This function has ComputerName on input, and outputs some single value (object) as a result of processing this computer
Function Get-MySpecialComputerStats
{
Param(
[String]$ComputerName
)
<#Some magic#>
# Here we make KSCustomObject form Hashtable. This is result object
return [PSCustomObject]#{
ComputerName = $ComputerName;
Result = 'OK'
SomeAdditionalInfo1 = 'whateverYouWant'
SomeAdditionalInfo2 = 42 # Because 42
}
}
# This is script that runs on background. It can not output anything.
# It takes 2 args: 1st is Input queue, 2nd is output queue
$JobScript = [scriptblock]{
$inQueue = [System.Collections.Concurrent.ConcurrentQueue[string]]$args[0]
$outBag = [System.Collections.Concurrent.ConcurrentBag[Object]]$args[1]
$compName = $null
# Logging inside, if you need it
$log = [System.Text.StringBuilder]::new()
# we work until inQueue is empty ( then TryDequeue will return false )
while($inQueue.TryDequeue([ref] $compName) -eq $true)
{
$r= $null
try
{
$r = Get-MySpecialComputerStats -ComputerName $compName -EA Stop
[void]$log.AppendLine("[_]: $($compName) : OK!")
[void]$outBag.Add($r) # We append result to outBag
}
catch
{
[void]$log.AppendLine("[E]: $($compName) : $($_.Exception.Message)")
}
}
# we return log.
return $log.ToString()
}
# Some progress counters
$i_max = $ToDo.Count
$i_cur = $i_max
# We start 20 jobs. Dont forget to say about our functions be available inside job
$jobs = #(1..20) <# Run 20 threads #> | % { Start-RSJob -ScriptBlock $JobScript -ArgumentList #($ToDo, $result) -FunctionsToImport 'Get-MySpecialComputerStats' }
# And once per 3 seconds we check, how much entries left in Queue ($todo)
while ($i_cur -gt 0)
{
Write-Progress -Activity 'Working' -Status "$($i_cur) left of $($i_max) computers" -PercentComplete (100 - ($i_cur / $i_max * 100))
Start-Sleep -Seconds 3
$i_cur = $ToDo.Count
}
# When there is zero, we shall wait for jobs to complete last items and return logs, and we collect logs
$logs = $jobs | % { Wait-RSJob -Job $_ } | % { Receive-RSJob -Job $_ }
# Logs is LOGS, not result
# Result is in the result variable.
$result | Export-Clixml -Path 'P:/ath/to/file.clixml' # Exporting result to CliXML file, or whatever you want
Please be careful: there is no output inside $JobScript done, so it must be perfectly done, and function Get-MySpecialComputerStats must be tested on unusual ways to return value that can be interpreted.
This article shows how to use Invoke-Async in PowerShell: https://sqljana.wordpress.com/2018/03/16/powershell-sql-server-run-in-parallel-collect-sql-results-with-print-output-from-across-your-sql-farm-fast/
I wish to run in parallel the copy-item cmdlet in PowerShell because the alternative is to use FileSystemObject via Excel and copy one file at a time out of a total of millions of files.
I have cobbled together the following:
.SYNOPSIS
<Brief description>
For examples type:
Get-Help .\<filename>.ps1 -examples
.DESCRIPTION
Copys files from one path to another
.PARAMETER FileList
e.g. C:\path\to\list\of\files\to\copy.txt
.PARAMETER NumCopyThreads
default is 8 (but can be 100 if you want to stress the machine to maximum!)
.EXAMPLE
.\CopyFilesToBackup -filelist C:\path\to\list\of\files\to\copy.txt
.NOTES
#>
[CmdletBinding()]
Param(
[String] $FileList = "C:\temp\copytest.csv",
[int] $NumCopyThreads = 8
)
$filesToCopy = New-Object "System.Collections.Generic.List[fileToCopy]"
$csv = Import-Csv $FileList
foreach($item in $csv)
{
$file = New-Object fileToCopy
$file.SrcFileName = $item.SrcFileName
$file.DestFileName = $item.DestFileName
$filesToCopy.add($file)
}
$sb = [scriptblock] {
param($file)
Copy-item -Path $file.SrcFileName -Destination $file.DestFileName
}
$results = Invoke-Async -Set $filesToCopy -SetParam file -ScriptBlock $sb -Verbose -Measure:$true -ThreadCount 8
$results | Format-Table
Class fileToCopy {
[String]$SrcFileName = ""
[String]$DestFileName = ""
}
the csv input for which looks like this:
SrcFileName,DestFileName
C:\Temp\dummy-data\101438\101438-0154723869.zip,\\backupserver\Project Archives\101438\0154723869.zip
C:\Temp\dummy-data\101438\101438-0165498273.xlsx,\\backupserver\Project Archives\101438\0165498273.xlsx
What am I missing to get this working, because when I run .\CopyFiles.ps1 -FileList C:\Temp\test.csv nothing happens. The files exist in the source path, but the file objects aren't being pulled from the -Set collection. (Unless I have misunderstood how the collection is used?)
No, I can't use robocopy to do this because there are millions of files which resolve to different paths depending upon their original location.
I have no explanation for your symptom based on the code in your question (see bottom section), but I suggest basing your solution on the (now) standard Start-ThreadJob cmdlet (comes with PowerShell Core; in Windows PowerShell, install it with Install-Module ThreadJob -Scope CurrentUser, for instance[1]):
Such a solution is more efficient than use of the third-party Invoke-Async function, which as of this writing is flawed in that it waits for jobs to finish in a tight loop, which creates unnecessary processing overhead.
Start-ThreadJob jobs are a lightweight, thread-based alternative to the process-based Start-Job background jobs, yet they integrate with the standard job-management cmdlets, such as Wait-Job and Receive-Job.
Here's a self-contained example based on your code that demonstrates its use:
Note: Whether you use Start-ThreadJob or Invoke-Async, you won't be able to explicit reference custom classes such as [fileToCopy] in the script block that runs in separate threads (runspaces; see bottom section), so the solution below simply uses [pscustomobject] instances with the properties of interest for simplicity and brevity.
# Create sample CSV file with 10 rows.
$FileList = Join-Path ([IO.Path]::GetTempPath()) "tmp.$PID.csv"
#'
Foo,SrcFileName,DestFileName,Bar
1,c:\tmp\a,\\server\share\a,baz
2,c:\tmp\b,\\server\share\b,baz
3,c:\tmp\c,\\server\share\c,baz
4,c:\tmp\d,\\server\share\d,baz
5,c:\tmp\e,\\server\share\e,baz
6,c:\tmp\f,\\server\share\f,baz
7,c:\tmp\g,\\server\share\g,baz
8,c:\tmp\h,\\server\share\h,baz
9,c:\tmp\i,\\server\share\i,baz
10,c:\tmp\j,\\server\share\j,baz
'# | Set-Content $FileList
# How many threads at most to run concurrently.
$NumCopyThreads = 8
Write-Host 'Creating jobs...'
$dtStart = [datetime]::UtcNow
# Import the CSV data and transform it to [pscustomobject] instances
# with only .SrcFileName and .DestFileName properties - they take
# the place of your original [fileToCopy] instances.
$jobs = Import-Csv $FileList | Select-Object SrcFileName, DestFileName |
ForEach-Object {
# Start the thread job for the file pair at hand.
Start-ThreadJob -ThrottleLimit $NumCopyThreads -ArgumentList $_ {
param($f)
$simulatedRuntimeMs = 2000 # How long each job (thread) should run for.
# Delay output for a random period.
$randomSleepPeriodMs = Get-Random -Minimum 100 -Maximum $simulatedRuntimeMs
Start-Sleep -Milliseconds $randomSleepPeriodMs
# Produce output.
"Copied $($f.SrcFileName) to $($f.DestFileName)"
# Wait for the remainder of the simulated runtime.
Start-Sleep -Milliseconds ($simulatedRuntimeMs - $randomSleepPeriodMs)
}
}
Write-Host "Waiting for $($jobs.Count) jobs to complete..."
# Synchronously wait for all jobs (threads) to finish and output their results
# *as they become available*, then remove the jobs.
# NOTE: Output will typically NOT be in input order.
Receive-Job -Job $jobs -Wait -AutoRemoveJob
Write-Host "Total time lapsed: $([datetime]::UtcNow - $dtStart)"
# Clean up the temp. file
Remove-Item $FileList
The above yields something like:
Creating jobs...
Waiting for 10 jobs to complete...
Copied c:\tmp\b to \\server\share\b
Copied c:\tmp\g to \\server\share\g
Copied c:\tmp\d to \\server\share\d
Copied c:\tmp\f to \\server\share\f
Copied c:\tmp\e to \\server\share\e
Copied c:\tmp\h to \\server\share\h
Copied c:\tmp\c to \\server\share\c
Copied c:\tmp\a to \\server\share\a
Copied c:\tmp\j to \\server\share\j
Copied c:\tmp\i to \\server\share\i
Total time lapsed: 00:00:05.1961541
Note that the output received does not reflect the input order, and that the overall runtime is roughly 2 times the per-thread runtime of 2 seconds (plus overhead), because 2 "batches" have to be run due to the input count being 10, whereas only 8 threads were made available.
If you upped the thread count to 10 or more (50 is the default), the overall runtime would drop to 2 seconds plus overhead, because all jobs then run concurrently.
Caveat: The above numbers stem from running in PowerShell Core, version on Microsoft Windows 10 Pro (64-bit; Version 1903), using version 2.0.1 of the ThreadJob module.
Inexplicably, the same code is much slower in Windows PowerShell, v5.1.18362.145.
However, for performance and memory consumption it is better to use batching (chunking) in your case, i.e, to process multiple file pairs per thread.
The following solution demonstrates this approach; tweak $chunkSize to find a batch size that works for you.
# Create sample CSV file with 10 rows.
$FileList = Join-Path ([IO.Path]::GetTempPath()) "tmp.$PID.csv"
#'
Foo,SrcFileName,DestFileName,Bar
1,c:\tmp\a,\\server\share\a,baz
2,c:\tmp\b,\\server\share\b,baz
3,c:\tmp\c,\\server\share\c,baz
4,c:\tmp\d,\\server\share\d,baz
5,c:\tmp\e,\\server\share\e,baz
6,c:\tmp\f,\\server\share\f,baz
7,c:\tmp\g,\\server\share\g,baz
8,c:\tmp\h,\\server\share\h,baz
9,c:\tmp\i,\\server\share\i,baz
10,c:\tmp\j,\\server\share\j,baz
'# | Set-Content $FileList
# How many threads at most to run concurrently.
$NumCopyThreads = 8
# How many files to process per thread
$chunkSize = 3
# The script block to run in each thread, which now receives a
# $chunkSize-sized *array* of file pairs.
$jobScriptBlock = {
param([pscustomobject[]] $filePairs)
$simulatedRuntimeMs = 2000 # How long each job (thread) should run for.
# Delay output for a random period.
$randomSleepPeriodMs = Get-Random -Minimum 100 -Maximum $simulatedRuntimeMs
Start-Sleep -Milliseconds $randomSleepPeriodMs
# Produce output for each pair.
foreach ($filePair in $filePairs) {
"Copied $($filePair.SrcFileName) to $($filePair.DestFileName)"
}
# Wait for the remainder of the simulated runtime.
Start-Sleep -Milliseconds ($simulatedRuntimeMs - $randomSleepPeriodMs)
}
Write-Host 'Creating jobs...'
$dtStart = [datetime]::UtcNow
$jobs = & {
# Process the input objects in chunks.
$i = 0
$chunk = [pscustomobject[]]::new($chunkSize)
Import-Csv $FileList | Select-Object SrcFileName, DestFileName | ForEach-Object {
$chunk[$i % $chunkSize] = $_
if (++$i % $chunkSize -ne 0) { return }
# Note the need to wrap $chunk in a single-element helper array (, $chunk)
# to ensure that it is passed *as a whole* to the script block.
Start-ThreadJob -ThrottleLimit $NumCopyThreads -ArgumentList (, $chunk) -ScriptBlock $jobScriptBlock
$chunk = [pscustomobject[]]::new($chunkSize) # we must create a new array
}
# Process any remaining objects.
# Note: $chunk -ne $null returns those elements in $chunk, if any, that are non-null
if ($remainingChunk = $chunk -ne $null) {
Start-ThreadJob -ThrottleLimit $NumCopyThreads -ArgumentList (, $remainingChunk) -ScriptBlock $jobScriptBlock
}
}
Write-Host "Waiting for $($jobs.Count) jobs to complete..."
# Synchronously wait for all jobs (threads) to finish and output their results
# *as they become available*, then remove the jobs.
# NOTE: Output will typically NOT be in input order.
Receive-Job -Job $jobs -Wait -AutoRemoveJob
Write-Host "Total time lapsed: $([datetime]::UtcNow - $dtStart)"
# Clean up the temp. file
Remove-Item $FileList
While the output is effectively the same, note how only 4 jobs were created this time, each of which processed (up to) $chunkSize (3) file pairs.
As for what you tried:
The screen shot you show suggests that the problem is that your custom class, [fileToCopy], isn't visible to the script block run by Invoke-Async.
Since Invoke-Async invokes the script block via the PowerShell SDK in separate runspaces that know nothing about the caller's state, it is to be expected that these runspaces don't know your class (this equally applies to Start-ThreadJob).
However, it is unclear why that is a problem in your code, because your script block doesn't make an explicit reference to you class: your script-block parameter $file is not type-constrained (it is implicitly [object]-typed).
Therefore, simply accessing the properties of your custom-class instance inside the script block should work, and indeed does in my tests on Windows PowerShell v5.1.18362.145 on Microsoft Windows 10 Pro (64-bit; Version 1903).
However, if your real script-block code were to explicitly reference custom class [fileToCopy] - such as by defining the parameter as param([fileToToCopy] $file) - you would see the symptom.
[1] In Windows PowerShell v3 and v4, which do not come with the PowerShellGet module, Install-Module isn't available by default. However, the module can be installed on demand, as described in Installing PowerShellGet.
I use the funtion foreach-parallel for run many scriptblock like runspace
the first load snapin correctly but all next return 2 error
1..5 | Foreach-Parallel -Throttle 5 -Timeout 5 -sleeptimer 5 {
add-PSSnapin 'Quest.ActiveRoles.ADManagement'
try {
get-qadUser 'Domain\me'
} catch {
return "$($Error[0].Exception)"
}
}
my errors is :
add-PSSnapin : An item with the same key has already been added
get-qadUser : function doesn't exist
In runspacePool the command Add-PSSnapin and Import-Module doesn't work !
It's need to add in Initial Default State not in ScriptBlock:
$ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
[void]$ISS.ImportPSSnapIn('Quest.ActiveRoles.ADManagement', [ref]$null)
[void]$ISS.ImportPSModule('PSTerminalServices')
$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)
$RunspacePool.Open()
I made a comment above on what I think you should do but to clarify you will want to ensure the snapin is loaded before your call so make sure its executed before you run your Foreach-parallel.
add-PSSnapin 'Quest.ActiveRoles.ADManagement'
Foreach-Parallel -Throttle 5 -Timeout 5 -sleeptimer 5 {
get-qadUser 'domain\testUser'
}
There is an easy way to test for a cmdlet and if not found add the PSSnapin or Import-Module
Adding the below to the start of your script will check for the presence of a CMDLET and if not found, add it.
IF(
Get-Command get-qadUser -ErrorAction SilentlyContinue){}
ELSE{add-PSSnapin 'Quest.ActiveRoles.ADManagement'}
Finaly I use another code, I use like that :
$Servers | Run-Parallel -scriptBlock $ScriptBlock -TimeOut 10 -PSModules 'PSTerminalServices' -PSSnapins 'Quest.ActiveRoles.ADManagement' | out-gridView
complet code
function Run-Parallel {
<#
.Synopsis
This is a quick and open-ended script multi-threader searcher
http://www.get-blog.com/?p=189#comment-28834
Improove by Alban LOPEZ 2016
.Description
This script will allow any general, external script to be multithreaded by providing a single
argument to that script and opening it in a seperate thread. It works as a filter in the
pipeline, or as a standalone script. It will read the argument either from the pipeline
or from a filename provided. It will send the results of the child script down the pipeline,
so it is best to use a script that returns some sort of object.
Authored by Ryan Witschger - http://www.Get-Blog.com
.PARAMETER ScriptBlock
This is where you provide the PowerShell ScriptBlock that you want to multithread.
.PARAMETER ItemObj
The ItemObj represents the arguments that are provided to the child script. This is an open ended
argument and can take a single object from the pipeline, an array, a collection, or a file name. The
multithreading script does it's best to find out which you have provided and handle it as such.
If you would like to provide a file, then the file is read with one object on each line and will
be provided as is to the script you are running as a string. If this is not desired, then use an array.
.PARAMETER InputParam
This allows you to specify the parameter for which your input objects are to be evaluated. As an example,
if you were to provide a computer name to the Get-Process cmdlet as just an argument, it would attempt to
find all processes where the name was the provided computername and fail. You need to specify that the
parameter that you are providing is the "ComputerName".
.PARAMETER AddParam
This allows you to specify additional parameters to the running command. For instance, if you are trying
to find the status of the "BITS" service on all servers in your list, you will need to specify the "Name"
parameter. This command takes a hash pair formatted as follows:
#{"key" = "Value"}
#{"key1" = "Value"; "key2" = 321; "key3" = 1..9}
.PARAMETER AddSwitch
This allows you to add additional switches to the command you are running. For instance, you may want
to include "RequiredServices" to the "Get-Service" cmdlet. This parameter will take a single string, or
an aray of strings as follows:
"RequiredServices"
#("RequiredServices", "DependentServices")
.PARAMETER MaxThreads
This is the maximum number of threads to run at any given time. If resources are too congested try lowering
this number. The default value is 20.
.PARAMETER SleepTimer
This is the time between cycles of the child process detection cycle. The default value is 200ms. If CPU
utilization is high then you can consider increasing this delay. If the child script takes a long time to
run, then you might increase this value to around 1000 (or 1 second in the detection cycle).
.PARAMETER TimeOut
this is the timeOut for slower instance, only each other are returned
.PARAMETER PSModules
List of PSModule name to include for use in ScriptBlock
.PARAMETER PSSapins
List of PSSapin name to include for use in ScriptBlock
.EXAMPLE
Both of these will execute the scriptBlock and provide each of the server names in AllServers.txt
while providing the results to GridView. The results will be the output of the child script.
gc AllServers.txt | Run-Parallel $ScriptBlock_GetTSUsers -MaxThreads $findOut_AD.ActiveDirectory.Servers.count -PSModules 'PSTerminalServices' | out-gridview
#>
Param(
[Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]
$ItemObj,
[ScriptBlock]$ScriptBlock = $null,
$InputParam = $Null,
[HashTable] $AddParam = #{},
[Array] $AddSwitch = #(),
$MaxThreads = 20,
$SleepTimer = 200,
$TimeOut = 5,
[string[]]$PSSapins = $null,
[string[]]$PSModules = $null
)
Begin{
$ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
ForEach ($Snapin in $PSSapins){
[void]$ISS.ImportPSSnapIn($Snapin, [ref]$null)
}
ForEach ($Module in $PSModules){
[void]$ISS.ImportPSModule($Module)
}
$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)
$RunspacePool.Open()
$Jobs = #()
if ($CommonObject.Settings.Debug) {$CommonObject.host.ui.WriteLine("[ Start ] :")}
}
Process{
Write-Progress -Activity "Preloading Threads" -Status "Starting Job $($jobs.count)"
#ForEach ($Object in $ItemObj){
if ($ItemObj){
$PowershellThread = [powershell]::Create().AddScript($ScriptBlock)
If ($InputParam -ne $Null){
$PowershellThread.AddParameter($InputParam, $ItemObj.ToString()) | out-null
}Else{
$PowershellThread.AddArgument($ItemObj.ToString()) | out-null
}
ForEach($Key in $AddParam.Keys){
$PowershellThread.AddParameter($Key, $AddParam.$key) | out-null
}
ForEach($Switch in $AddSwitch){
$PowershellThread.AddParameter($Switch) | out-null
}
$PowershellThread.RunspacePool = $RunspacePool
$Handle = $PowershellThread.BeginInvoke()
$Job = [pscustomobject][ordered]#{Handle=''; Thread=''; object=''}
$Job.Handle = $Handle
$Job.Thread = $PowershellThread
$Job.Object = $ItemObj.ToString()
$Jobs += $Job
if ($CommonObject.Settings.Debug) {$CommonObject.host.ui.WriteLine("`t$ItemObj")}
}
#}
}
End{
$ResultTimer = Get-Date
While (#($Jobs | Where-Object {$_.Handle -ne $Null}).count -gt 0) {
$Remaining = "$($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False}).object)"
If ($Remaining.Length -gt 60){
$Remaining = $Remaining.Substring(0,60) + "..."
}
Write-Progress `
-Activity "Waiting for Jobs - $($MaxThreads - $($RunspacePool.GetAvailableRunspaces())) of $MaxThreads threads running" `
-PercentComplete (($Jobs.count - $($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False}).count)) / $Jobs.Count * 100) `
-Status "$(#($($Jobs | Where-Object {$_.Handle.IsCompleted -eq $False})).count) remaining - $remaining"
ForEach ($Job in $($Jobs | Where-Object {$_.Handle.IsCompleted -eq $True})){
$Job.Thread.EndInvoke($Job.Handle)
$Job.Thread.Dispose()
$Job.Thread = $Null
$Job.Handle = $Null
$ResultTimer = Get-Date
}
If (($(Get-Date) - $ResultTimer).totalseconds -gt $TimeOut){
$NOK = $($Jobs | Where-Object {$_.Handle})
if ($CommonObject.Settings.Debug) {$CommonObject.host.ui.WriteErrorLine("[ TimeOut ] $($NOK.count) : $($NOK.object)")}
$NOK | %{
if ($CommonObject.Settings.Debug) {$CommonObject.host.ui.WriteErrorLine("[ stop ] $($_.object)")}
$_.thread.Stop() | Out-Null
if ($CommonObject.Settings.Debug) {$CommonObject.host.ui.WriteErrorLine("[ dispose ] $($_.object)")}
$_.thread.Dispose()
if ($CommonObject.Settings.Debug) {$CommonObject.host.ui.WriteErrorLine("[ null ] $($_.object)")}
$_.Thread = $Null
$_.Handle = $Null
}
$RunspacePool.Close() | Out-Null
$RunspacePool.Dispose() | Out-Null
exit
}
Start-Sleep -Milliseconds $SleepTimer
}
$RunspacePool.Close() | Out-Null
$RunspacePool.Dispose() | Out-Null
}
}
I am looking for a way to restart three services on multiple servers simultaneously. I know how to restart services against a list of servers by using a loop but as I have many servers it would take a long time to wait for each service on each server to restart in a sequential order. Is there a way to send restart service command to all servers at once instead of waiting for each server?
You could try to work with jobs. Jobs are run in the background and you have to retrieve them with Get-Job to see their status. Please read the information to Powershell jobs on these two sites:
http://msdn.microsoft.com/en-us/library/dd878288%28v=vs.85%29.aspx
http://technet.microsoft.com/de-DE/library/hh847783.aspx
Your code would look something like this:
$servernames | ForEach-Object {Start-Job -Name "Job-$_" -Scriptblock {"Enter your code here -Computername $_"}}
This will create a background job for each servername. As already mentioned you can see the status using the cmdlet Get-Job. To get the result use the cmdlet Receive-Job.
you can use the invoke-command cmdlet
invoke-command -computername computer1,computer2,computer3 {restart-service servicename}
I use and improove a multi-thread Function, you can use it like :
$Script = {
param($Computername)
restart-service servicename -Computername $Computername
}
#('Srv1','Srv2') | Run-Parallel -ScriptBlock $Script
include this code in your script
function Run-Parallel {
<#
.Synopsis
This is a quick and open-ended script multi-threader searcher
http://www.get-blog.com/?p=189#comment-28834
Improove by Alban LOPEZ 2016
.Description
This script will allow any general, external script to be multithreaded by providing a single
argument to that script and opening it in a seperate thread. It works as a filter in the
pipeline, or as a standalone script. It will read the argument either from the pipeline
or from a filename provided. It will send the results of the child script down the pipeline,
so it is best to use a script that returns some sort of object.
.PARAMETER ScriptBlock
This is where you provide the PowerShell ScriptBlock that you want to multithread.
.PARAMETER ItemObj
The ItemObj represents the arguments that are provided to the child script. This is an open ended
argument and can take a single object from the pipeline, an array, a collection, or a file name. The
multithreading script does it's best to find out which you have provided and handle it as such.
If you would like to provide a file, then the file is read with one object on each line and will
be provided as is to the script you are running as a string. If this is not desired, then use an array.
.PARAMETER InputParam
This allows you to specify the parameter for which your input objects are to be evaluated. As an example,
if you were to provide a computer name to the Get-Process cmdlet as just an argument, it would attempt to
find all processes where the name was the provided computername and fail. You need to specify that the
parameter that you are providing is the "ComputerName".
.PARAMETER AddParam
This allows you to specify additional parameters to the running command. For instance, if you are trying
to find the status of the "BITS" service on all servers in your list, you will need to specify the "Name"
parameter. This command takes a hash pair formatted as follows:
#{"key" = "Value"}
#{"key1" = "Value"; "key2" = 321; "key3" = 1..9}
.PARAMETER AddSwitch
This allows you to add additional switches to the command you are running. For instance, you may want
to include "RequiredServices" to the "Get-Service" cmdlet. This parameter will take a single string, or
an aray of strings as follows:
"RequiredServices"
#("RequiredServices", "DependentServices")
.PARAMETER MaxThreads
This is the maximum number of threads to run at any given time. If ressources are too congested try lowering
this number. The default value is 20.
.PARAMETER SleepTimer_ms
This is the time between cycles of the child process detection cycle. The default value is 200ms. If CPU
utilization is high then you can consider increasing this delay. If the child script takes a long time to
run, then you might increase this value to around 1000 (or 1 second in the detection cycle).
.PARAMETER TimeOutGlobal
this is the TimeOut in second for listen the last thread, after this timeOut All thread are closed, only each other are returned
.PARAMETER TimeOutThread
this is the TimeOut in second for each thread, the thread are aborted at this time
.PARAMETER PSModules
List of PSModule name to include for use in ScriptBlock
.PARAMETER PSSapins
List of PSSapin name to include for use in ScriptBlock
.EXAMPLE
1..20 | Run-Parallel -ScriptBlock {param($i) Start-Sleep $i; "> $i sec <"} -TimeOutGlobal 15 -TimeOutThread 5
.EXAMPLE
Both of these will execute the scriptBlock and provide each of the server names in AllServers.txt
while providing the results to GridView. The results will be the output of the child script.
gc AllServers.txt | Run-Parallel $ScriptBlock_GetTSUsers -MaxThreads $findOut_AD.ActiveDirectory.Servers.count -PSModules 'PSTerminalServices' | out-gridview
#>
Param(
[Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]
$ItemObj,
[ScriptBlock]$ScriptBlock = $null,
$InputParam = $Null,
[HashTable] $AddParam = #{},
[Array] $AddSwitch = #(),
$MaxThreads = 20,
$SleepTimer_ms = 100,
$TimeOutGlobal = 300,
$TimeOutThread = 100,
[string[]]$PSSapins = $null,
[string[]]$PSModules = $null,
$Modedebug = $true
)
Begin{
$ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
ForEach ($Snapin in $PSSapins){
[void]$ISS.ImportPSSnapIn($Snapin, [ref]$null)
}
ForEach ($Module in $PSModules){
[void]$ISS.ImportPSModule($Module)
}
$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)
$RunspacePool.CleanupInterval=1000
$RunspacePool.Open()
$Jobs = #()
}
Process{
#ForEach ($Object in $ItemObj){
if ($ItemObj){
Write-Host $ItemObj -ForegroundColor Yellow
$PowershellThread = [powershell]::Create().AddScript($ScriptBlock)
If ($InputParam -ne $Null){
$PowershellThread.AddParameter($InputParam, $ItemObj.ToString()) | out-null
}Else{
$PowershellThread.AddArgument($ItemObj.ToString()) | out-null
}
ForEach($Key in $AddParam.Keys){
$PowershellThread.AddParameter($Key, $AddParam.$key) | out-null
}
ForEach($Switch in $AddSwitch){
$PowershellThread.AddParameter($Switch) | out-null
}
$PowershellThread.RunspacePool = $RunspacePool
$Handle = $PowershellThread.BeginInvoke()
$Job = [pscustomobject][ordered]#{
Handle = $Handle
Thread = $PowershellThread
object = $ItemObj.ToString()
Started = Get-Date
}
$Jobs += $Job
}
#}
}
End{
$GlobalStartTime = Get-Date
$continue = $true
While (#($Jobs | Where-Object {$_.Handle -ne $Null}).count -gt 0 -and $continue) {
ForEach ($Job in $($Jobs | Where-Object {$_.Handle.IsCompleted -eq $True})){
$out = $Job.Thread.EndInvoke($Job.Handle)
$out # return vers la sortie srandard
#Write-Host $out -ForegroundColor green
$Job.Thread.Dispose() | Out-Null
$Job.Thread = $Null
$Job.Handle = $Null
}
foreach ($InProgress in $($Jobs | Where-Object {$_.Handle})) {
if ($TimeOutGlobal -and (($(Get-Date) - $GlobalStartTime).totalseconds -gt $TimeOutGlobal)){
$Continue = $false
#Write-Host $InProgress -ForegroundColor magenta
}
if (!$Continue -or ($TimeOutThread -and (($(Get-Date) - $InProgress.Started).totalseconds -gt $TimeOutThread))) {
$InProgress.thread.Stop() | Out-Null
$InProgress.thread.Dispose() | Out-Null
$InProgress.Thread = $Null
$InProgress.Handle = $Null
#Write-Host $InProgress -ForegroundColor red
}
}
Start-Sleep -Milliseconds $SleepTimer_ms
}
$RunspacePool.Close() | Out-Null
$RunspacePool.Dispose() | Out-Null
}
}
Here is what I'd like to achieve in one way or another.
I have a custom assembly defining some objects. In my script, I create a custom object that I'd like to pass to a script block, keeping that object behavior.
Add-Type -AssemblyName MyCustomDLL
$global:object = new-object MyCustomDLL.MyCustomObject()
$object | gm
$jobWork = { param ($object) $object | gm } # I'd like to keep my object behavior in that block
$job = Start-Job -ScriptBlock $jobWork -ArgumentList $object
Wait-Job $job
Receive-Job $job
How can I do that or achieve the same effect? Thanks for your help
Instead of background jobs you may use PowerShell with BeginInvoke, EndInvoke. Here is the simple but working example of passing a live object in a "job", changing it there, getting the results:
# live object to be passed in a job and changed there
$liveObject = #{ data = 42}
# job script
$script = {
param($p1)
$p1.data # some output (42)
$p1.data = 3.14 # change the live object data
}
# create and start the job
$p = [PowerShell]::Create()
$null = $p.AddScript($script).AddArgument($liveObject)
$job = $p.BeginInvoke()
# wait for it to complete
$done = $job.AsyncWaitHandle.WaitOne()
# get the output, this line prints 42
$p.EndInvoke($job)
# show the changed live object (data = 3.14)
$liveObject
Background jobs are built on top of PowerShell remoting and as such, perform similar actions when passing objects around. They would serialize/ deserialize them rather than pass them with all their complexity.
My guess is that the only way to get complex object is just to pass constructor arguments and/ or operations as -ArgumentList and create object inside job.
In such a case also adding assembly would have to be part of the job:
Start-Job {
param ($ConstructorArguments)
Add-Type -AssemblyName MyCustomDll
$object = New-Object MyCustomDll.MyCustomObject $ConstructorArguments
$object | Get-Member
} -ArgumentList Foo, Bar | Wait-Job | Receive-Job