Do threads still execute using -asjob with wait-job? - powershell

Hello all and good afternoon!
I had a quick question regarding -asjob running with invoke-command.
If I run 2 Invoke-Command's using -asjob, does it run simultaneously when I try to receive the ouput? Does this mean wait-job waits till the first job specified is finished running to get the next results?
Write-Host "Searching for PST and OST files. Please be patient!" -BackgroundColor White -ForegroundColor DarkBlue
$pSTlocation = Invoke-Command -ComputerName localhost -ScriptBlock {Get-Childitem "C:\" -Recurse -Filter "*.pst" -ErrorAction SilentlyContinue | % {Write-Host $_.FullName,$_.lastwritetime}} -AsJob
$OSTlocation = Invoke-Command -ComputerName localhost -ScriptBlock {Get-Childitem "C:\Users\me\APpdata" -Recurse -Filter "*.ost" -ErrorAction SilentlyContinue | % {Write-Host $_.FullName,$_.lastwritetime} } -AsJob
$pSTlocation | Wait-Job | Receive-Job
$OSTlocation | Wait-Job | Receive-Job
Also, another question: can i save the output of the jobs to a variable without it showing to the console? Im trying to make it where it checks if theres any return, and if there is output it, but if theres not do something else.
I tried:
$job1 = $pSTlocation | Wait-Job | Receive-Job
if(!$job1){write-host "PST Found: $job1"} else{ "No PST Found"}
$job2 = $OSTlocation | Wait-Job | Receive-Job
if(!$job2){write-host "OST Found: $job2"} else{ "No OST Found"}
No luck, it outputs the following:

Note: This answer does not directly answer the question - see the other answer for that; instead, it shows a reusable idiom for a waiting for multiple jobs to finish in a non-blocking fashion.
The following sample code uses the child-process-based Start-Job cmdlet to create local jobs, but the solution equally works with local thread-based jobs created by Start-ThreadJob as well as jobs based on remotely executing Invoke-Command -ComputerName ... -AsJob commands, as used in the question.
It shows a reusable idiom for a waiting for multiple jobs to finish in a non-blocking fashion that allows for other activity while waiting, along with collecting per-job output in an array.
Here, the output is only collected after each job completes, but note that collecting it piecemeal, as it becomes available, is also an option, using (potentially multiple) Receive-Job calls even before a job finishes.
# Start two jobs, which run in parallel, and store the objects
# representing them in array $jobs.
# Replace the Start-Job calls with your
# Invoke-Command -ComputerName ... -AsJob
# calls.
$jobs = (Start-Job { Get-Date; sleep 1 }),
(Start-Job { Get-Date '1970-01-01'; sleep 2 })
# Initialize a helper array to keep track of which jobs haven't finished yet.
$remainingJobs = $jobs
# Wait iteratively *without blocking* until any job finishes and receive and
# output its output, until all jobs have finished.
# Collect all results in $jobResults.
$jobResults =
while ($remainingJobs) {
# Check if at least 1 job has terminated.
if ($finishedJob = $remainingJobs | Where State -in Completed, Failed, Stopped, Disconnected | Select -First 1) {
# Output the just-finished job's results as part of custom object
# that also contains the original command and the
# specific termination state.
[pscustomobject] #{
Job = $finishedJob.Command
State = $finishedJob.State
Result = $finishedJob | Receive-Job
}
# Remove the just-finished job from the array of remaining ones...
$remainingJobs = #($remainingJobs) -ne $finishedJob
# ... and also as a job managed by PowerShell.
Remove-Job $finishedJob
} else {
# Do other things...
Write-Host . -NoNewline
Start-Sleep -Milliseconds 500
}
}
# Output the jobs' results
$jobResults
Note:
It's tempting to try $remainingJobs | Wait-Job -Any -Timeout 0 to momentarily check for termination of any one job without blocking execution, but as of PowerShell 7.1 this doesn't work as expected: even already completed jobs are never returned - this appears to be bug, discussed in GitHub issue #14675.

If I run 2 Invoke-Command's using -asjob, does it run simultaneously when I try to receive the output?
Yes, PowerShell jobs always run in parallel, whether they're executing remotely, as in your case (with Invoke-Command -AsJob, assuming that localhost in the question is just a placeholder for the actual name of a different computer), or locally (using Start-Job or Start-ThreadJob).
However, by using (separate) Wait-Job calls, you are synchronously waiting for each jobs to finish (in a fixed sequence, too). That is, each Wait-Job calls blocks further execution until the target job terminates.[1]
Note, however, that both jobs continue to execute while you're waiting for the first one to finish.
If, instead of waiting in a blocking fashion, you want to perform other operations while you wait for both jobs to finish, you need a different approach, detailed in the the other answer.
can i save the output of the jobs to a variable without it showing to the console?
Yes, but the problem is that in your remotely executing script block ({ ... }) you're mistakenly using Write-Host in an attempt to output data.
Write-Host is typically the wrong tool to use, unless the intent is to write to the display only, bypassing the success output stream and with it the ability to send output to other commands, capture it in a variable, or redirect it to a file. To output a value, use it by itself; e.g., $value instead of Write-Host $value (or use Write-Output $value, though that is rarely needed); see this answer.
Therefore, your attempt to collect the job's output in a variable failed, because the Write-Host output bypassed the success output stream that variable assignments capture and went straight to the host (console):
# Because the job's script block uses Write-Host, its output goes to the *console*,
# and nothing is captured in $job1
$job1 = $pSTlocation | Wait-Job | Receive-Job
(Incidentally, the command could be simplified to
$job1 = $pSTlocation | Receive-Job -Wait).
[1] Note that Wait-Job has an optional -Timeout parameter, which allows you to limit waiting to at most a given number of seconds and return without output if the target job hasn't finished yet. However, as of PowerShell 7.1, -Timeout 0 for non-blocking polling for whether jobs have finished does not work - see GitHub issue #14675.

Related

Why does Write-Host not work when run in a powershell job?

Sorry if I'm being a dumb powershell noob, but what's wrong with jobs apparently being unable to write to the terminal? And how can I fix that?
# test.ps1
function myjob {
Write-Host "Hello, World!" # doesn't show
}
Start-Job -Name MyJob -ScriptBlock ${function:myjob}
Wait-Job MyJob
Remove-Job MyJob
It sounds like you're trying to use Write-Host to directly, synchronously write to the console (terminal) from a background job.
However, PowerShell jobs do not allow direct access to the caller's console. Any output - even to the PowerShell host (which in foreground use is the console, if run in one) is routed through PowerShell's system of output streams (see the conceptual about_Redirection help topic).
Therefore, you always need the Receive-Job cmdlet in order to receive output from a PowerShell job.
The following example receives the job output synchronously, i.e. it blocks execution until the job finishes (-Wait) and then removes it (-AutoRemoveJob); see the bottom section for an asynchronous (polling, non-blocking) approach.
$null = Start-Job -Name MyJob -ScriptBlock { Write-Host "Hello, World!" }
Receive-Job -Wait -AutoRemoveJob -Name MyJob
Caveat re use of Write-Host in jobs:
In foreground use, Write-Host output - even though primarily designed to go to the host (console) - can be redirected or captured via the information stream (whose number is 6, available in PSv5+); e.g.:
# OK - no output
Write-Host 'silence me' 6>$null
Write-Host output received via a (child-process-based) background job, however, can not be redirected or captured, as of PowerShell 7.2.1:
# !! `silence me` still prints.
Start-Job { Write-Host 'silence me' } | Receive-Job -Wait -AutoRemoveJob 6>$null
By contrast, it can be redirected/captured when using a (generally preferable) thread-based background job (as opposed to a child-process-based background job), via Start-ThreadJob:
# OK - no output
Start-ThreadJob { Write-Host 'silence me' } | Receive-Job -Wait -AutoRemoveJob 6>$null
Waiting for a job to complete in a non-blocking fashion, passing job output through as it becomes available:
# Start a simple job that writes a "." to the host once a second,
# for 5 seconds
$job = Start-Job $job -ScriptBlock {
1..5| ForEach-Object { Write-Host -NoNewLine .; Start-Sleep 1 }
}
"Waiting for job $($job.Id) to terminate while passing its output through..."
do {
$job | Receive-Job # See if job output is available (non-blocking) and pass it through
Start-Sleep 1 # Do other things or sleep a little.
} while (($job | Get-Job).State -in 'NotStarted', 'Running')
"`nJob terminated with state '$($job.State)'."
$job | Remove-Job # Clean up.
Note: In this simple case, the expected termination states are Completed (either no or only non-terminating errors occurred) or Failed (a script-terminating error was generated with throw (and not caught inside the job)).
See the [System.Management.Automation.JobState] enumeration for the complete list of possible states.
The job object returned by Start-Job - rather than a self-chosen name via the -Name parameter - is used to interact with the job. This eliminates the ambiguity of possibly multiple jobs being present with a given -Name, all of which would be targeted.

How can I speed up a PowerShelll foreach loop

I have a PowerShell script that connects to a database and pulls a list of user data. I take this data and create a foreach loop to run a script for the data.
This is working but its slow as the results could be 1000+ entries, and it has to complete the Script.bat for User A before it can start User B. The Script.bat for a single user is independent from another and takes ~30s for each user.
Is there a way to speed this up at all? I've been playing with -Parallel, ForEach-Object and workflow but I can't get it to work, likely due to me being a noob in PS.
foreach ($row in $Dataset.tables[0].rows)
{
$UserID=$row.value
$DeviceID=$row.value1
$EmailAddress=$row.email_address
cmd.exe /c "`"$PSScriptRoot`"\bin\Script.bat -c `" -Switch $UserID`" >> `"$PSScriptRoot`"\${FileName3}_REST_${DateTime}.txt 2> nul";
}
You said it yourself, your bottleneck is with the batch file in your script, not the loop itself. foreach (as opposed to ForEach-Object) is already the faster foreach loop mechanism in PowerShell. Investigate your batch file to find out why it takes 30 seconds to complete, and optimize it where you can.
Using Jobs
Note: Start-Job will run the job under another process. If you have PowerShell Core you can make use of the Start-ThreadJob cmdlet in lieu of Start-Job. This will start your job as part of another thread of the same process instead of starting another process.
If you can't optimize your batch script or optimize it to meet your needs, then you can consider using Start-Job to kick off the job to execute asynchronously, and then check the result and get any output from it using Receive-Job. For example:
# Master list of jobs you need to check the result of later
$jobs = New-Object System.Collections.Generic.List[System.Management.Automation.Job]
# Run your script for each row
foreach ($row in $Dataset.tables[0].rows)
{
$UserID=$row.value
$DeviceID=$row.value1
$EmailAddress=$row.email_address
# Use Start-Job here to kick off the script and store the job information
# for later retrieval.
# The $using: scope modifier allows you to make use of variables that were
# defined in the session calling Start-Job
$job = Start-Job -ScriptBlock { cmd.exe /c "`"${using:PSScriptRoot}`"\bin\Script.bat -c `" -Switch ${using:UserID}`" >> `"${using:PSScriptRoot}`"\${using:FileName3}_REST_${DateTime}.txt 2> nul"; }
# Add the execution to the $jobs list to check the result of later
# Casting to void here prevents the Add method from returning the object
# we've added.
[void]$jobs.Add($job)
}
# Wait for the jobs to be done
Write-Host 'Waiting for all jobs to complete...'
while( $jobs | Where-Object { $_.State -eq 'Running' } ){
Start-Sleep -s 10
}
# Retrieve the output of the jobs
foreach( $j in $jobs ) {
Receive-Job $j
}
Note: Since you have ~1000 times you need to execute this script, you may want to consider writing your logic to only run a certain number of jobs at a time. My example above starts all necessary jobs without regarding the number that may execute at once.
For more information about jobs and the properties you can inspect on a running/completed job, check the links below:
About Jobs
Job Class
Using Scope*
* The documentation states that the using scope can only be declared when working with remote sessions, but this seems to work fine with Start-Job even if the job is local.

PowerShell Receive-Job Output To Variable Or File But Not Screen

I am trying to get the job details without outputting the data to the screen. However, regardless of what option I try, the job logs always get sent to the console. Any ideas on how to save the logs in a variable or file without outputting that data to console?
Receive-Job -Id $id -Keep -ErrorAction Continue > C:\Temp\Transcript-$VM.txt
$info = Receive-Job -Id $id -Keep -ErrorAction Continue
You state that your job uses Write-Host output and that you're running Windows PowerShell v5.1.
In order to also capture Write-Host output - which in v5+ is sent to the information stream (stream number 6) - use redirection 6>&1:
# Capture both success output and information-stream output
# (Write-Host) output in $info.
$info = Receive-Job -Id $id -Keep -ErrorAction Continue 6>&1
Unfortunately, due to a known bug, you'll still get console output as well (bug is still present in PowerShell Core 7.0.0-preview.5).
Catch-all redirection *>&1 normally routes all streams through the success output stream.
Unfortunately, due to the bug linked to above, the following streams cannot be captured or redirected at all when using background jobs or remoting:
verbose messages (4)
debug messages (5)
The only workaround is to capture the streams inside the job and save them to a file from there, and then access the files from the caller later.
Of course, this requires that you have control over how the jobs are created.
A simplified example:
# Redirect all output streams *inside* the job to a file...
Start-Job {
& {
# The job's commands go here.
# Note that for any *verbose* output to be captured,
# verbose output must explicitly turned on, such as with
# the -Verbose common parameter here.
# You can also set $VerbosePreference = 'Continue', which
# cmdlets (including advanced functions/scripts) will honor.
'success'; write-verbose -Verbose 'verbose'; write-host 'host'
} *> $HOME/out.txt
} | Receive-Job -Wait -AutoRemove
# ... then read the resulting file.
Get-Content $HOME/out.txt
Note that I've used a full path as the redirection target, because, unfortunately, in v6- versions of PowerShell script blocks executed in background jobs do not inherit the caller's current location. This will change in PowerShell Core v7.0.
Try placing it in a pipeline, and see if that works:
Receive-Job -Id $id -Keep -ErrorAction Continue | Set-Content 'C:\Temp\Transcript-$VM.txt'

Assign a variable inside of a scriptblock while running a job

Related to Terminate part of powershell script and continue.
Partially related to Powershell Job Always Shows Complete.
My script runs locally and access the registry hive of a remote PC. I need the value of registry keys to be written into a $RegHive variable. And I want to monitor it as a job in case some PC freezes, I can terminate the command and move on to another PC.
My original code would be:
$global:RegHive = $null
$job = Start-Job -ScriptBlock {
$RegHive = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey("SomeKeyName", "SomePCName")
}
But no matter what I do, the variable $RegHive is empty.
If I do $RegHive = (Get-Job | Receive-Job) some value gets assigned to $RegHive that on one side looks exactly as if I would run it normally without a job/scriptblock, ie:
$RegHive = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey("SomeKeyName", "SomePCName")
and even has the same $RegHive.SubKeyCount
But the "normal" one has $RegHive.GetSubKeyName() method and the one from job doesn't.
How do I escape assigning a variable with Receive-Job and do the assignment directly inside the scriptblock, which is run as a job?
In simple words:
$job = Start-Job -ScriptBlock {$a = 1 + 2}
How to get $a be equal to 3 without $a = (Get-job | Receive-job)?
This might be helpful for you. The job is sort of like a variable
What you can do is name the job and then call it by name with -Keep to maintain it's value stored - aka it will store all final output inside itself until you call it. (it can be kept but the default is to remove it once called)
$global:RegHive = $null
Start-Job -Name "RegHive" -ScriptBlock {
[Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey("SomeKeyName", "SomePCName")
}
Receive-Job -Name "RegHive" -Keep
obviously calling the Receive-Job immediately after defeats the purpose of jobs, they add a lot of overhead, and are only efficient when needing to do multiple things at once. - if you call for 100s or thousands at once, you could do get-job | wait-job then when finished start using their outputs ---- wait-job also accepts job names or can wait on your entire list of jobs.
another option to set the variable is
$RegHive = "Receive-Job -Name "RegHive"
and finally, you can do this to use the value
get-<insert command> -value "$(Receive-Job -Name 'RegHive' -Keep)" -argument2 "YADA YADA"
remember keep will not delete the value and can be "Received" again later.

Getting Processes via Get-Counter does not refresh when processes change and throws errors if processes end

My script monitors the CPU usage of the process, looping the code every 5 sec and writing it to a file. Which works fine.
But I found when a new Process runs my script will not find it until I stop the script and rerun it again.
Also if a process ends/stops, the script give an this error:
Get-Counter : The data in one of the performance counter samples is
not valid. View the Status property for each
PerformanceCounterSample object to make sure it contains valid data.
At line:2 char:34
It seems PowerShell retrieves the Process information only once and caches it.
If I run the bellow script (which is a part of all my script), it runs perfectly:
while($true) {
$ProcessId = (Get-Counter "\Process(*)\ID Process").CounterSamples
$ProcessId.count
Start-Sleep -s 5
}
If I have 50 process it will gives 50, but if a new process starts it will keep giving 50 until I restart the script.
If I stop any process it will give the same error above.
Any idea how to solve this problem and force PowerShell to reread the process list without restarting the script?
You could use PowerShell Jobs to execute it in a new background process on each iteration and use -ErrorAction SilentlyContinue to suppress the error messages that might occur if one or more processes stopped during a check:
while($true) {
$ProcessId = Start-Job -ScriptBlock { (Get-Counter "\Process(*)\ID Process" -ErrorAction SilentlyContinue).CounterSamples } | Wait-Job | Receive-Job
$ProcessId.Count
Start-Sleep -Seconds 5
}