Powershell parallel processing : How to set value in a global varible - powershell

I am calling an API 500 times with 10 parallel threads as part of load testing. I want to capture the result of API call in a global variable (a counter outside script block scope) so, that I can process further for validation.
Example- In below code , I want to check if all 500 API call is success or not.
PFB code snippet-
$invokeAPI =
{
try {
$bodyContent = Get-Content $Using:inputFilepath
$Response = (Invoke-WebRequest -Method 'Post' -Uri $Using:headUri -Headers $Using:blobHeaders -Body $bodyContent).StatusCode
Write-Host -BackgroundColor Green "status Code :" $Response
}
catch [System.Exception] {
Write-Host -ForegroundColor Red "Exception caught while invoking API :" $_.ErrorDetails.Message
[int]$_.Exception.Response.StatusCode
}
}
1..500 | ForEach-Object -Parallel $invokeAPI -ThrottleLimit 10
<# ToDo...Capture API invocation Result to validate results#>

Updated:
Turns out I overcomplicated my initial answer by thinking jobs would be necessary. But it looks like they aren't. It appears it should be as simple as just outputting to a variable.
Sample script which will randomly test various HTTP statuses:
$invokeAPI = {
try {
$statusCode = 200,200,200,200,200,301,400,404,500 | Get-Random;
(iwr "http://httpbin.org/status/$statusCode").StatusCode;
}
catch {
[int]$_.Exception.Response.StatusCode;
};
};
$statuscodes = 1..20 | % -Parallel $invokeAPI -ThrottleLimit 5;
$statuscodes;
OLD - I thought Jobs would be needed, turns out you don't, see edit above
Change this:
1..500 | ForEach-Object -Parallel $invokeAPI -ThrottleLimit 10
To this:
$output = 1..500 | ForEach-Object -Parallel $invokeAPI -ThrottleLimit 10 -AsJob | Wait-Job | Receive-Job
$output
Explanation:
-AsJob - Causes it to run each task as a PowerShell job in the background
Wait-Job - Wait for the jobs to finish
Receive-Job - Get the return data for all the jobs
By running -AsJob, it will store the results in the background. You can then retrieve the job, which is the stored results of that jobs output.
Thanks to:
https://devblogs.microsoft.com/powershell/powershell-foreach-object-parallel-feature/
In fact, your example is very very similar to this example in the documentation:
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object?view=powershell-7.1#example-13--run-in-parallel-as-a-job

Related

Progress bar is not showing correctly in PowerShell 7 in ForEach -Parallel

this show 0%:
this show 4%:
the realted script:
$iWrapper = [hashtable]::Synchronized(#{ i = 0 })
$srcfile = "C:\Users\wnune\OneDrive\Escritorio\imagenes\cardlist.txt"
$urls = Get-Content $srcfile
$lines = 0
switch -File $srcfile { default { ++$lines } }
Write-Host "Total Urls to process: $lines "
Write-Progress -Activity "Downloading files" -Status "In progress" -PercentComplete $i;
$urls | ForEach-Object -Parallel {
try {
$url = $_
$filename = Split-Path $url -Leaf
$destination = "C:\Users\wnune\OneDrive\Escritorio\imagenes\$filename"
$ProgressPreference = 'SilentlyContinue'
$response = Invoke-WebRequest -Uri $url -ErrorAction SilentlyContinue
if ($response.StatusCode -ne 200) {
Write-Warning "============================================="
Write-Warning "Url $url return Error. "
continue
}
if (Test-Path $destination) {
Write-Warning "============================================="
Write-Warning "File Exist in Destination: $filename "
continue
}
$job = Start-BitsTransfer -Source $url -Destination $destination -Asynchronous
while (($job | Get-BitsTransfer).JobState -eq "Transferring" -or ($job | Get-BitsTransfer).JobState -eq "Connecting")
{
Start-Sleep -m 250
}
Switch(($job | Get-BitsTransfer).JobState)
{
"Transferred" {
Complete-BitsTransfer -BitsJob $job
}
"Error" {
$job | Format-List
}
}
}
catch
{
Write-Warning "============================================="
Write-Warning "There was an error Downloading"
Write-Warning "url: $url"
Write-Warning "file: $filename"
Write-Warning "Exception Message:"
Write-Warning "$($_.Exception.Message)"
}
$j = ++($using:iWrapper).i
$k = $using:lines
$percent = [int](100 * $j / $k)
Write-Host "PercentCalculated: $percent"
Write-Host "Progress bar not Show the %"
Write-Progress -Activity "Downloading files " -Status " In progress $percent" -PercentComplete $percent
}
Write-Progress -Activity "Downloading files" -Status "Completed" -Completed
If I am passing in -PercentComplete $percent which is an integer why does the progress bar not receive it correctly?
I have verified that the script and the environment are correctly configured but I cannot validate because the progress bar is not seen correctly.
Note:
A potential future enhancement has been proposed in GitHub issue #13433, suggesting adding parameter(s) such as -ShowProgressBar to ForEach-Object -Parallel so that it would automatically show a progress bar based on how many parallel threads have completed so far.
Leaving the discussion about whether Start-BitsTransfer alone is sufficient aside:
At least as of PowerShell v7.3.1, it seemingly is possible to call Write-Progress from inside threads created by ForEach-Object -Parallel, based on a running counter of how many threads have exited so far.
However, there are two challenges:
You cannot directly update a counter variable in the caller's runspace (thread), you can only refer to an object that is an instance of a .NET reference type in the caller's runspace...
...and modifying such an object, e.g. a hashtable must be done in a thread-safe manner, such as via System.Threading.Monitor.
Note that I don't know whether calling Write-Progress from different threads is officially supported, but it seems to work in practice, at least when the call is made in a thread-safe manner, as below.
Bug alert, as of PowerShell 7.3.1: Irrespective of whether you use ForEach-Object -Parallel or not, If you call Write-Progress too quickly in succession, only the first in such a sequence of calls takes effect:
See GitHub issue #18848
As a workaround, the code below inserts a Start-Sleep -Milliseconds 200 call after each Write-Progress call.
Not only should this not be necessary, it slows down overall execution, because threads then take longer to exit, which affects not only a given thread, but overall execution time, because it delays when threads "give up their slot" in the context of thread throttling (5 threads are allowed to run concurrently by default; use -ThrottleLimit to change that.
A simple proof of concept:
# Sample pipeline input
$urls = 1..100 | ForEach-Object { "foo$_" }
# Helper hashtable to keep a running count of completed threads.
$completedCount = #{ Value = 0 }
$urls |
ForEach-Object -parallel { # Process the input objects in parallel threads.
# Simulate thread activity of varying duration.
Start-Sleep -Milliseconds (Get-Random -Min 0 -max 3000)
# Produce output.
$_
# Update the count of completed threads in a thread-safe manner
# and update the progress display.
[System.Threading.Monitor]::Enter($using:completedCount) # lock access
($using:completedCount).Value++
# Calculate the percentage completed.
[int] $percentComplete = (($using:completedCount).Value / ($using:urls).Count) * 100
# Update the progress display, *before* releasing the lock.
Write-Progress -Activity Test -Status "$percentComplete% complete" -PercentComplete $percentComplete
# !! Workaround for the bug above - should *not* be needed.
Start-Sleep -Milliseconds 200
[System.Threading.Monitor]::Exit($using:completedCount) # release lock
}
An alternative approach in which the calling thread centrally tracks the progress of all parallel threads:
Doing so requires adding the -AsJob switch to ForEach-Object -Parallel, which, instead of the synchronous execution that happens by default, starts a (thread-based) background job, and returns a [System.Management.Automation.PSTasks.PSTaskJob] instance that represents all parallel threads as PowerShell (thread) jobs in the .ChildJobs property.
A simple proof of concept:
# Sample pipeline input
$urls = 1..100 | ForEach-Object { "foo$_" }
Write-Progress -Activity "Downloading files" -Status "Initializing..."
# Launch the parallel threads *as a background (thread) job*.
$job =
$urls |
ForEach-Object -AsJob -Parallel {
# Simulate thread activity of varying duration.
Start-Sleep -Milliseconds (Get-Random -Min 0 -max 3000)
$_ # Sample output: pass the URL through
}
# Monitor and report the progress of the thread job's
# child jobs, each of which tracks a parallel thread.
do {
# Sleep a bit to allow the threads to run - adjust as desired.
Start-Sleep -Seconds 1
# Determine how many jobs have completed so far.
$completedJobsCount =
$job.ChildJobs.Where({ $_.State -notin 'NotStarted', 'Running' }).Count
# Relay any pending output from the child jobs.
$job | Receive-Job
# Update the progress display.
[int] $percent = ($completedJobsCount / $job.ChildJobs.Count) * 100
Write-Progress -Activity "Downloading files" -Status "$percent% complete" -PercentComplete $percent
} while ($completedJobsCount -lt $job.ChildJobs.Count)
# Clean up the job.
$job | Remove-Job
While this is more work and less efficient due to the polling loop, it has two advantages:
The script blocks running in the parallel threads need not be burdened with progress-reporting code.
The polling loop affords the opportunity to perform other foreground activity while the parallel threads are running in the background.
The bug discussed above needn't be worked around, assuming your Start-Sleep interval in the polling loop is at least 200 msecs.

Add Write-Progress to Get-Job/Wait-Job

I'm using the below code to display the results of PowerShell Jobs with a timeout of 120 seconds. I would like to enhance this code by incorporating Write-Progress (based on number of jobs completed). I tried using this example as a reference, however, when I try to incorporate that code, the progress bar displays briefly after all the jobs are all done already.
$Jobs = #()
$ForceStoppedIds = #{}
$Jobs += Get-Job
$Jobs | Wait-Job -Timeout 120 | Out-Null
$Jobs | ?{$_.State -eq 'Running'} | Stop-Job -PassThru | %{$ForceStoppedIds[$_.Id] = $true}
foreach ($Job in $Jobs) {
$Name = $Job.Name
$Output = (Get-Job -Name $Name | Receive-Job)
if ($ForceStoppedIds.Contains($Job.Id)) {
Write-Output "$($Name) - Device unable to process request within 2 minutes"
} else {
Write-Output $Output
}
}
Wait-Job -Timeout 120 will block the thread until the specified timeout or all jobs have completed, hence, is not possible to display progress and wait for them at the same time.
There are 2 alternatives that I can think of, the first one would be to create a proxy command / proxy function around this cmdlet to extend it's functionality.
These blogs demonstrate how to do it:
https://devblogs.microsoft.com/scripting/proxy-functions-spice-up-your-powershell-core-cmdlets/
https://devblogs.microsoft.com/powershell/extending-andor-modifing-commands-with-proxies/
proxy function
You can also follow the indications from this helpful answer.
The other alternative is to define your own function that does a similar work as Wait-Job but, instead of blocking the thread, you can add a loop that will run based on 2 conditions:
That the elapsed time is lower than or equal to the Timeout we passed as argument to the function (we can use Diagnostics.Stopwatch for this).
And, that the jobs are still Running (the $jobs List<T> is still populated).
Note, the function below should work in most cases however is purely for demonstration purposes only and should not be relied upon.
First we define a new function that can be used to display progress as well as wait for our jobs based on a timeout:
using namespace System.Collections.Generic
using namespace System.Diagnostics
using namespace System.Threading
using namespace System.Management.Automation
function Wait-JobWithProgress {
[cmdletbinding()]
param(
[parameter(Mandatory, ValueFromPipeline)]
[object[]] $InputObject,
[parameter()]
[double] $TimeOut
)
begin {
$jobs = [List[object]]::new()
}
process {
foreach($job in $InputObject) {
$jobs.Add($job)
}
}
end {
$timer = [Stopwatch]::StartNew()
$total = $jobs.Count
$completed = 0.1
$expression = { $true }
if($PSBoundParameters.ContainsKey('TimeOut')) {
$expression = { $timer.Elapsed.TotalSeconds -le $TimeOut }
}
while((& $expression) -and $jobs) {
$remaining = $total - $completed
$average = $timer.Elapsed.TotalSeconds / $completed
$estimate = [math]::Round($remaining * $average)
$status = 'Completed Jobs: {0:0} of {1} - ETC: {2}s' -f $completed, $total, $estimate
$progress = #{
Activity = 'Waiting for Jobs'
PercentComplete = $completed / $total * 100
Status = $status
}
Write-Progress #progress
$id = [WaitHandle]::WaitAny($jobs.Finished, 200)
if($id -eq [WaitHandle]::WaitTimeout) {
continue
}
# output this job
$jobs[$id]
# remove this job
$jobs.RemoveAt($id)
$completed++
}
# Stop the jobs not yet Completed and remove them
$jobs | Stop-Job -PassThru | ForEach-Object {
Remove-Job -Job $_
"Job [#{0} - {1}] did not complete on time and was removed." -f $_.Id, $_.Name
} | Write-Warning
Write-Progress #progress -Completed
}
}
Then for testing it, we can create a few jobs with a random timer:
0..10 | ForEach-Object {
Start-Job {
Start-Sleep (Get-Random -Minimum 5 -Maximum 15)
[pscustomobject]#{
Job = $using:_
Result = 'Hello from [Job #{0:D2}]' -f $using:_
}
}
} | Wait-JobWithProgress -TimeOut 10 |
Receive-Job -AutoRemoveJob -Wait | Format-Table -AutoSize

PowerShell Start-Job not running a Scriptblock

I have been struggling for a few days now with running Start-Job with a -Scriptblock.
If I run this, I get today's date back in Receive-Job:
$sb = {Get-Date}
Start-Job -ScriptBlock $sb
Now, I have a script which does some reporting via 65+ API calls for 8 different statuses. In total, it does 500+ API calls and takes almost 20mins to complete (2-5seconds per API call, x 500), so I am looking to run each foreach block in parallel.
The script works well as a script but not as a Start-Job.
The Scriptblock will get a session token, gets data from API server in a foreach loop and then populates the results into a $Global: variable as it goes.
When I run the job, it "Completes" instantly and Has More Data is False (so I can't see any error messages):
67 Job67 BackgroundJob Completed False localhost foreach ($item....
Any suggestions as to where I am going wrong?
My sample code looks like this:
$sb = {# Get count of Status1
foreach ($item in $Global:GetItems) {
$item1 = $item.id
If ($null -eq $TokenExpiresIn) { .\Get-Token.ps1 }
$Global:TokenExpiresIn = New-TimeSpan -Start $TokenExpiresNow -End $Script:TokenExpiresTime
$Method = "GET"
$Fromitems = $item1
$Status = "foo"
$Limit = "1"
$Fields = "blah"
$URL = "https://$APIServer/apis/v1.1/status?customerId=1&itemId=$Fromitems&status=$Status&fields=$Fields&limit=$Limit"
$Header = #{"Accept" = "*/*"; "Authorization" = "Bearer $SessionToken" }
$itemStatsResponse = Invoke-WebRequest -Method $Method -Uri $URL -Header $Header
$itemStatsJSON = ConvertFrom-Json -InputObject $itemStatsResponse
$itemCount = $itemStatsJSON.metadata.count
$Global:GetItems | Where-Object -Property id -EQ $item1 | Add-Member -MemberType NoteProperty "foo" -Value $itemCount
}
}
I am running the Scriptblock as follows:
Start-Job -Name "foo" -ScriptBlock $sb
I am using PowerShell 7.1. The script as a whole runs successfully through all 500+ API calls, however it's a bit slow, so I am also looking at how I can "milti-thread" my API calls moving forward for better performance.
Thanks in advance for your support/input. The other posts on StackOverflow relating to PowerShell and Scriptblocks have not assisted me thus far.

Powershell: Run multiple jobs in parralel and view streaming results from background jobs

Overview
Looking to call a Powershell script that takes in an argument, runs each job in the background, and shows me the verbose output.
Problem I am running into
The script appears to run, but I want to verify this for sure by streaming the results of the background jobs as they are running.
Code
###StartServerUpdates.ps1 Script###
#get list of servers to update from text file and store in array
$servers=get-content c:\serverstoupdate.txt
#run all jobs, using multi-threading, in background
ForEach($server in $servers){
Start-Job -FilePath c:\cefcu_it\psscripts\PSPatch.ps1 -ArgumentList $server
}
#Wait for all jobs
Get-Job | Wait-Job
#Get all job results
Get-Job | Receive-Job
What I am currently seeing:
Id Name State HasMoreData Location Command
-- ---- ----- ----------- -------- -------
23 Job23 Running True localhost #patch server ...
25 Job25 Running True localhost #patch server ...
What I want to see:
Searching for approved updates ...
Update Found: Security Update for Windows Server 2003 (KB2807986)
Update Found: Windows Malicious Software Removal Tool - March 2013 (KB890830)
Download complete. Installing updates ...
The system must be rebooted to complete installation.
cscript exited on "myServer" with error code 3.
Reboot required...
Waiting for server to reboot (35)
Searching for approved updates ...
There are no updates to install.
cscript exited on "myServer" with error code 2.
Servername "myServer" is fully patched after 2 loops
I want to be able to see the output or store that somewhere so I can refer back to be sure the script ran and see which servers rebooted, etc.
Conclusion:
In the past, I ran the script and it went through updating the servers one at a time and gave me the output I wanted, but when I started doing more servers - this task took too long, which is why I am trying to use background jobs with "Start-Job".
Can anyone help me figure this out, please?
You may take a look at the module SplitPipeline.
It it specifically designed for such tasks. The working demo code is:
# import the module (not necessary in PS V3)
Import-Module SplitPipeline
# some servers (from 1 to 10 for the test)
$servers = 1..10
# process servers by parallel pipelines and output results immediately
$servers | Split-Pipeline {process{"processing server $_"; sleep 1}} -Load 1, 1
For your task replace "processing server $_"; sleep 1 (simulates a slow job) with a call to your script and use the variable $_ as input, the current server.
If each job is not processor intensive then increase the parameter Count (the default is processor count) in order to improve performance.
Not a new question but I feel it is missing an answer including Powershell using workflows and its parallel possibilities, from powershell version 3. Which is less code and maybe more understandable than starting and waiting for jobs, which of course works good as well.
I have two files: TheScript.ps1 which coordinates the servers and BackgroundJob.ps1 which does some kind of check. They need to be in the same directory.
The Write-Output in the background job file writes to the same stream you see when starting TheScript.ps1.
TheScript.ps1:
workflow parallelCheckServer {
param ($Servers)
foreach -parallel($Server in $Servers)
{
Invoke-Expression -Command ".\BackgroundJob.ps1 -Server $Server"
}
}
parallelCheckServer -Servers #("host1.com", "host2.com", "host3.com")
Write-Output "Done with all servers."
BackgroundJob.ps1 (for example):
param (
[Parameter(Mandatory=$true)] [string] $server
)
Write-Host "[$server]`t Processing server $server"
Start-Sleep -Seconds 5
So when starting the TheScript.ps1 it will write "Processing server" 3 times but it will not wait for 15 seconds but instead 5 because they are run in parallel.
[host3.com] Processing server host3.com
[host2.com] Processing server host2.com
[host1.com] Processing server host1.com
Done with all servers.
In your ForEach loop you'll want to grab the output generated by the Jobs already running.
Example Not Tested
$sb = {
"Starting Job on $($args[0])"
#Do something
"$($args[0]) => Do something completed successfully"
"$($args[0]) => Now for something completely different"
"Ending Job on $($args[0])"
}
Foreach($computer in $computers){
Start-Job -ScriptBlock $sb -Args $computer | Out-Null
Get-Job | Receive-Job
}
Now if you do this all your results will be mixed. You might want to put a stamp on your verbose output to tell which output came from.
Or
Foreach($computer in $computers){
Start-Job -ScriptBlock $sb -Args $computer | Out-Null
Get-Job | ? {$_.State -eq 'Complete' -and $_.HasMoreData} | % {Receive-Job $_}
}
while((Get-Job -State Running).count){
Get-Job | ? {$_.State -eq 'Complete' -and $_.HasMoreData} | % {Receive-Job $_}
start-sleep -seconds 1
}
It will show all the output as soon as a job is finished. Without being mixed up.
If you're wanting to multiple jobs in-progress, you'll probably want to massage the output to help keep what output goes with which job straight on the console.
$BGList = 'Black','Green','DarkBlue','DarkCyan','Red','DarkGreen'
$JobHash = #{};$ColorHash = #{};$i=0
ForEach($server in $servers)
{
Start-Job -FilePath c:\cefcu_it\psscripts\PSPatch.ps1 -ArgumentList $server |
foreach {
$ColorHash[$_.ID] = $BGList[$i++]
$JobHash[$_.ID] = $Server
}
}
While ((Get-Job).State -match 'Running')
{
foreach ($Job in Get-Job | where {$_.HasMoreData})
{
[System.Console]::BackgroundColor = $ColorHash[$Job.ID]
Write-Host $JobHash[$Job.ID] -ForegroundColor Black -BackgroundColor White
Receive-Job $Job
}
Start-Sleep -Seconds 5
}
[System.Console]::BackgroundColor = 'Black'
You can get the results by doing something like this after all the jobs have been received:
$array=#()
Get-Job -Name * | where{$array+=$_.ChildJobs.output}
.ChildJobs.output will have anything that was returned in each job.
function OutputJoblogs {
[CmdletBinding(DefaultParameterSetName='Name')]
Param
(
[Parameter(Mandatory=$true, Position=0)]
[System.Management.Automation.Job] $job,
[Parameter(Mandatory=$true, Position=1)]
[string] $logFolder,
[Parameter(Mandatory=$true, Position=2)]
[string] $logTimeStamp
)
#Output All logs
while ($job.sate -eq "Running" -or $job.HasMoreData){
start-sleep -Seconds 1
foreach($remotejob in $job.ChildJobs){
if($remotejob.HasMoreData){
$output=(Receive-Job $remotejob)
if($output -gt 0){
$remotejob.location +": "+ (($output) | Tee-Object -Append -file ("$logFolder\$logTimeStamp."+$remotejob.Location+".txt"))
}
}
}
}
#Output Errors
foreach($remotejob in $job.ChildJobs){
if($remotejob.Error.Count -gt0){$remotejob.location +": "}
foreach($myerr in $remotejob.Error){
$myerr 2>&1 | Tee-Object -Append -file ("$logFolder\$logTimeStamp."+$remotejob.Location+".ERROR.txt")
}
if($remotejob.JobStateInfo.Reason.ErrorRecord.Count -gt 0){$remotejob.location +": "}
foreach($myerr in $remotejob.JobStateInfo.Reason.ErrorRecord){
$myerr 2>&1 | Tee-Object -Append -file ("$logFolder\$logTimeStamp."+$remotejob.Location+".ERROR.txt")
}
}
}
#example of usage
$logfileDate="$((Get-Date).ToString('yyyy-MM-dd-HH.mm.ss'))"
$job = Invoke-Command -ComputerName "servername1","servername2" -ScriptBlock {
for ($i=1; $i -le 5; $i++) {
$i+"`n";
if($i -gt 2){
write-error "Bad thing happened"};
if($i -eq 4){
throw "Super Bad thing happened"
};
start-sleep -Seconds 1
}
} -asjob
OutputJoblogs -Job $job -logFolder "$PSScriptRoot\logs" -logTimeStamp $logfileDate

Powershell: get output from Receive-Job

I have a collection of jobs that are running. When they complete I use receive-job and it writes the output to the screen. I'd like to take that output and log it to a file. I don't want to tee the output produced while the jobs are running because with multiple jobs running at once the logging would be interspersed. Get-Job | Receive-Job prints the output in a nice organized manner.
I have tried all of the following and no output is written to the file or stored in a variable, it just goes to the screen:
#Wait for last job to complete
While (Get-Job -State "Running") {
Log "Running..." -v $info
Start-Sleep 10
}
Log ("Jobs Completed. Output:") -v $info
# Getting the information back from the jobs
foreach($job in Get-Job){
Receive-Job -Job $job | Out-File c:\Test.log -Append
$output = Receive-Job -Job $job
Log ("OUTPUT: "+$output)
Receive-Job -Job $job -OutVariable $foo
Log ("FOO: "+$foo)
}
EDIT:
I have removed the extra Receive-Job calls in the foreach to the following after seeing Keith's comment:
# Getting the information back from the jobs
foreach($job in Get-Job){
Receive-Job -Job $job -OutVariable temp
Write-Host ("Temp: "+$temp)
$temp | Out-File -FilePath c:\Test.log -Append
}
I also verified I'm not using Receive-Job anywhere else in the script. The write-host $temp and the out-file still produce no output though.
If the job uses Write-Host to produce output, Receive-Job returns $null, but the results get written to the host. However, if the job uses Write-Output to produce output in lieu of Write-Host, Receive-Job returns a string array [string[]] of the job output.
To demonstrate, enter this innocuous code at the PowerShell prompt:
$job = Start-Job -ScriptBlock {
[int] $counter = 0
while ($counter -lt 10) {
Write-Output "Counter = $counter."
Start-Sleep -Seconds 5
$counter++
}
}
Wait about 20-30 seconds for the job to produce some output, then enter this code:
$result = Receive-Job -Job $job
$result.Count
$result
$result | Get-Member
The $result object contains the strings produced by the job.
NOTE: After some experimentation I have found that using write-output or just outputting a variable directly is not a good solution either. If you have a function that uses either of those methods and also returns a value the output will be concatenated with the return value!
The best way I have found to log output from jobs is to use write-host in combination with the Start-Transcript and Stop-Transcript cmdlets. There are some disadvantages such as the ISE does not support Transcript cmdlets. Also I haven't found a way to output to the transcript without it going to the host (I wanted a robust log with a less verbose user experience), but it seems to be the best solution available currently. It is the only way I've been able to log concurrent jobs without interspersing the output without having to right multiple temporary log files for each job and then combining them.
Below was my previous answer that I left just for documentation reasons:
The issue here was not getting output from Receive-Job. The output from the jobs I was expecting to see is being written in the job by write-host. When receive-job was called I saw all my output on the screen, but nothing was available to pipe elsewhere. It seems when I call receive-job all those write-host cmdlets execute, but there's nothing for me to pick up and pipe to out-file. The following script demonstrates this. You'll notice "Sleepy Time" appears in both the log file and the host, while "Slept" only appears in the host. So, rather than trying to pipe Receive-Job to write-host, I need to modify my Log function to not use write-host or to split the output as shown below.
cls
rm c:\jobs.log
$foo = #(1,2,3)
$jobs = #()
foreach($num in $foo){
$s = { get-process -name "explorer"
Start-Sleep $args[0]
$x = ("Sleepy Time: "+$args[0])
write-host $x
$x
write-host ("Slept: "+$args[0])
}
$id = [System.Guid]::NewGuid()
$jobs += $id
Start-Job -Name $id -ScriptBlock $s -args $num
}
While (Get-Job -State "Running") {
cls
Get-Job
Start-Sleep 1
}
cls
Get-Job
write-host "Jobs completed, getting output"
foreach($job in $jobs){
Receive-Job -Name $job | out-file c:/jobs.log -append
}
Remove-Job *
notepad c:\jobs.log
Below is a sample that will run concurrent jobs and then log them sequentially as per mbourgon's request. You'll notice the timestamps indicate the work was interspersed, but the logging is displayed sequentially by job.
#This is an example of logging concurrent jobs sequentially
#It must be run from the powershell prompt as ISE does not support transcripts
cls
$logPath = "c:\jobs.log"
rm $logPath
Start-Transcript -Path $logPath -Append
#Define some stuff
$foo = #(0,1,2)
$bar = #(#(1,"AAA"),#(2,"BBB"),#(3,"CCC"))
$func = {
function DoWork1 {
Log "This is DoWork1"
return 0
}
function DoWork2 {
Log "This is DoWork2"
return 0
}
function Log {
Param([parameter(ValueFromPipeline=$true)]$InputObject)
$nl = [Environment]::NewLine.Chars(0)
$time = Get-Date -Format {HH:mm:ss}
Write-Host ($time+">"+$InputObject+$nl)
}
}
#Start here
foreach($num in $foo){
$s = { $num = $args[0]
$bar = $args[1]
$logPath = $args[2]
Log ("Num: "+$num)
for($i=0; $i -lt 5; $i++){
$out = ([string]::$i+$bar[$num][1])
Log $out
start-sleep 1
}
Start-Sleep $num
$x = DoWork1
Log ("The value of x is: "+$x)
$y = DoWork2
Log ("The value of y is: "+$y)
}
Start-Job -InitializationScript $func -ScriptBlock $s -args $num, $bar, $logPath
}
While (Get-Job -State "Running") {
cls
Get-Job
Start-Sleep 1
}
cls
Get-Job
write-host "Jobs completed, getting output"
Get-Job | Receive-Job
Remove-Job *
Stop-Transcript
notepad $logPath
I found soulution how to receive Write-Host results to variable with Receive-Job CMDlet, it's working on Powershell 5.1:
$var = Receive-Job $job 6>&1
I know this is old. I don't use jobs very often so I was obviously having the same problem when I stumbled on this thread. Anyway, I arrived at a different solution so I thought I'd throw it out there.
Also in PowerShell 5.1 I simply referenced the properties on the Job Object(s) In my case this looked like:
$Jobs.ChildJobs.Output | Out-File <LogFile> -Append
It looks like all the streams are recorded in this way. Simple echoes go to the Output, presumable the success stream. Write-Host commands go to Information Stream.
Output : {}
Error : {}
Progress : {}
Verbose : {}
Debug : {}
Warning : {}
Information : {}
When you receive the results of a job those results are deleted such that further calls to Receive-Job will have nothing to retrieve (assuming the job is completed). If you don't want Receive-Job to delete the results use the -Keep switch. Now this doesn't explain why your first call to Receive-Job isn't outputting anything to the log file ... unless there is a part of the script you're not showing that is doing a Receive-Job before this.
BTW when you use -OutVariable supply it what a variable name and not a variable e.g. -ov temp. Then you later access that info with $temp.
You could pipe your Get-Process to Format-Table, so you can see all of the output.
Get-Process -Name "explorer" | format-Table -hidetableheaders -AutoSize
And you could use something like this at the end of the script...
Get-job | Receive-Job 2>&1 >> c:\path\to\your\log\file.log
The "2>&1 >>" with grab all output from Get-Job | Receive-Job
A simple solution to the problem. Write-Verbose "blabla" -Verbose
$s = New-PSSession -ComputerName "Computer"
$j = Invoke-Command -Session $s -ScriptBlock { Write-Verbose "Message" -Verbose } -AsJob
Wait-Job $j
$Child = $j.ChildJobs[0]
$Child.Verbose | out-file d:/job.log -append
What I have done is to not call Receive-Job at all. But create a wrapper scriptblock that starts and stops a transcripts. Then in that wrapper script block I call the jobs script block like
&$ScriptBlock < Parameters >
So the transcript will capture all the output to a file.