I am starting several jobs (with Start-Job) and at the end of my script i do a check to see if the jobs have been running more than X seconds. I would then like to take the Running and Failed jobs and restart them until they succeed.
The jobs are named after the server that I'd like to run against (with for example Test-Connection).
My problem is that I can't figure out how to re-submit the jobs!
get-job | where { $_.state -eq "running" } | remove-job -force | start-job -ScriptBlock { echo $_ }
How can I pipe the Name of the failed/hanged job(s) to the new job I am starting?
How can i wait for the remove-job to finish before I continue on Start-Job?
Kind regards:)
One way to restart failed jobs:
Start-Job -Name Foo -ScriptBlock {
$ErrorActionPreference = 'Stop'
Get-Item C:\DoesNotExists
} | Wait-Job > $null
Get-Job | ? { $_.State -eq 'Failed' } | % {
Start-Job -Name $_.Name -ScriptBlock { iex $args[0] } -ArgumentList $_.Command | Wait-Job | Receive-Job
}
Related
I have a small script that runs through all share paths within a csv file and invokes a quick Get-ChildItem to count the number of files within the folder/subfolders etc. This is working but it is taking an age, especially if a folder have more than 500k files.
I would like to add a timeout within the loop, so that it will just write-host and then output into another column 'TIMED OUT' but i can't seem to get it to work.
I have tried a few different ways but somehow my loop breaks and runs the top line in my csv over and over.
This is the code i have so far:
...
#Import middle to filter unique
$sharesMiddle = Import-Csv -Path './middle.csv' | Sort-Object NFTSPath -Unique
$result = foreach ($share in $sharesMiddle) {
#Replace the colon for $ within the path
$nftsPath = join-path \\ $share.AssetName $share.Path.Replace(':', '$')
$path = $share.Path
$count = Invoke-Command -computername $share.AssetName -ScriptBlock { param($path) (Get-ChildItem $path -File -Recurse).count } -Credential $Cred -Verbose -ArgumentList $path
Write-Host $nftsPath : $count
$share | Select-Object *, #{n = "Files"; e = { $count } }
}
$result | Export-CSV '.\newcsvfile.csv' -NoTypeInformation
Any help on this would be super!
Thanks.
I tested the following which should be suitable for you:
# define the time limit in seconds
$TimeoutLimitSeconds = 2
# Define the job, start job
$job = Start-Job -ScriptBlock {420+69 ; start-sleep -Seconds 3}
# Await job
$job | Wait-Job -Timeout ( $TimeoutLimitSeconds ) | Out-Null
# If job doesn't finish before the time limit is reached
if ($job.state -eq 'Running') {
# Job timed out
$job | Stop-Job | Remove-Job
$job = $null
$output = "TIMED OUT"
# If job managed to finalize in time
}Else{
# Finished on time
$job | Stop-Job
$output = Receive-Job $job
$job | Remove-Job
}
# Write the output to host
Write-Host "output is: $output"
output is: TIMED OUT
# define the time limit in seconds
$TimeoutLimitSeconds = 4
# Define the job, start job
$job = Start-Job -ScriptBlock {420+69 ; start-sleep -Seconds 3}
# Await job
$job | Wait-Job -Timeout ( $TimeoutLimitSeconds ) | Out-Null
# If job doesn't finish before the time limit is reached
if ($job.state -eq 'Running') {
# Job timed out
$job | Stop-Job | Remove-Job
$job = $null
$output = "TIMED OUT"
# If job managed to finalize in time
}Else{
# Finished on time
$job | Stop-Job
$output = Receive-Job $job
$job | Remove-Job
}
# Write the output to host
Write-Host "output is: $output"
output is: 489
I have a relatively long/complex script that blindly runs a batch of commands against several devices (as jobs). Once in awhile, a couple of these jobs continue to run indefinitely. I’ve added a Wait-Job -Timeout command to my script (see below) in order to force-stop jobs that are taking too long to run.
I’d like to change the output for these hung jobs to read “This device is busy or unstable”. How can I do this? I'm guessing I need to add something to the tail-end of the pipeline (in the last line of code below).
$Jobs += Get-Job
$jobs | Wait-Job -Timeout 5 | out-null
Get-Job | ? {$_.State -eq 'Running'} | Stop-Job -PassThru
One way is to iterate over the jobs that are currently running and write your message for each one:
$Jobs += Get-Job
$Jobs | Wait-Job -Timeout 5 | Out-Null
Get-Job | ? { $_.State -eq 'Running' } | Stop-Job -PassThru | % { Write-Host "This device is busy or unstable" }
You can also add info from the jobs that are being stopped, like the job ID for example:
Get-Job | ? { $_.State -eq 'Running' } | Stop-Job -PassThru | % { Write-Host "This device is busy or unstable: $($_.Id)" }
UPDATE: You can use a hashtable to store the job IDs that were "force stopped". Then iterate through the Jobs using Receive-Job to get the output, and check if the job is in the ForceStoppedIds table. If it is write your custom message, otherwise just output the message. Here's a simple test I ran.
Start-Job -ScriptBlock { Write-Output "Starting 1."; Start-Sleep -Seconds 3; Write-Output "1 complete."; } | Out-Null
Start-Job -ScriptBlock { Write-Output "Starting 2."; Start-Sleep -Seconds 60; Write-Output "2 complete."; } | Out-Null
Start-Job -ScriptBlock { Write-Output "Starting 3."; Start-Sleep -Seconds 2; Write-Output "3 complete."; } | Out-Null
$Jobs += Get-Job
$Jobs | Wait-Job -Timeout 5 | Out-Null
$ForceStoppedIds = #{}
$Jobs | ? { $_.State -eq 'Running' } | Stop-Job -PassThru | % { $ForceStoppedIds[$_.Id] = $true }
foreach ($job in $Jobs) {
$jobOutput = Receive-Job $job -Wait
if ($ForceStoppedIds.Contains($job.Id)) {
Write-Host "Custom message about stopped job: $($jobOutput)"
}
else {
Write-Host $jobOutput
}
}
One thing to be cautious of is how jobs output information (ie. Write-Host, Write-Output, return, etc.). If you're not getting the results you expect, double check the job's ScriptBlock to see how the information is being written/returned/etc. I'm sure there are much more elegant ways of doing this, but hopefully this will help.
Im looking for a way in powershell to run a script block, or some code after an exception or a terminating write-error has occurred.
Basically, if I hit a throw, or a write-error, then I want the highest scope (main) to close out all PSSessions and do a get-job | remove-job before we stop execution.
Im not sure exactly if this is possible if I have set $ErrorActionPreference = "Stop"
This is what I came up with while waiting on an answer.. but Ill keep this question up incase someone has a better solution:
#Wait on feedback from children
try{
While (Get-Job -State "Running")
{
Get-Job | Receive-Job
Start-Sleep 2
}
Get-Job | Receive-Job
} finally {
Get-PSSession | Remove-PSSession
Get-Job | Remove-Job
}
thanks!
Jobs will not send an error to the host by default, you'll need to throw an error manually.
#Wait on feedback from children
try{
While (Get-Job -State "Running")
{
if (Get-Job -State 'Failed'){ throw 'Job failed.' }
Get-Job | Receive-Job
Start-Sleep 2
}
Get-Job | Receive-Job
} finally {
Get-PSSession | Remove-PSSession
Get-Job | Remove-Job
}
You may get more information about the failure by checking the JobState property on the job object.
I have a script which starts a job Start-Job which monitors a process. Whenever this process dies, the script will automatically restart the process, and then restart the script.
If the script can't restart the process it will send an e-mail and should exit PowerShell completely, to avoid the restart of the script. I did this by just exiting inside the job, but that doesn't seem to work.
How can I tell my script to not restart, if the process could not be restarted?
Here's my code:
$sb = {
while ((Get-Process LogikWebserver).Responding) { sleep -m 50 }
if (!(Get-Process LogikWebserver).Responding) {
#restart process
try {
Start-Process "$processpath\LogikWebserver.exe" -EA Stop
sleep -s 2
if (Get-Process LogikWebserver -EA Stop) {
Send-MailMessage
}
} catch {
Send-MailMessage
<# With this exit I want to tell the script to exit completely #>
exit
}
}
}
# start and get job
Start-Job -Name LogikWebserverWatch -ScriptBlock $sb
Wait-Job -Name LogikWebserverWatch
Get-Job -Name LogikWebserverWatch | Remove-Job -Force
# restart script
$skript = "{0}\{1}" -f $PSScriptRoot, "Watch-LogikWebserver.ps1"
& $skript
Don't catch exceptions inside the scriptblock. Receive the job output and catch exceptions there. Also, use an infinite loop for re-running the job rather than re-running the script.
$sb = {
while ((Get-Process LogikWebserver).Responding) {
Start-Sleep -Milliseconds 50
}
# restart process (at this point .Responding was already false, thus no
# point in checking again)
Start-Process "$processpath\LogikWebserver.exe" -ErrorAction Stop
Start-Sleep -Seconds 2
if (Get-Process LogikWebserver -ErrorAction Stop) {
Send-MailMessage
}
}
while ($true) {
# start and get job
try {
Start-Job -Name LogikWebserverWatch -ScriptBlock $sb |
Wait-Job |
Receive-Job -ErrorAction Stop
} catch {
Send-MailMessage
exit
} finally {
Get-Job -Name LogikWebserverWatch | Remove-Job -Force
}
}
I have written a ScriptBlock that POSTs file uploads to a custom http server and saves the response to disk. It is intended to be used for testing said server and in addition to checking the correct response I'd like to run it in parallel to load the server.
The code in the ScriptBlock was built as separate ps1 script and works. I have then transplanted it into a framework-script using Start-Job for managing the jobs and eventually presenting overall results.
Unfortunately it does not work after that transplantation.
The symptom is that using Start-Job the job never finishes. For testing I have switched to Invoke-Command. A simple Invoke-Command works, InvokeCommand -AsJob does not.
Works:
Invoke-Command -Scriptblock $ScriptBlock -ArgumentList $Name,"C:\Projects\DCA\CCIT\ServiceBroker4\Java\eclipse-workspace\XFAWorker\testdata","P7-T"
Do not work:
Invoke-Command -AsJob -Computer localhost -Scriptblock $ScriptBlock -ArgumentList $Name,"C:\Projects\DCA\CCIT\ServiceBroker4\Java\eclipse-workspace\XFAWorker\testdata","P7-T"
Start-Job -Name $Name -ScriptBlock $Scriptblock -ArgumentList $Name,"C:\Projects\DCA\CCIT\ServiceBroker4\Java\eclipse-workspace\XFAWorker\testdata","P7-T"
The ScriptBlock being used is rather longish. It starts off by declaring Parameters:
$Scriptblock = {
Param (
[string]$JobName,
[string]$path,
[string]$BASEJOBNAME
)
And further down uses HttpWebRequest and a few other .NET classes:
$url = "http://localhost:8081/fillandflatten"
[System.Net.HttpWebRequest] $req = [System.Net.WebRequest]::create($url)
...
$xfabuffer = [System.IO.File]::ReadAllBytes("$path\$BASEJOBNAME.xml")
...
$header = "--$boundary`r`nContent-Disposition: form-data; name=`"xfa`"; filename=`"xfa`"`r`nContent-Type: text/xml`r`n`r`n"
$buffer = [Text.Encoding]::ascii.getbytes($header)
...
[System.Net.httpWebResponse] $res = $req.getResponse()
As described when using Start-Job or Invoke-Command -AsJob the child job started simply remains in Running state forever.
What can cause this behaviour? What can be used to debug it? If there is some error can I force the child-job to terminate and tell me what it does not like?
I am using PowerShell 2.0 on Windows XP.
In the framework I came up with I do the Start-Job (currently just one of them, but I plan to ramp up that number for stress-testing). Then I have a loop waiting for them all to terminate:
do {
$Jobs = #(Get-Job | Where { $_.State -eq "Running" -and $_.Name.StartsWith($BASEJOBNAME) });
$n = $Jobs.Count
if ($n -gt 0)
{
Log -message "Waiting for $n jobs to finish..."
Get-Job | Where { $_.Name.StartsWith($BASEJOBNAME) } | Format-Table -AutoSize -Property Id,Name,State,HasMoreData,Location
start-Sleep -Seconds 3
}
} until ($n -eq 0)
This produces output of the form:
2014-08-01 18:58:52 - Waiting for 1 jobs to finish...
Id Name State HasMoreData Location
-- ---- ----- ----------- --------
2 XFA001 Running True localhost
Forever.
A full minimal test-case is:
# Code for the Jobs:
$Scriptblock = {
[System.Net.HttpWebRequest] $req = [System.Net.WebRequest]::create("http://locahost/index.html")
return "done"
}
# Start a job. Three variants, The Job-based ones hang.
# Invoke-Command -Scriptblock $ScriptBlock
#Invoke-Command -AsJob -Computer localhost -Scriptblock $ScriptBlock
$Job = Start-Job -Name $Name -ScriptBlock $Scriptblock
## The rest of the code is only applicable for the Start-Job-Variant
# Wait for all Jobs to finish
do {
$Jobs = #(Get-Job | Where { $_.State -eq "Running" });
$n = $Jobs.Count
if ($n -gt 0)
{
Write-Host "Waiting for $n jobs to finish..."
Get-Job | Format-Table -AutoSize -Property Id,Name,State,HasMoreData
Start-Sleep -Seconds 3
}
} until ($n -eq 0)
# Get output from all jobs
$Data = ForEach ($Job in (#(Get-Job | Where { $_.Name.StartsWith($BASEJOBNAME) } ))) {
Receive-Job $Job
}
# Clean out all jobs
ForEach ($Job in (#(Get-Job | Where { $_.Name.StartsWith($BASEJOBNAME) } ))) {
Remove-Job $Job
}
# Dump output
Write-Host "Output data:"
$Data | Format-Table
Write-Host ""
This hangs for me. If I comment out the line creating the WebRequest object it works.
Thank you.
When you run Get-Job , does the job's "HasMoreData" properties is "True" ?
If yes, check the output of the job :
Receive-Job <JobName or JobID> -Keep