We have a PowerShell build script (Cake bootstrapper) that occasionally fails to download/restore nuget packages. How would I create a generic .ps1 script that can be used to make calls to another .ps1 script, and retry if failed and the output contained a specific string (I don't want to retry if it's a build error)?
Possible example usage:
.\ExecuteWithRetry.ps1 -Script "Build.ps1 -target Restore" -OutputTrigger "RemoteException"
Created a script that does the trick.
Example usage:
./Execute-With-Retry.ps1 -RetryDelay 1 -MaxRetries 2 { & ./Build.ps1 -target Restore } -RetryFilter RemoteException
Based on this gist from Alex Bevilacqua.
<#
This script can be used to pass a ScriptBlock (closure) to be executed and returned.
The operation retried a few times on failure, and if the maximum threshold is surpassed, the operation fails completely.
Params:
Command - The ScriptBlock to be executed
RetryDelay - Number (in seconds) to wait between retries
(default: 5)
MaxRetries - Number of times to retry before accepting failure
(default: 5)
VerboseOutput - More info about internal processing
(default: false)
RetyFilter - Only retry if retry filter value is contained in the command output
Examples:
./Execute-With-Retry.ps1 -RetryDelay 1 -MaxRetries 2 { & ./Build.ps1 -target Restore } -RetryFilter RemoteException
#>
param(
[Parameter(ValueFromPipeline, Mandatory)]
$Command,
$RetryDelay = 5,
$MaxRetries = 5,
$VerboseOutput = $false,
$RetryFilter
)
$currentRetry = 0
$success = $false
$cmd = $Command.ToString()
do {
try {
$result = & $Command
$success = $true
if ($VerboseOutput -eq $true) {
$Host.UI.WriteDebugLine("Successfully executed [$cmd]")
}
return $result
}
catch [System.Exception] {
$currentRetry = $currentRetry + 1
$exMessage = $_.Exception.Message;
if ($RetryFilter -AND !$exMessage.Contains($RetryFilter)) {
throw $exMessage
}
if ($VerboseOutput -eq $true) {
$Host.UI.WriteErrorLine("Failed to execute [$cmd]: " + $_.Exception.Message)
}
if ($currentRetry -gt $MaxRetries) {
throw "Could not execute [$cmd]. The error: " + $_.Exception.ToString()
}
else {
if ($VerboseOutput -eq $true) {
$Host.UI.WriteDebugLine("Waiting $RetryDelay second(s) before attempt #$currentRetry of [$cmd]")
}
Start-Sleep -s $RetryDelay
}
}
} while (!$success);
Related
Highly influenced by other questions here on Stackoverflow I have ended up with this method for starting processes from my Powershell-scripts
function global:system-diagnostics-processstartinfo {
[CmdletBinding(SupportsShouldProcess=$True,ConfirmImpact='Low')]
param
(
[Parameter(Mandatory=$True,HelpMessage='Full path to exectuable')]
[Alias('exectuable')]
[string]$exe,
[Parameter(Mandatory=$True,HelpMessage='All arguments to be sent to exectuable')]
[Alias('args')]
[string]$arguments
)
if (!(Test-Path $exe)) {
$log.errorFormat("Did not find exectuable={0}, aborting script", $exe)
exit 1
}
$log.infoFormat("Start exectuable={0} with arguments='{1}'", $exe, $arguments)
$processStartInfo = New-Object System.Diagnostics.ProcessStartInfo($exe)
$processStartInfo.FileName = $exe
$processStartInfo.RedirectStandardError = $true
$processStartInfo.RedirectStandardOutput = $true
$processStartInfo.UseShellExecute = $false
$processStartInfo.Arguments = $arguments
$p = New-Object System.Diagnostics.Process
$p.StartInfo = $processStartInfo
$log.info("Start exectuable and wait for exit")
$p.Start() | Out-Null
#$p.WaitForExit()
$stdout = $p.StandardOutput.ReadToEnd()
$stderr = $p.StandardError.ReadToEnd()
$log.infoFormat("exectuable={0} stdout: {1}", $exe, $stdout)
$log.debugFormat("exectuable={0} stderr: {1}", $exe,$stderr)
$global:ExitCode = $p.ExitCode
$log.debugFormat("exectuable={0} Exitcode: {1}", $exe, $p.ExitCode)
return $stdout
}
Pretty straight forward with some added logging etc. And it works in all my current use cases execpt one. I have created a script that copies the database dump for our production instance of Confluence to our test server. Then it uses the above method to drop existing database, all fine. But the actual restore just hangs for ever and ever. So right now I have to exit the script and then run the following command manually
d:\postgresql\bin\pg_restore.exe -U postgres -d confluencedb -v -1 d:\temp\latest-backup.pgdump
It takes some time and there is quite a lot of output. Which makes me belive that there must be either one the following causing the issue
The amount of output makes a buffer overflow and stalls the script
It takes to much time
Anyone with similar experiences who can help me resolve this. It would enable to schedule the import, not having to do it manually as today.
I had to do the following right after process. Start:
# Capture output during process execution so we don't hang
# if there is too much output.
do
{
if (!$process.StandardOutput.EndOfStream)
{
[void]$StdOut.AppendLine($process.StandardOutput.ReadLine())
}
if (!$process.StandardError.EndOfStream)
{
[void]$StdErr.AppendLine($process.StandardError.ReadLine())
}
Start-Sleep -Milliseconds 10
}
while (!$process.HasExited)
# Capture any standard output generated between our last poll and process end.
while (!$process.StandardOutput.EndOfStream)
{
[void]$StdOut.AppendLine($process.StandardOutput.ReadLine())
}
# Capture any error output generated between our last poll and process end.
while (!$process.StandardError.EndOfStream)
{
[void]$StdErr.AppendLine($process.StandardError.ReadLine())
}
# Wait for the process to exit.
$process.WaitForExit()
LogWriteFunc ("END process: " + $ProcessName)
if ($process.ExitCode -ne 0)
{
LogWriteFunc ("Error: Script execution failed: " + $process.ExitCode )
$FuncResult = 1
}
# Log and display any standard output.
if ($StdOut.Length -gt 0)
{
LogWriteFunc ($StdOut.ToString())
}
# Log and display any error output.
if ($StdErr.Length -gt 0)
{
LogWriteFunc ($StdErr.ToString())
}
I have to execute this Exchange command:
$command="Disable-Remotemailbox -Identitiy x.y#corp.com -Confirm:$false -Archive"
...and need to check if it was successfully executed.
try {
$result = Invoke-Expression $command
$success = 1
catch {
$success = 0
}
sf
$result = Invoke-Expression $command
if ($?) {
$success = 1
} else {
$success = 0
}
However, this is not working like expected. It returns 1 anyways.
It seems that it only shows if the Invoke-Expression command was successful. Which is all the time.
How to archive this?
I have a few functions that get called either from Jenkins as part of a pipeline, they also get called from a pester test or lastly they can get called from the powershell console. The issue I have really stems from Jenkins not seeming to handle write-output in the way I think it should.
So what I am doing is creating a Boolean param that will allow my to choose if I terminate my function with a exit code or a return message. The exit code will be used by my pipeline logic and the return message for the rest ?
Is there a alternate approach I should be using this seems to be a bit of a hack.
function Get-ServerPowerState
{
[CmdletBinding()]
param
(
[string[]]$ilo_ip,
[ValidateSet('ON', 'OFF')]
[string]$Status,
[boolean]$fail
)
BEGIN
{
$here = Split-Path -Parent $Script:MyInvocation.MyCommand.Path
$Credentials = IMPORT-CLIXML "$($here)\Lib\iLOCred.xml"
}
PROCESS
{
foreach ($ip in $ilo_ip)
{
New-LogEntry -Message ("Getting current powerstate " + $ip)
If (Test-Connection -ComputerName $ip.ToString() -Count 1 -Quiet)
{
$hostPower = Get-HPiLOhostpower -Server $ip -Credential
$Credentials -DisableCertificateAuthentication
}
}
}
END
{
If($fail){
New-LogEntry -Message "Script been set to fail with exit code" -Log Verbose
New-LogEntry -Message "The host is powered - $($HostPower.Host_Power)" -Log Verbose
If($hostPower.HOST_POWER -match $Status)
{
Exit 0
}
else {
Exit 1
}
}
else {
New-LogEntry -Message "Script been set to NOT fail with exit code" -Log Verbose
New-LogEntry -Message "The host is powered - $($HostPower.Host_Power)" -Log Verbose
If($hostPower.HOST_POWER -match $Status)
{
return 0
}
else {
return 1
}
}
}
}
Like this
function Get-Output {
param ([switch]$asint)
if ($asint) {
return 1
}
else {
write-output 'one'
}
}
Get-Output
Get-Output -asint
If you intend to use the output in the pipeline then use Write-Output. If you intend to only send it to the host process then use Write-Host. I typically use the return keyword if I want to assign a return value to a variable.
[int]$result = Get-Output -asint
I am creating a retry command in powershell to invoke some script block with retry logic:
function Invoke-CommandWithRetry
{
[CmdletBinding()]
Param
(
[Parameter(Mandatory = $true, ValueFromPipeline = $true)]
[ScriptBlock]$Command,
[Parameter(Mandatory = $false)]
[ValidateRange(0, [UInt32]::MaxValue)]
[UInt32]$Retry = 3,
[Parameter(Mandatory = $false, Position = 3)]
[ValidateRange(0, [UInt32]::MaxValue)]
[UInt32]$DelayInMs = 1000
)
process {
$retryCount = 0
while( $true )
{
try
{
return (Invoke-Command -ScriptBlock $Command)
}
catch
{
if( $retryCount -ge $Retry )
{
throw
}
else
{
Write-Warning $_.Exception
Start-Sleep -Milliseconds $DelayInMs
$retryCount++
Write-Warning "Retry command $retryCount time: $Command"
}
}
}
}
}
The command should be straightforward, try to run a script block and if exception found, re-run it.
But I found some issue with this function when the sub pipeline throw exception, here is my test code:
{
Write-Host "Start";
1,2,3
} | Invoke-CommandWithRetry | ForEach-Object {
$_
throw "Error"
}
Write-Host "End"
I would expect that, because of exception, so 2,3 object will be skipped.
The result is just as I expected, but the retry is working not only for the first pipeline script block, but seems also involve the following pipeline as the body of first pipeline, which is really weird for me, here is the result:
Start
1
WARNING: System.Management.Automation.RuntimeException: Error
WARNING: Retry command 1 time:
Write-Host "Start";
1,2,3
Start
1
WARNING: System.Management.Automation.RuntimeException: Error
WARNING: Retry command 2 time:
Write-Host "Start";
1,2,3
Start
1
WARNING: System.Management.Automation.RuntimeException: Error
WARNING: Retry command 3 time:
Write-Host "Start";
1,2,3
Start
1
Error
At line:7 char:9
+ throw "Error"
+ ~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (Error:String) [], RuntimeException
+ FullyQualifiedErrorId : Error
Anyone can help to explain why this happened? and How to avoid the first script block retry when sub pipeline throw exception?
I'm using PowersHell to automate iTunes but find the error handling / waiting for com objects handling to be less than optimal.
Example code
#Cause an RPC error
$iTunes = New-Object -ComObject iTunes.Application
$LibrarySource = $iTunes.LibrarySource
# Get "playlist" objects for main sections
foreach ($PList in $LibrarySource.Playlists)
{
if($Plist.name -eq "Library") {
$Library = $Plist
}
}
do {
write-host -ForegroundColor Green "Running a loop"
foreach ($Track in $Library.Tracks)
{
foreach ($FoundTrack in $Library.search("$Track.name", 5)) {
# do nothing... we don't care...
write-host "." -nonewline
}
}
} while(1)
#END
Go into itunes and do something that makes it pop up a message - in my case I go into the Party Shuffle and I get a banner "Party shuffle automatically blah blah...." with a "Do not display" message.
At this point if running the script will do this repeatedly:
+ foreach ($FoundTrack in $Library.search( <<<< "$Track.name", 5)) {
Exception calling "Search" with "2" argument(s): "The message filter indicated
that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVER
CALL_RETRYLATER))"
At C:\Documents and Settings\Me\My Documents\example.ps1:17 char:45
+ foreach ($FoundTrack in $Library.search( <<<< "$Track.name", 5)) {
Exception calling "Search" with "2" argument(s): "The message filter indicated
that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVER
CALL_RETRYLATER))"
At C:\Documents and Settings\Me\My Documents\example.ps1:17 char:45
If you waited until you you had a dialog box before running the example then instead you'll get this repeatedly:
Running a loop
You cannot call a method on a null-valued expression.
At C:\Documents and Settings\Me\example.ps1:17 char:45
+ foreach ($FoundTrack in $Library.search( <<<< "$Track.name", 5)) {
That'll be because the $Library handle is invalid.
If my example was doing something important - like converting tracks and then deleting the old ones, not handling the error correctly could be fatal to tracks in itunes.
I want to harden up the code so that it handles iTunes being busy and will silently retry until it has success. Any suggestions?
Here's a function to retry operations, pausing in between failures:
function retry( [scriptblock]$action, [int]$wait=2, [int]$maxRetries=100 ) {
$results = $null
$currentRetry = 0
$success = $false
while( -not $success ) {
trap {
# Set status variables at function scope.
Set-Variable -scope 1 success $false
Set-Variable -scope 1 currentRetry ($currentRetry + 1)
if( $currentRetry -gt $maxRetries ) { break }
if( $wait ) { Start-Sleep $wait }
continue
}
$success = $true
$results = . $action
}
return $results
}
For the first error in your example, you could change the inner foreach loop like this:
$FoundTracks = retry { $Library.search( "$Track.name", 5 ) }
foreach ($FoundTrack in $FoundTracks) { ... }
This uses the default values for $wait and $maxRetries, so it will attempt to call $Library.search 100 times, waiting 2 seconds between each try. If all retries fail, the last error will propagate to the outer scope. You can set $ErrorActionPreference to Stop to prevent the script from executing any further statements.
COM support in PowerShell is not 100% reliable. But I think the real issue is iTunes itself. The application and COM model wasn't designed, IMO, for this type of management. That said, you could implement a Trap into your script. If an exception is raised, you could have the script sleep for a few seconds.
Part of your problem might be in how $Track.name is being evaluated. You could try forcing it to fully evaluate the name by using $($Track.name).
One other thing you might try is using the -strict parameter with your new-object command/