PowerShell : Retry logic in Scripts

One of my projects required me to copy a CSV file (important step) to a VM running on Server 2012 R2.
I’ve found this excellent tip by Ravi on using Copy-VMfile cmdlets in Server 2012 R2 Hyper-V. To use this cmdlet, I had to enable "Guest Service Interface" component in the Integration Services (below is what documentation says about the service).

This new component in the Integration Services allows copying files to a running VM without any network connection (How cool is that?).

The tip mentioned earlier talks about how to enable the component using Enable-VMIntegrationService, but there is a delay between enabling the component and successfully using the Copy-VMfile cmdlet. 

So how do I go about making sure that the service is running before the cmdlet is issued, or keep retrying the cmdlet until it succeeds ?

Simplest way would be to use Start-Sleep to induce a delay and take a guess that the service would be running by the time cmdlet executes, like done in below function definition:

001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037

Function Copy-ImportantFileToVM {     [CmdletBinding()]     param($VMName)         $VM = Get-VM -Name $VMName         #Check if Guest Integration Service is enabled         $GuestService = $VM.VMIntegrationService.Where({$_.Name -eq 'Guest Service Interface'})         if (-not $GuestService.Enabled) {             #Enable the GSI             $GuestServiceStatus = $VM | Get-VMIntegrationService -Name "Guest Service Interface" | Enable-VMIntegrationService -Passthru             if (-not $GuestServiceStatus.Enabled) {                 throw "Couldn't enable Guest Service Interface"             }         }             # Induce sleep in the script for 120 seconds just to be sure         Start-Sleep -Seconds 120         # Critical Step -> Copy test CSV to VM         if (Test-Path -Path "$PSScriptRoot\test.csv") {            TRY {                 $CopyFileHash = @{                     Name=$VMName;                     SourcePath="$PSScriptRoot\test.csv";                     DestinationPath='C:\temp\test.csv';                     FileSource='Host';                     CreateFullPath=$true;                     Force = $true;                     Verbose=$true;                     ErrorAction='Stop';                 }                 Copy-VMFile  @CopyFileHash                 }             CATCH {                 # Put error handling here - maybe log it                 $PSCmdlet.ThrowTerminatingError($PSItem)             }         } # end if }

That brings to another question-- what if the delay put is not enough or is it too much?

Similarly, a more practical use case is for the Azure cmdlets which make the REST API calls behind the scenes, what if while calling one of the REST endpoint the network fluctuated and the cmdlet failed at a critical step. 

Bottom line is I am really looking for a retry logic within my scripts, so that my code retries an important step for few times before it dies out.

While researching around the topic found this article by Pawel & this article by Alex on retry logic in PowerShell. Using these as a reference, I wrote the below function named Invoke-ScriptBlockWithRetry

001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084

function Invoke-ScriptBlockWithRetry { <# .Synopsis    Invokes a script block with resiliency. .DESCRIPTION    The function takes a script block as a mandatory argument and tries to run it certain number of times (argument to -MaxRetries).    It delays execution between subsequent[AN1] [DD2] retries by 10 seconds (default), can be passed a custom value to –RetryDelay parameter. .EXAMPLE    First create a script block with -ErrorAction set to Stop and then pass it to the function    PS> $CopyLambda = {Copy-Item -Path \\fileserver\Info\test.csv -Destination C:\Temp -ErrorAction Stop}    PS> Invoke-ScriptBlockWithRetry -Command $CopyLambda1 -MaxRetries 5 -Verbose .EXAMPLE    Script blocks have access to the current scope variables, so if you set a variable in the current scope, you can use that within the script block    PS> $name = 'notepad'    PS>Invoke-ScriptBlockWithRetry -Command {Get-Process -Name $name -EA Stop} -MaxRetries 5 -Verbose .NOTES    Credits    Inspired by -         1. http://www.pabich.eu/2010/06/generic-retry-logic-in-powershell.html         2. http://www.alexbevi.com/blog/2015/02/06/block-retry-using-powershell/ #>     [CmdletBinding()]     [OutputType([PSObject])]     Param     (         # Param1 help description         [Parameter(Mandatory=$true,                    ValueFromPipelineByPropertyName=$true,                    Position=0)]         [System.Management.Automation.ScriptBlock]         $ScriptBlock,         # Number of retries. Default is 10.         [Parameter(Position=1)]         [ValidateNotNullOrEmpty()]         [int]$MaxRetries=10,         # Number of seconds delay in retrying. Default is 10 seconds.         [Parameter(Position=2)]         [ValidateNotNullOrEmpty()]         [int]$RetryDelay=10     )     Begin     {         Write-verbose -Message "[BEGIN] Starting the function"         $currentRetry = 1         $Success = $False     }     Process     {         do {            try             {                 Write-Verbose -Message "Running the passed script block -> $($ScriptBlock)"                 $result = & $ScriptBlock # invoke the script block                 $success = $true                 Write-Verbose -Message "Script block ran successfully -> $($ScriptBlock)"                 return $result             }             catch             {                 $currentRetry = $currentRetry + 1                               Write-Error -Message "Failed to execute -> $($ScriptBlock) .`n Error-> ($_.Exception)"  # Write non-terminating error for allowed retries                      if ($currentRetry -gt $MaxRetries) {                              # If the current try count has exceeded maximum retries, throw a terminating error and come out. In place to avoid an infinite loop                      Write-Warning -Message "Could not execute -. $($ScriptBlock).`n Error: -> $($_.Exception)"                     $PSCmdlet.ThrowTerminatingError($PSitem) # Raise the exception back for caller. This is a terminating error as the retries have exceeded MaxRetries allowed.                 }                 else {                     Write-verbose -Message "Waiting $RetryDelay second(s) before attempting again"                     Start-Sleep -seconds $RetryDelay                 }             }         } while(-not $Success) # Do until you succeed     }     End     {         Write-verbose -Message "[END] Ending the function"     } }

Now the trick to using this function in your scripts is to pass it a script block with -ErrorAction set to Stop for the steps you think can probably fail and you want them to be retried.

Let’s rewrite our Copy-ImportantFileToVM function using the Invoke-ScriptBlockWithRetry :

001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038

Function Copy-ImportantFileToVM {     [CmdletBinding()]     param(         [Parameter(Mandatory)]         [ValidateNotNullOrEmpty()]         [String]$VMName         )         $VM = Get-VM -Name $VMName         #Check if Guest integration Service is enabled         $GuestService = $VM.VMIntegrationService.Where({$_.Name -eq 'Guest Service Interface'})         if (-not $GuestService.Enabled) {             #Enable the Guest Integration Service             $GuestServiceStatus = $VM | Get-VMIntegrationService -Name "Guest Service Interface" | Enable-VMIntegrationService -Passthru             if (-not $GuestServiceStatus.Enabled) {                 throw "Couldn't enable Guest Service Interface"             }         }         $CopyFileHash = @{                             Name=$VMName;                             SourcePath="$PSScriptRoot\test.csv";                             DestinationPath='C:\temp\test.csv';                             FileSource='Host';                             CreateFullPath=$true;                             Force = $true;                             Verbose=$true;                             ErrorAction='Stop';                         }         # Copy test CSV to VM         if (Test-Path -Path "$PSScriptRoot\test.csv") {             # Critical step to copy the CSV, I want it to be retried             Invoke-ScriptBlockWithRetry  { Copy-VMFile @CopyFileHash }             # Non-critical step, just an example -> I don't care if notepad is running             Get-Process -Name notepad -ErrorAction SilentlyContinue         } # end if }

If you have a good eye, you would have noticed the non-critical step placed in the script block. I just put it to show that it is possible to have steps within your script block which you don’t care if they throw an exception (-ErrorAction SilentlyContinue will suppress error messages for the non-critical step).

Note that I can also pass the number of maximum retries to be done along with wait interval (in seconds) between the retries. Now you can be very creative and extend this as per your needs in various scenarios.

Have fun exploring!