added new section

Author: SnehaSudhirG

articles/automation/manage-runbooks.md

@@ -3,7 +3,7 @@ title: Manage runbooks in Azure Automation
 description: This article tells how to manage runbooks in Azure Automation.
 services: automation
 ms.subservice: process-automation
-ms.date: 11/02/2021
+ms.date: 01/16/2022
 ms.topic: conceptual
 ms.custom: devx-track-azurepowershell
 ---
@@ -216,6 +216,105 @@ If your runbook normally runs within a time constraint, have the script implemen
 > [!NOTE]
 > The local time on the Azure sandbox process is set to UTC. Calculations for date and time in your runbooks must take this fact into consideration.
 
+## Retry logic in runbook to avoid transient failures
+
+As part of the core logic, most runbooks make calls to the remote systems. For example, the calls are to:
+- Azure via ARM
+- Azure Resource Graph
+- Other web services
+- SQL Services
+
+When the system that the runbooks are calling are busy, temporary unavailable or implementing throttling under load, the calls are vulnerable to have runtime errors. To build resiliency into the runbooks, you must implement the retry logic while making the calls so that the runbooks can handle a transient problem without failing. 
+
+For more information, refer [Retry pattern](https://learn.microsoft.com/azure/architecture/patterns/retry) and [General REST and retry guidelines](https://learn.microsoft.com/azure/architecture/best-practices/retry-service-specific#general-rest-and-retry-guidelines).
+
+### Example 1: If your runbook makes only one or two calls
+
+```powershell
+$searchServiceURL = "https://$searchServiceName.search.windows.net"
+$resource = Get-AzureRmResource -ResourceType "Microsoft.Search/searchServices" -ResourceGroupName $searchResourceGroupName -ResourceName  $searchServiceName -ApiVersion 2015-08-19
+$searchAPIKey = (Invoke-AzureRmResourceAction -Action listAdminKeys -ResourceId $resource.ResourceId -ApiVersion 2015-08-19 -Force).PrimaryKey
+```
+When you call `Invoke-AzureRmResourceAction`, transient failures are observed. In such scenario, we recommend that you implement the following basic pattern around the call to the cmdlet.
+
+```powershell
+$searchServiceURL = "https://$searchServiceName.search.windows.net"
+$resource = Get-AzureRmResource -ResourceType "Microsoft.Search/searchServices" -ResourceGroupName $searchResourceGroupName -ResourceName  $searchServiceName -ApiVersion 2015-08-19
+
+    # Adding in a retry
+    $Stoploop = $false
+    $Retrycount = 0
+ 
+    do {
+        try   {
+               $searchAPIKey = (Invoke-AzureRmResourceAction -Action listAdminKeys -ResourceId $resource.ResourceId -ApiVersion 2015-08-19 -Force).PrimaryKey
+               write-verbose "Invoke-AzureRmResourceAction on $resource.ResourceId completed"
+               $Stoploop = $true
+              }
+        catch {
+               if ($Retrycount -gt 3)
+                 {
+                  Write-verbose "Could not Invoke-AzureRmResourceAction on $resource.ResourceId after 3 retrys."
+                  $Stoploop = $true
+                 }
+               else  
+                 {
+                  Write-verbose "Could not Invoke-AzureRmResourceAction on $resource.ResourceId retrying in 30 seconds..."
+                  Start-Sleep -Seconds 30
+                  $Retrycount = $Retrycount + 1
+                 }
+               }
+        }
+    While ($Stoploop -eq $false)
+```
+>[!NOTE]
+>The attempt to retry the call is up to three times, sleeping for 30 seconds each time.
+
+### Example 2 : If the runbook is making frequent remote calls
+
+If the runbook is making frequent remote calls that it could experience transient runtime issues, then create a function that will implement this logic for each call that is made and pass the call to be made in as a script block to execute.
+
+```powershell
+Function ResilientRemoteCall {
+
+         param(
+               $scriptblock
+               )
+        
+         $Stoploop = $false
+         $Retrycount = 0
+ 
+         do {
+             try   {
+                    Invoke-Command -scriptblock $scriptblock 
+                    write-verbose "Invoked $scriptblock completed"
+                    $Stoploop = $true
+                   }
+             catch {
+                    if ($Retrycount -gt 3)
+                      {
+                       Write-verbose "Invoked $scriptblock failed 3 times and we will not try again."
+                       $Stoploop = $true
+                      }
+                    else  
+                      {
+                       Write-verbose "Invoked $scriptblock failed  retrying in 30 seconds..."
+                       Start-Sleep -Seconds 30
+                       $Retrycount = $Retrycount + 1
+                      }
+                    }
+             }
+         While ($Stoploop -eq $false)
+}
+```
+
+You can then pass each remote call into the function as </br>
+
+`ResilientRemoteCall { Get-AzVm }` </br> or </br>
+
+`ResilientRemoteCall { $searchAPIKey = (Invoke-AzureRmResourceAction -Action listAdminKeys -ResourceId $resource.ResourceId -ApiVersion 2015-08-19 -Force).PrimaryKey}`
+
+
 ## Work with multiple subscriptions
 
 Your runbook must be able to work with [subscriptions](automation-runbook-execution.md#subscriptions). For example, to handle multiple subscriptions, the runbook uses the [Disable-AzContextAutosave](/powershell/module/Az.Accounts/Disable-AzContextAutosave) cmdlet. This cmdlet ensures that the authentication context isn't retrieved from another runbook running in the same sandbox.