[Wf-Diagnostics] add troubleshooting guide for activity and workflow retries

Author: sankari165

src/.vuepress/config.js

@@ -178,6 +178,7 @@ module.exports = {
             '08-workflow-troubleshooting/',
             '08-workflow-troubleshooting/01-timeouts',
             '08-workflow-troubleshooting/02-activity-failures',
+            '08-workflow-troubleshooting/03-retries',
           ],
         },
         {

src/docs/08-workflow-troubleshooting/01-timeouts.md

@@ -27,20 +27,37 @@ Optionally you can also increase the number of pollers per worker by providing t
 [Link to options in go client](https://pkg.go.dev/go.uber.org/[email protected]/internal#WorkerOptions)
 [Link to options in java client](https://github.com/uber/cadence-java-client/blob/master/src/main/java/com/uber/cadence/internal/worker/PollerOptions.java#L124)
 
-## Timeouts without heartbeating enabled
+## Timeouts without heartbeat timeout or retry policy configured
 
 Activities time out StartToClose or ScheduleToClose if the activity took longer than the configured timeout.
 
 [Link to description of timeouts](https://cadenceworkflow.io/docs/concepts/activities/#timeouts)
 
 For long running activities, while the activity is executing, the worker can die due to regular deployments or host restarts or failures. Cadence doesn't know about this and will wait for  StartToClose or ScheduleToClose timeouts to kick in.
 
-Mitigation: Consider enabling heartbeating
+Mitigation: Consider configuring heartbeat timeout and a retry policy
 
 [Configuring heartbeat timeout example](https://github.com/uber-common/cadence-samples/blob/df6f7bdba978d6565ad78e9f86d9cd31dfac9f78/cmd/samples/expense/workflow.go#L23)
+[Check retry policy for activity](https://cadenceworkflow.io/docs/concepts/activities/#retries)
 
 For short running activities, heart beating is not required but maybe consider increasing the timeout value to suit the actual activity execution time.
 
+## Timeouts without heartbeat timeout configured but a retry policy configured
+
+Retry policies are good to be configured so that activities can be retried after timeouts or failures. For long running activities, while the activity is executing, the worker can die due to regular deployments or host restarts or failures. Cadence doesn't know about this and will wait for  StartToClose or ScheduleToClose timeouts to kick in. The retry is attempted only after this timeout. Enabling heartbeating would cause the activity to timeout earlier and will be retried on another worker.
+
+Mitigation: Consider configuring heartbeat timeout
+
+[Configuring heartbeat timeout example](https://github.com/uber-common/cadence-samples/blob/df6f7bdba978d6565ad78e9f86d9cd31dfac9f78/cmd/samples/expense/workflow.go#L23)
+
+## Timeouts with heartbeating enabled but without a retry policy configured
+
+Heartbeat timeouts are used to detect when a worker died or restarted during deployments. With heartbeat timeout enabled, the activity will timeout faster. But without a retry policy, it will not be scheduled again on a healthy worker.
+
+Mitigation: Consider adding retry policy to an activity
+
+[Check retry policy for activity](https://cadenceworkflow.io/docs/concepts/activities/#retries)
+
 ## Heartbeat Timeouts after enabling heartbeating
 
 Activity has enabled heart beating but the activity timed out with heart beat timeout. This is because the server did not receive a heart beat in the time interval configured as the heart beat timeout.

src/docs/08-workflow-troubleshooting/03-retries.md

@@ -0,0 +1,33 @@
+---
+layout: default
+title: Retries
+permalink: /docs/workflow-troubleshooting/retries
+---
+
+# Retries
+
+Cadence has a retry feature where a retry policy can be configured so that an activity or a workflow can be retried when it fails or times out.
+
+Read more about [activity retries](https://cadenceworkflow.io/docs/concepts/activities/#retries) and [workflow retries](https://cadenceworkflow.io/docs/concepts/workflows/#workflow-retries)
+
+## Workflow execution history of retries
+
+One thing to note is how activity retries and workflow retries are shown in the Cadence Web UI. All the activity retries are not part of workflow execution history and only the last attempt is shown with the attempt number.
+
+Moreover, attempt number starts from 0, so Attempt:0 refers to the first and original attempt or Attempt:1 refers to the second attempt or first retried attempt.
+
+For workflow retries, when a workflow fails or times out and is retried, it completes the previous execution with a ContinuedAsNew event and a new execution is started with Attempt 1. The ContinuedAsNew event holds the details of the failure reason.
+
+## Configuration of activity retries and workflow retries
+
+Some of the configurable values could be misconfigured and a result will not have the intended behaviour. These are listed here.
+
+## MaximumAttempts set to 1
+
+In both activity retries and workflow retries it is sufficient to mention a maximum number of attempts or an expiration interval. However, the maximum number of attempts counts the original attempt of the activity also. As a result, setting maximum number of attempts to 1 means the activity or workflow will not be retried.
+
+## ExpirationIntervalInSeconds less than InitialIntervalInSeconds
+
+In both activity retries and workflow retries it is sufficient to mention a maximum number of attempts or an expiration interval. The first retry attempt waits for the InitialIntervalInSeconds before starting and when an expiration interval is set lower than the initial interval, the retry policy becomes invalid and the activity or workflow will not be retried.
+
+