fix(cluster-forwarding): do not forward activity heartbeat in selected-apis-forwarding-v2 (#7488)

<!-- Describe what has changed in this PR -->
**What changed?**
Remove activity heartbeat from selected-apis-forwarding-v2

<!-- Tell your future self why have you made these changes -->
**Why?**
Prevent issues after failover when a large number of heartbeat requests
arrive before replication. Activity heartbeat can happen very frequently
and result in a large number of requests.

<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**


<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**

<!-- Is it notable for release? e.g. schema updates, configuration or
data migration required? If so, please mention it, and also update
CHANGELOG.md -->
**Release notes**
Removing activity heartbeat from selected-apis-forwarding-v2 because of
issues after failover due to possible large volume of heartbeat requests
forwarded before event replication.

<!-- Is there any documentation updates should be made for config,
https://cadenceworkflow.io/docs/operation-guide/setup/ ? If so, please
open an PR in https://github.com/cadence-workflow/cadence-docs -->
**Documentation Changes**

Signed-off-by: fimanishi <[email protected]>

Author: fimanishi

service/frontend/wrappers/clusterredirection/policy.go

@@ -53,21 +53,19 @@ const (
 	// please also reference selectedAPIsForwardingRedirectionPolicyAPIAllowlist and DCRedirectionPolicySelectedAPIsForwardingV2
 	DCRedirectionPolicySelectedAPIsForwarding = "selected-apis-forwarding"
 	// DCRedirectionPolicySelectedAPIsForwardingV2 forwards everything in DCRedirectionPolicySelectedAPIsForwarding,
-	// as well as activity completions (sync and async) and heartbeats.
+	// as well as activity completions (sync and async).
 	// This is done because activity results are generally deemed "useful" and relatively costly to re-do (when it is
 	// even possible to redo), but activity workers themselves may be datacenter-specific.
 	//
 	// This will likely replace DCRedirectionPolicySelectedAPIsForwarding soon.
 	//
 	// 1-6. from DCRedirectionPolicySelectedAPIsForwarding
-	// 7. RecordActivityTaskHeartbeat
-	// 8. RecordActivityTaskHeartbeatByID
-	// 9. RespondActivityTaskCanceled
-	// 10. RespondActivityTaskCanceledByID
-	// 11. RespondActivityTaskCompleted
-	// 12. RespondActivityTaskCompletedByID
-	// 13. RespondActivityTaskFailed
-	// 14. RespondActivityTaskFailedByID
+	// 7. RespondActivityTaskCanceled
+	// 8. RespondActivityTaskCanceledByID
+	// 9. RespondActivityTaskCompleted
+	// 10. RespondActivityTaskCompletedByID
+	// 11. RespondActivityTaskFailed
+	// 12. RespondActivityTaskFailedByID
 	// please also reference selectedAPIsForwardingRedirectionPolicyAPIAllowlistV2
 	DCRedirectionPolicySelectedAPIsForwardingV2 = "selected-apis-forwarding-v2"
 	// DCRedirectionPolicyAllDomainAPIsForwarding means forwarding all the worker and non-worker APIs based domain,
@@ -142,8 +140,6 @@ var selectedAPIsForwardingRedirectionPolicyAPIAllowlistV2 = map[string]struct{}{
 	"TerminateWorkflowExecution":       {},
 	"ResetWorkflowExecution":           {},
 	// additional endpoints
-	"RecordActivityTaskHeartbeat":      {},
-	"RecordActivityTaskHeartbeatByID":  {},
 	"RespondActivityTaskCanceled":      {},
 	"RespondActivityTaskCanceledByID":  {},
 	"RespondActivityTaskCompleted":     {},