[DO_NOT_MERGE] Add new Nexus outbound task for cancelling activities by rkannan82 · Pull Request #9232 · temporalio/temporal

rkannan82 · 2026-02-05T20:00:42Z

Add new transfer task type for activity cancellation control:

ActivityCancelControlTask struct with WorkerInstanceKey field
Serialization/deserialization for persistence
No-op executor (dispatch logic in separate PR)

What changed?

Added a new transfer task type called CancelActivityNexusTask transfer task type. This will be used to dispatch activity cancellation requests to workers via their Nexus control queue.

Main changes:
Task definition
Proto

task.proto: New task type: TASK_TYPE_TRANSFER_CANCEL_ACTIVITY_NEXUS
executions.proto: Task definition

Struct

cancel_activity_nexus_task.go
task_serializers.go: Serialization/deserialization support

Task creation

Controlled by dynamic config: EnableActivityCancellationNexusTask
workflow_task_completed_handler.go: Groups activities by the worker and creates 1 task per group.

Task dispatch using Nexus

Controlled by dynamic config: EnableActivityCancellationNexusTask
Will be implemented in separate PR.

Why?

To support activity cancellation without activity heartbeat.

Overall flow:

[[DO_NOT_MERGE] Store worker attributes needed by server to propagate nexus tasks to worker #9231] Store worker instance key in ActivityInfo as part of RecordActivityTaskStarted call.
[This PR] When user cancels a workflow, create 1 or more transfer tasks. Group all activities belonging to a worker into a transfer task (for efficiency).
[[DO_NOT_MERGE] Dispatch activity cancellation to worker using Nexus #9233] Lookup the Nexus task queue for each worker, and send a Nexus operation for each transfer task.
[SDK] Worker will receive this cancel task and cancel the running activities.

How did you test it?

Potential risks

No. The new task creation is disabled by default.

proto/internal/temporal/server/api/persistence/v1/executions.proto

yycptt · 2026-02-12T00:19:28Z

service/history/api/respondworkflowtaskcompleted/workflow_task_completed_handler.go

+			handler.mutableState.AddTasks(&tasks.CancelActivityNexusTask{
+				WorkflowKey:       handler.mutableState.GetWorkflowKey(),
+				ScheduledEventIDs: []int64{ai.ScheduledEventId},
+				WorkerInstanceKey: ai.WorkerInstanceKey,
+			})


I'd move the logic into ApplyActivityTaskCanceledEvent in mutable state impl, and do AddTasks in task_generator.go, to be consistent with rest of the codebase for task generation.

I kept the top level check in handler because we need to generate the nexus tasks only if the activity is started. Moved the AddTasks code inside the generator as you suggested. Please see if this makes sense.

Just fyi: I have a separate PR to batch activities by the queue. Right now we end up creating 1 task per activity.

service/history/api/respondworkflowtaskcompleted/workflow_task_completed_handler.go

yycptt · 2026-02-12T00:34:36Z

service/history/api/respondworkflowtaskcompleted/workflow_task_completed_handler.go

what about RespondActivityTaskCanceled api?

RespondActivityTaskCanceled is what the SDK sends after it has cancelled. Here, we are initiating that cancel (after Step-5). From my understanding, this is how it works e2e.

User calls RequestCancelWorkflowExecution API

Server creates a new workflow task for the workflow

SDK picks up the workflow task

Workflow code sees cancellation, decides to cancel activities

SDK sends RespondWorkflowTaskCompleted with RequestCancelActivityTask commands

I think standalone activities needs to be handled separately.

RespondActivityTaskCanceled is what the SDK sends after it has cancelled.

ah ok make sense. I was thinking about the ByID version of RespondActivityTaskCanceled api, but yeah you are right, they only get called when the activity is actually cancelled.

What about other cases like workflow termination/timeout?

service/history/transfer_queue_active_task_executor.go

rkannan82 · 2026-02-12T06:39:52Z

service/history/transfer_queue_standby_task_executor.go

 		err = t.processDeleteExecutionTask(ctx, task, false)
 	case *tasks.ChasmTask:
 		err = t.executeChasmSideEffectTransferTask(ctx, task)
+	case *tasks.CancelActivityNexusTask:


@yycptt Need help figuring out if it is OK to skip in standby.

yeah I think it's best effort anyway so ok to skip and won't cause workflow to stuck. Worker also won't poll standby matching I think.

yycptt · 2026-02-18T01:20:52Z

proto/internal/temporal/server/api/enums/v1/task.proto

+}
+
+// ActivityNotificationType specifies the type of notification to send to activities.
+enum ActivityNotificationType {


nit: probably should have its own file

yycptt · 2026-02-18T01:30:12Z

service/history/api/respondworkflowtaskcompleted/workflow_task_completed_handler.go

RespondActivityTaskCanceled is what the SDK sends after it has cancelled.

ah ok make sense. I was thinking about the ByID version of RespondActivityTaskCanceled api, but yeah you are right, they only get called when the activity is actually cancelled.

What about other cases like workflow termination/timeout?

yycptt · 2026-02-18T01:36:12Z

service/history/tasks/notify_activity_task.go

+		// NotificationType specifies the type of notification.
+		NotificationType enumsspb.ActivityNotificationType
+		// ScheduledEventIDs of activities to notify (batched by worker).
+		ScheduledEventIDs []int64


I think we still need failover version here, scheduledEventID is not unique for global namespace in the case of conflict resolution.

- Define ActivityCommandTask in tasks package - Add serialization/deserialization for ActivityCommandTask - Add task generation in TaskGenerator - Add batching logic in workflow task completed handler - Forward WorkerControlTaskQueue through matching service - Add unit tests for task generation and batching - Add metrics support with command type

rkannan82 requested review from a team as code owners February 5, 2026 20:00

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch 3 times, most recently from 9eea93d to 835d5bf Compare February 5, 2026 21:10

rkannan82 requested a review from yycptt February 5, 2026 21:11

This was referenced Feb 5, 2026

[DO_NOT_MERGE] Store worker attributes needed by server to propagate nexus tasks to worker #9231

Open

[DO_NOT_MERGE] Dispatch activity cancellation to worker using Nexus #9233

Open

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from d402868 to 54b6d1a Compare February 6, 2026 00:36

rkannan82 force-pushed the kannan/activity-cancel/persist-worker-key branch from 5bae03a to a5e43dd Compare February 11, 2026 19:48

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch 2 times, most recently from 417606c to 6246c4e Compare February 11, 2026 21:04

rkannan82 force-pushed the kannan/activity-cancel/persist-worker-key branch from f1e1ee5 to e795849 Compare February 11, 2026 21:07

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch 2 times, most recently from fec3c41 to 1dd975d Compare February 11, 2026 21:22

yycptt reviewed Feb 12, 2026

View reviewed changes

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 11c74b6 to 2cb3108 Compare February 12, 2026 00:43

rkannan82 commented Feb 12, 2026

View reviewed changes

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from d0f60d7 to fa4be0a Compare February 12, 2026 06:53

rkannan82 requested a review from yycptt February 12, 2026 07:04

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch 2 times, most recently from 89d2cf5 to 79e4e3e Compare February 15, 2026 03:33

rkannan82 force-pushed the kannan/activity-cancel/persist-worker-key branch from bf3c061 to 1218cae Compare February 15, 2026 03:42

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch 3 times, most recently from 9231795 to bc58c4f Compare February 16, 2026 02:44

rkannan82 force-pushed the kannan/activity-cancel/persist-worker-key branch from 04db6f1 to b72cf9f Compare February 16, 2026 02:48

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from bc58c4f to 3eb028c Compare February 16, 2026 02:48

rkannan82 changed the title ~~Add new transfer task type for cancelling activities using Nexus~~ [DO_NOT_MERGE] Add new Nexus outbound task for cancelling activities Feb 16, 2026

yycptt reviewed Feb 18, 2026

View reviewed changes

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 3eb028c to 92b1a81 Compare February 18, 2026 02:32

rkannan82 force-pushed the kannan/activity-cancel/persist-worker-key branch from b72cf9f to e319a6c Compare February 18, 2026 03:34

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from 92b1a81 to b775ca6 Compare February 18, 2026 03:34

rkannan82 added 11 commits February 17, 2026 19:43

Define CancelActivityNexusTask transfer task type

ac7be08

Fix lint errors

113ca15

Regen proto

5ad9aeb

Remove worker_instance_key as it is not needed

05adf1a

Move CancelActivityNexusTask creation to task_generator

39c1759

Add WorkerControlTaskQueue to CancelActivityNexusTask

131c278

Add metrics tag and low priority for CancelActivityNexusTask

7bc8ada

Update comment for standby executor

01e4822

Add Version field to CancelActivityNexusTask for multi-cluster support

e9b1fac

Add comment

faff7ee

rkannan82 force-pushed the kannan/activity-cancel/task-definition branch from b775ca6 to d6af322 Compare February 18, 2026 03:44

rkannan82 force-pushed the kannan/activity-cancel/persist-worker-key branch from e319a6c to 5390759 Compare February 18, 2026 03:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DO_NOT_MERGE] Add new Nexus outbound task for cancelling activities#9232

[DO_NOT_MERGE] Add new Nexus outbound task for cancelling activities#9232
rkannan82 wants to merge 11 commits intokannan/activity-cancel/persist-worker-keyfrom
kannan/activity-cancel/task-definition

rkannan82 commented Feb 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

yycptt Feb 12, 2026

Uh oh!

rkannan82 Feb 12, 2026

Uh oh!

Uh oh!

yycptt Feb 12, 2026

Uh oh!

rkannan82 Feb 12, 2026

Uh oh!

yycptt Feb 18, 2026

Uh oh!

Uh oh!

Uh oh!

rkannan82 Feb 12, 2026

Uh oh!

yycptt Feb 18, 2026

Uh oh!

yycptt Feb 18, 2026

Uh oh!

yycptt Feb 18, 2026

Uh oh!

yycptt Feb 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

rkannan82 commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed?

Why?

How did you test it?

Potential risks

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rkannan82 commented Feb 5, 2026 •

edited

Loading