Add batch workflow refresh tasks #8793

awln-temporal · 2025-12-10T16:48:29Z

What changed?

Add batch workflow refresh tasks Admin API.

Add tdbg commands to invoke StartAdminBatchOperation.

Details

AdminBatchOperations are executed in the Batched System Worker, similar to existing Frontend BatchOperations. The worker activity will fetch pages of workflow executions to perform operations on, and periodically heartbeat its results. For RefreshWorkflowTasks, each individual Refresh call will go through the admin handler.

Refreshing Workflow Tasks regenerates all pending tasks of an execution given its mutable state.

CLI usage

tdbg workflow batch-refresh-tasks \
  --namespace <ns> \
  --query "WorkflowType='MyType'" \
  --reason "fixing stuck workflows" \
  [--job-id <optional-job-id>] \
  [--archetype <optional-archetype>]

Why?

Unblock new Matcher migration and allow for general use case batch Admin calls. Currently, only supports BatchOperationRefreshWorkflowTasks.

How did you test it?

yycptt · 2025-12-10T23:27:57Z

Haven't review the actual changes yet but for the tdbg experience, let’s match what we have in the temporal CLI. so something like `tdbg workflow refresh-tasks -q '' means to start a batch operation. and if -q is not specified and --wid is specified, then it's not a batch op, but only operating on that single workflow.

proto/internal/temporal/server/api/adminservice/v1/request_response.proto

proto/internal/temporal/server/api/batch/v1/request_response.proto

yycptt

cc @prathyushpv Can you help review this PR as well?

common/resource/fx.go

service/frontend/admin_handler.go

service/worker/batcher/activities.go

yycptt · 2025-12-15T06:44:03Z

After this PR is landed, CGS can considering adding a batch force migrate command as well.
cc @yux0 @xwduan @hai719

awln-temporal · 2025-12-15T20:20:01Z

Haven't review the actual changes yet but for the tdbg experience, let’s match what we have in the temporal CLI. so something like `tdbg workflow refresh-tasks -q '' means to start a batch operation. and if -q is not specified and --wid is specified, then it's not a batch op, but only operating on that single workflow.

Do we care about setting Memos for the Admin Batch Operation, eg. Reason, JobID, that are common with User Batch Operations, but not usable for single RefresWorkflowTasks ops?

dnr · 2025-12-16T00:21:57Z

service/worker/batcher/activities.go

+	case *adminservice.StartAdminBatchOperationRequest_RefreshWorkflowTasksOperation:
+		return processTask(ctx, limiter, task,
+			func(execution *commonpb.WorkflowExecution) error {
+				_, err := a.AdminClient.RefreshWorkflowTasks(ctx, &adminservice.RefreshWorkflowTasksRequest{


just to be clear, we're okay with any user being able to do this at any time?

refresh tasks is safe, but my concern is that someone is going to add a more dangerous admin operation here and not realize that it's exposed to all users in the namespace.

I would say we need a big warning comment somewhere but I'm not sure where it should go.

I can add a big warning to the proto definition

I think we will need a general way for preventing user from starting this workflow directly, even for normal operations like signal/terminate etc, which allows them to bypass, for example, the concurrent limit.

service/frontend/admin_handler.go

service/worker/batcher/workflow.go

proto/internal/temporal/server/api/adminservice/v1/request_response.proto

proto/internal/temporal/server/api/batch/v1/request_response.proto

service/frontend/admin_handler.go

service/worker/common/chasm_util.go

service/worker/batcher/activities.go

yycptt · 2025-12-22T23:12:14Z

Haven't review the actual changes yet but for the tdbg experience, let’s match what we have in the temporal CLI. so something like `tdbg workflow refresh-tasks -q '' means to start a batch operation. and if -q is not specified and --wid is specified, then it's not a batch op, but only operating on that single workflow.

Do we care about setting Memos for the Admin Batch Operation, eg. Reason, JobID, that are common with User Batch Operations, but not usable for single RefresWorkflowTasks ops?

I think it's nice to have them (reason and batch type) in the memo. JobID is not part of the memo I believe.

common/dynamicconfig/constants.go

prathyushpv · 2025-12-23T18:16:47Z

proto/internal/temporal/server/api/adminservice/v1/request_response.proto

+
+// StartAdminBatchOperationRequest starts an admin batch operation.
+// WARNING: Batch Operations are exposed to all users of the namespace. Admin Batch Operations should be exercised with caution.
+message StartAdminBatchOperationRequest {


I was wondering if a field to specify batch operation RPS would be helpful in input. We have max_operations_per_second in StartBatchOperationRequest. We can use this field to control the rate. It may not be strictly necessary since we are marking this batch operation with the lowest priority, and it should not block other operations.
cc: @yycptt

looks like this max_operations_per_second field isn't properly handled by the activity? It seems to always default to the Dynamic Config RPS value https://github.com/temporalio/temporal/blob/admin-batch-refresh-workflows/service/worker/batcher/fx.go#L89.

yeah looks like it's not used at all right now and we just rely on the dc value to protect us.

tools/tdbg/tdbg_commands.go

tools/tdbg/commands.go

service/worker/deployment/deployment_reachability_test.go

yycptt · 2025-12-27T00:26:42Z

tools/tdbg/commands.go

+
+	_, err = adminClient.StartAdminBatchOperation(ctx, &adminservice.StartAdminBatchOperationRequest{
+		Namespace:       nsName,
+		VisibilityQuery: query,


not related to this PR.

Does visibility query parser support BusinessID today? I mean we added support for each component to define it's own alias for the underlying WorkflowID column, but we will also need a default alias I think. Very low priority though.

At least right now, WorkflowID should be able to be used as a default system search attribute to retrieve the BusinessID, we never made the change to disallow querying against WorkflowID yet, since that might impact scheduler query backwards compatibility.

oh I didn't mean to disallow WorkflowID. just we need to allow BusinessID.

gotcha, I just recall a discussion with Roey/Rodrigo about disallowing before, but maybe we just ignore that.

tools/tdbg/commands.go

yycptt · 2025-12-29T21:33:58Z

tools/tdbg/commands.go

+
+	_, err = adminClient.StartAdminBatchOperation(ctx, &adminservice.StartAdminBatchOperationRequest{
+		Namespace:       nsName,
+		VisibilityQuery: query,


oh I didn't mean to disallow WorkflowID. just we need to allow BusinessID.

tools/tdbg/commands.go

yycptt · 2025-12-29T21:43:13Z

tests/admin_batch_refresh_workflow_tasks_test.go

nit: consider adding tests for 1. admin batch and normal batch don't share the same limit and 2. ListBatchOperations doesn't show admin batch op.

added tests to validate separate namespace divisions.

Add batch workflow refresh tasks

d2a96ac

awln-temporal requested review from a team as code owners December 10, 2025 16:48

awln-temporal requested a review from yycptt December 10, 2025 16:48

Add tdbg commands for batch refresh workflow tasks

a224081

awln-temporal force-pushed the admin-batch-refresh-workflows branch from bfd3e1b to 32e6d5e Compare December 10, 2025 17:14

Address linter and depedency injection issues

94f9a55

awln-temporal force-pushed the admin-batch-refresh-workflows branch from 32e6d5e to 94f9a55 Compare December 10, 2025 18:22

yycptt reviewed Dec 11, 2025

View reviewed changes

Merge branch 'main' into admin-batch-refresh-workflows

b2f7454

yycptt reviewed Dec 15, 2025

View reviewed changes

common/resource/fx.go Outdated Show resolved Hide resolved

service/frontend/admin_handler.go Show resolved Hide resolved

service/worker/batcher/activities.go Outdated Show resolved Hide resolved

service/worker/batcher/activities.go Outdated Show resolved Hide resolved

yycptt requested a review from prathyushpv December 15, 2025 06:44

dnr reviewed Dec 16, 2025

View reviewed changes

awln-temporal requested a review from spkane31 December 16, 2025 20:12

awln-temporal force-pushed the admin-batch-refresh-workflows branch from a7bd46f to 54bd801 Compare December 17, 2025 17:37

Address comments, parse Archetype from TemporalNamespaceDivision

30b305c

awln-temporal force-pushed the admin-batch-refresh-workflows branch from 54bd801 to 30b305c Compare December 17, 2025 20:39

yycptt reviewed Dec 22, 2025

View reviewed changes

prathyushpv reviewed Dec 23, 2025

View reviewed changes

common/dynamicconfig/constants.go Show resolved Hide resolved

prathyushpv reviewed Dec 23, 2025

View reviewed changes

Address comments

f083e5b

yycptt reviewed Dec 23, 2025

View reviewed changes

tools/tdbg/tdbg_commands.go Outdated Show resolved Hide resolved

awln-temporal added 3 commits December 23, 2025 17:21

Merge branch 'main' into admin-batch-refresh-workflows

142a7ea

Move tdbg commands to unified refresh tasks cli call

3551271

Merge branch 'main' into admin-batch-refresh-workflows

bc0cbad

yycptt reviewed Dec 27, 2025

View reviewed changes

awln-temporal added 2 commits December 29, 2025 12:05

Remove deleted files from main

1268eff

Add user confirmation for cli command

a286808

awln-temporal force-pushed the admin-batch-refresh-workflows branch from acea49c to a286808 Compare December 29, 2025 19:06

yycptt approved these changes Dec 29, 2025

View reviewed changes

Add additional test for different limits and namespace divisions

cb3823c

awln-temporal force-pushed the admin-batch-refresh-workflows branch from 3a579fa to cb3823c Compare January 6, 2026 21:04

Merge branch 'main' into admin-batch-refresh-workflows

3a726cc

awln-temporal merged commit 98e7254 into main Jan 7, 2026
60 checks passed

awln-temporal deleted the admin-batch-refresh-workflows branch January 7, 2026 00:21

Add batch workflow refresh tasks #8793

Add batch workflow refresh tasks #8793

Uh oh!

Conversation

awln-temporal commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed?

Details

CLI usage

Why?

How did you test it?

Uh oh!

yycptt commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yycptt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yycptt commented Dec 15, 2025

Uh oh!

awln-temporal commented Dec 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yycptt commented Dec 22, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

awln-temporal Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

awln-temporal commented Dec 10, 2025 •

edited

Loading

yycptt commented Dec 10, 2025 •

edited

Loading

awln-temporal Dec 23, 2025 •

edited

Loading