temporal-operator: Build ID regeneration for non-workflow code changes causes excessive deployments with PINNED versioning

[tshtark]

• Overview

We are using the temporal worker controller with PINNED versioning strategy for long-running workflows (days-weeks).

However, we're experiencing deployment proliferation because the build_id is
regenerated for ANY pod spec change, including non-workflow code changes.

• Use Case

  • Service: Multiple workflow workers
  • Workflow characteristics:
    • New workflow starts every ~30 minutes
    • Workflows run for days
    • Long-running workflows cannot be patched mid-execution (business requirements)
  • Deployment frequency:
    • Workflow code changes: Infrequent (detected by CI - files ending in .workflow.ts)
    • Non-workflow code changes (main.ts, configuration, dependencies): Every ~2 hours
  • Versioning strategy: PINNED (required due to long-running workflows and desire to make
    breaking changes)

• The Problem

  Current Behavior:

  1. ComputeBuildID() hashes the entire pod template spec
  2. ANY change (image tag, env vars, resources, etc.) generates a new build_id
  3. New build_id → new Kubernetes Deployment → new worker version registered with Temporal
  4. With PINNED versioning, old deployments stay alive until their workflows complete

• Result:

  • Deploying non-workflow code changes (bug fixes in main.ts, dependency updates, etc.)
    creates new deployments
  • Combined with our timing constraints (new workflow every 30min, workflows run for days), we end up with 10-15 active deployments running simultaneously
  • This causes resource waste and operational complexity

• Expected Behavior

  Ideally, we'd like to:

  1. Workflow code changes → new build_id → new deployment/version (current behavior - this is correct)
  2. Non-workflow code changes → same build_id → rolling update of existing deployment pods (new behavior)

  This would allow us to:

  • Deploy bug fixes and non-workflow improvements frequently
  • Keep only 1-2 active deployments at any time
  • Maintain workflow execution integrity with PINNED versioning

[carlydf]

Hi there, our reasoning for including the entire pod spec in the build ID was in case things such as secrets and environment variables (esp. feature flags) would cause branching in the Workflow definition that is nondeterministic (even with the same workflow code / image, a feature flag could be a breaking change).

The controller only has access to the pod spec and therefore the image name + surrounding pod spec. It doesn't know whether some files within that image changed while some didn't. We felt that the safest default behavior would be to change the build id whenever the image changed or when the pod spec changed, because that does not require users to correctly identify which changes are "workflow breaking" or not.

However, I totally see that more advanced users might want to change their build id less frequently, on a cadence that they have more control over.

If there was a place in the TemporalWorkerDeployment spec to manually specify your Target Build ID, would you be able to populate that correctly within your kubernetes resource deployment pipeline?

[tshtark]

Hi @carlydf ,
Thanks for the clarification!

YES. If there was a place in the TemporalWorkerDeployment spec to manually specify your Target Build ID, I Would be able to populate that correctly within your kubernetes resource deployment pipeline.

My approach would be to, during the CI pipeline, hash the full content of workflow bundle file and use it as BuildID. that way only changes to workflow files will cause change in the buildID and thus new deployment.

What do you think?