[Bug] Replay workflow history fails with nondeterminism error, child workflow ids do not match

[mishabruml]

Related: #1582

What are you really trying to do?

I am trying to write a "workflow history compatibiltiy test" using Worker.runReplayHistory, so it can be run in CI to protect our team against accidentally introducing non-deterministic changes into our workflows.

Describe the bug

I am grabbing workflow history from a recent successful workflow run, pasting it as a JSON blob, and running a test like this:

import { Worker } from '@temporalio/worker';
import workflowHistory from './workflow-history.json';

describe('Workflow history compatibility', () => {
  test('Workflow history is compatible', async () => {
    await Worker.runReplayHistory(
      {
        workflowsPath: require.resolve('../workflow.ts'),
      },
      workflowHistory
    );
  });
});

The workflow contains several child workflows, for example:

  const child = executeChild(ChildWorkflow, {
    args: [childBody],
    taskQueue: QueueConfig.ChildWorkflow,
  });

When I run the test (locally), I get the following error:

DeterminismViolationError: Replay failed with a nondeterminism error. This means that the workflow code as written is not compatible with the history that was fed in. Details: Workflow activation completion failed: Failure { failure: Some(Failure { message: "[TMPRL1100] Nondeterminism error: Child workflow id of scheduled event '83ade076-5748-4685-b311-39985365e54c' does not match child workflow id of command '01324793-0653-4b87-ba22-639e2eaffe97'", source: "", stack_trace: "", encoded_attributes: None, cause: None, failure_info: Some(ApplicationFailureInfo(ApplicationFailureInfo { r#type: "", non_retryable: false, details: None, next_retry_delay: None, category: Unspecified })) }), force_cause: NonDeterministicError }

I have also tried modifying the workflow so that the child workflow executions start like this:

  import { uuid4 } from '@temporalio/workflow';

  const child = executeChild(ChildWorkflow, {
    args: [childBody],
    taskQueue: QueueConfig.ChildWorkflow,
    workflowId: uuid4(),
  });

And ran the workflow succesfully, grabbed the history from this run, and dropped that into my test- same outcome.

This seems like a bug, the child workflowId should not affect the determinism of the workflow run.

Environment/Versions

• OS and processor: [e.g. Apple M2 Pro, Sequoia 5.6.1 (24G90)]
• Temporal Version:

    "@temporalio/activity": "^1.11.1",
    "@temporalio/client": "^1.11.3",
    "@temporalio/common": "^1.11.2",
    "@temporalio/interceptors-opentelemetry": "^1.11.7",
    "@temporalio/nyc-test-coverage": "^1.11.2",
    "@temporalio/worker": "^1.11.1",
    "@temporalio/workflow": "^1.11.1",

• Are you using Docker or Kubernetes or building Temporal from source? Docker

[mjameswh]

    "@temporalio/activity": "^1.11.1",
    "@temporalio/client": "^1.11.3",
    "@temporalio/common": "^1.11.2",
    "@temporalio/interceptors-opentelemetry": "^1.11.7",
    "@temporalio/nyc-test-coverage": "^1.11.2",
    "@temporalio/worker": "^1.11.1",
    "@temporalio/workflow": "^1.11.1",

Those would be version numbers as they are declared in your package.json, but not necessarily what's actually installed. Can you print the installed version numbers of those dependencies? You should be able to get them using npm ls.

Mixing packages of different versions of @temporalio/* packages is not supported, not even at the patch level. Our packages have exact dependencies to other packages to avoid such incoherencies. So in practice, I expect that all of these packages will have actually been installed at 1.11.7 (or 1.11.3 if using an older version of NPM that didn't support peer dependencies).

[mishabruml]

thanks, it appears to be 1.13.0

we're on yarn 4.9.2, node 22.18.0 (npm 10.9.3)

│ ├── @temporalio/activity@1.13.0
│ ├── @temporalio/client@1.13.0
│ ├── @temporalio/common@1.13.0
│ ├── @temporalio/interceptors-opentelemetry@1.13.0
│ ├── @temporalio/nyc-test-coverage@1.13.0
│ ├── @temporalio/testing@1.13.0
│ ├── @temporalio/worker@1.13.0
│ ├── @temporalio/workflow@1.13.0

[mishabruml]

hi sorry @mjameswh any update on this? 🙏

[mjameswh]

Given that you are using the OpenTelemetry Interceptors, I suppose you are hitting the issue described in #1677. We are working on a fix for that bug.

To confirm, how is your workflow getting started? e.g. are you using signal-with-start or update-with-start? And from where are you starting child workflows? e.g. directly from your workflow function, from signal/update handler, or both?

[mishabruml]

We're starting workflows with client.workflow.start() and we're starting child workflows directly from the workflow function using startChild/executeChild

Thanks for the hint, I'll look into #1677 👀