Cascade of Disconnects from Overloaded Worker

[MarcusSorealheis]

NativeLink uploads outputs concurrently and uses update_with_whole_file, which moves the file into CAS. If two uploads race on the same path, the second one hits NotFound on rename even though the digest is already in CAS.

What happens next is RunningActionImpl did not cleanup and it is usually just a consequence of futures being dropped during a cascade of disconnects.

Logs look like this:

2025-12-10T13:28:21.427171Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the
requirements, will attempt to do it in the background., operation_id: bac6bbc0-0ae9-4503-beb4-c714e00ee88c        at
nativelink-worker/src/running_actions_manager.rs:1393       in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
                                  in nativelink::worker with name: "worker_0"
      2025-12-10T13:28:21.427181Z  WARN nativelink_worker::running_actions_manager: Sending kill to running operation, operation_id:
Uuid(c2b53c14-f948-4f13-88e4-54ccaf58829d)                      at nativelink-worker/src/running_actions_manager.rs:2069       in
nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver                                                 in nativelink::worker with
name: "worker_0"                                                                                 2025-12-10T13:28:21.427189Z ERROR
nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the
background., operation_id: c2b53c14-f948-4f13-88e4-54ccaf58829d
    at nativelink-worker/src/running_actions_manager.rs:1393
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

[rejuvenile]

I've experienced something similar; all of my workers can die.

  2025-12-12T18:59:28.154476Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: fdb0f80b-9c1c-44bf-a9ac-766c5436642c
    at nativelink-worker/src/running_actions_manager.rs:1393
    in nativelink_worker::local_worker::worker_start_action
    in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "fdb0f80b-9c1c-44bf-a9ac-766c5436642c", digest_function: BLAKE3
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

^[[A^[[A^[[B^[[B  2025-12-12T18:59:40.
    at nativelink-worker/src/local_worker.rs:701
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"


thread 'main' (4361087) panicked at src/bin/nativelink.rs:701:9:
Error { code: Internal, messages: ["UpdateForWorker stream closed early", "Actions in transit did not reach zero before we disconnected from the scheduler"] }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace```

[rejuvenile]

Even with your fix, I'll still see workers panic en masse with:

  2025-12-12T20:08:01.184984Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 89de2e51-a9c9-4b3e-a5db-1aa38f9b3f85
    at nativelink-worker/src/running_actions_manager.rs:1415
    in nativelink_worker::local_worker::worker_start_action
    in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "89de2e51-a9c9-4b3e-a5db-1aa38f9b3f85", digest_function: BLAKE3
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"


  2025-12-12T20:08:13.941515Z ERROR nativelink_worker::local_worker: ERROR_MSG: "Actions in transit did not reach zero before we disconnected from the scheduler"
    at nativelink-worker/src/local_worker.rs:701
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"


thread 'main' (4379664) panicked at src/bin/nativelink.rs:701:9:
Error { code: Internal, messages: ["UpdateForWorker stream closed early", "Actions in transit did not reach zero before we disconnected from the scheduler"] }
stack backtrace:
   0:        0x105f1c418 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hd359db2e609495b6
   1:        0x104eff454 - core::fmt::write::h4460d07d062ab387
   2:        0x105f1bde0 - std::io::Write::write_fmt::h64a62b8f2b64042a
   3:        0x105f1c224 - std::sys::backtrace::BacktraceLock::print::h8dd7f3b3558e990e
   4:        0x105f1ba88 - std::panicking::panic_with_hook::hd264492bb1e3c5c2
   5:        0x105f3aeac - std::panicking::panic_handler::{{closure}}::he5fa71baf3eb765b
   6:        0x105f3ae1c - std::sys::backtrace::__rust_end_short_backtrace::h5498fcf1ce8965cd
   7:        0x105f3ae10 - __rustc[10fe9f08a196ddda]::rust_begin_unwind
   8:        0x1060ebdb0 - core::panicking::panic_fmt::h267147df5ee76d62
   9:        0x1053ad768 - nativelink::inner_main::{{closure}}::h1562a3b8a39531ec.52528
  10:        0x105265654 - nativelink::main::h9d571719d44a5c5a
  11:        0x105334f24 - std::sys::backtrace::__rust_begin_short_backtrace::h04f7136a406ad9a0
  12:        0x10529aa8c - _main

[rejuvenile]

These new failures may be the result of the directory cache; since disabling the cache I haven't seen en masse failures. I'll continue to test.

[MarcusSorealheis]

Your reports are very helpful.

[MarcusSorealheis]

@rejuvenile any updates? Still having issues with the latest version + no DirectoryCache?

[rejuvenile]

I don't see every worker panic anymore. Last night a single one did, however.

  2025-12-17T23:11:08.496352Z ERROR nativelink_worker::running_actions_manager: Child process was not cleaned up before dropping the call to execute(), killing in background spawn., result: Ok(None)
    at nativelink-worker/src/running_actions_manager.rs:1011
    in nativelink_worker::local_worker::worker_start_action
    in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "fbab0120-13e4-48d9-a0c5-456b079d7475", digest_function: BLAKE3
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

Then a series of these, for at least a couple of dozen unique operation_ids:

  2025-12-17T23:11:08.502679Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 2c56880d-295d-466d-beeb-8b8ffd19a5b1
    at nativelink-worker/src/running_actions_manager.rs:1415
    in nativelink_worker::local_worker::worker_start_action
    in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "2c56880d-295d-466d-beeb-8b8ffd19a5b1", digest_function: BLAKE3
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

Finally:

  2025-12-17T23:11:10.681105Z ERROR nativelink_store::filesystem_store: Failed to delete file, file_path: "/Users/user/Work/nativelink/data-worker-test/tmp_path-cas/d/6d310b14dc6c3eaddb9e06cad6f769a006264d899c1af925bdeb000000000000-159", err: Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Failed to remove file /Users/user/Work/nativelink/data-worker-test/tmp_path-cas/d/6d310b14dc6c3eaddb9e06cad6f769a006264d899c1af925bdeb000000000000-159"] }
    at nativelink-store/src/filesystem_store.rs:140
    in nativelink_store::filesystem_store::filesystem_delete_file
    in nativelink_worker::local_worker::worker_start_action
    in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "6df1e34d-8583-4459-81ff-a9c99f4c96a7", digest_function: BLAKE3
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

  2025-12-17T23:11:21.157323Z ERROR nativelink_worker::local_worker: ERROR_MSG: "Actions in transit did not reach zero before we disconnected from the scheduler"
    at nativelink-worker/src/local_worker.rs:701
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"


thread 'main' (305026) panicked at src/bin/nativelink.rs:701:9:
Error { code: Internal, messages: ["UpdateForWorker stream closed early", "Actions in transit did not reach zero before we disconnected from the scheduler"] }