SSH cleanup race: rm -rf Directory not empty aborts ratio_unfinished submission #682

Bug: SSH cleanup fails with Directory not empty after ratio_unfinished early termination

Summary

With an SSH context on a NAS-backed remote filesystem, a submission using strategy.ratio_unfinished > 0 can reach the unfinished-task threshold, call remove_unfinished_tasks(), and then fail in SSHContext._rmtree() during rm -rf. The non-zero return raises RuntimeError and aborts the DP-GEN stage even though completed task results were already downloaded.

Observed behavior

• 32 model-deviation tasks; 26 finished (81.25%).
• ratio_unfinished = 0.2 triggered remove_unfinished_tasks().
• Remaining task directories were still being finalized on the remote NAS.
• rm -rf <remote_root> returned: rm: cannot remove .../task.000.000000: Directory not empty.
• SSHContext._rmtree() propagated the error; the parent DP-GEN process exited with RuntimeError.
• 26 completed model_devi.out files had already been downloaded.

Environment

• dpdispatcher 1.0.3
• SSH context
• remote root on NAS-backed storage
• Python 3.12

Expected behavior

Cleanup should tolerate transient ENOTEMPTY/directory metadata races (for example, retry with backoff or use best-effort asynchronous cleanup) and return the completed submission state instead of aborting the workflow.

Relevant code

• dpdispatcher/submission.py::remove_unfinished_tasks
• dpdispatcher/contexts/ssh_context.py::_rmtree

The issue is reproducible when unfinished tasks are still writing restart/finalization files as cleanup starts.