Decoupling Task Removal to Improve `get_task_result` Performance

[pcrespov]

When handling long-running tasks, calling get_task_result has become noticeably slower. The root cause is that this function blocks until remove_task completes (due to wait_for_removal=True).

SEE LRT: create (1), inspect x2 (2) and result (3)

Why is wait_for_removal=True used?

This option was introduced to preserve the pre-refactor behavior of the LRT module (before RPC integration and related changes).

How to restore performance?

To make get_task_result faster, we should run with wait_for_removal=False. However, this requires adjustments to ensure safety:

• Redefine mark_task_for_removal: Once a task is marked for removal, the TasksManager API should treat it as gone.

  • Calls like list or get must not return that task.
  • Conceptually, marking a task for removal should move it to “trash,” making it inaccessible while the actual cleanup completes in the background.

With this approach, we can safely use wait_for_removal=False without risking inconsistent access to partially removed tasks.

[sanderegg]

@GitHK @pcrespov
I would still suggest to make it a client decision whether this should be awaited or not. but yes this should also be high priority.
But I think we also discussed this in #8198