Files: Files worker can get into a semi-stuck state

[dvernon-tacc]

Jan 30, 2026 - files worker was stuck on a transfer task with 500+ parents with a single child. When the task started it used up all worker threads, but all of the connections were timing out (30 seconds) plus retries. Since we had a bunch of stuff pre-loaded and couldn't wrap that work up quickly, the "fair scheduling" couldn't kick in. It would have eventually, but too many timeouts to wait for it. I cancelled the transfer task via the database, started a second worker, and everything went through fine.

1. Check to see if there's some way to detect this situation and re-assign thing?
2. Check to see if anything is blocked in the ssh pool when this is going on
3. Implement the "try again after time" logic that is in the archive worker.

[dvernon-tacc]

Looks like the issue could be when we can get to the source machine and create a bunch of child tasks, but we cannot get to the destination machine to do the actual copying.

Making a connection is a serial operation - each thread that needs a session will first see if there's a connection, and then if not make one. If a connection fails, we can let all waiting threads know that the connection failed and the will not block on trying over and over to make a connection that's never going to succeed.

We can also port the archive worker's mechanism for retries to the parent/child worker. This means that when failures occur they get put back into the db and retried at some later point.