Rare SendErrors in Certain Zero

[NicEastvillage]

SendErrors happen when one thread tries to send a message to another thread, and the other thread does not attempt to receive it. In #183 most SendErrors where resolved by

• making the master thread receive the final assignments even if not interested
• making workers stop immediately after receiving a terminate message

However, rare SendErrors still happens. I believe this occurs when worker A sends a message to worker B, and worker B has received the terminate message and shut down, while worker A has just started a task requiring communication with B and missed the terminate message.

In order to handle this, we might need another two passes of the token. One pass, where once a worker receives the token, it stops sending messages to other workers, and a second pass where it receives incoming messages and then shuts down.

[falkecarlsen]

Futhermore, we do not have a resend method which is what I gather generates this error when running many concurrent benches:
thread 'main' panicked at 'failed to spawn thread: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/thread/mod.rs:717:29

The full stderr trace on mexi3p3hp: https://gist.github.com/falkecarlsen/2aaee32933b2174dc3863f4982868edf

[NicEastvillage]

That looks unrelated. I think you are simply spawning too many threads too fast and ran out of some OS resource (no idea which), and that means some workers will fail to send, because they cannot send to the missing workers.

[falkecarlsen]

Yes, perhaps unrelated but suspiciously showed up after incorporating changes proposed by #183 and #188.
It might be a node problem otherwise it smells like a bug in how we use threads, e.g. an async reader which never gets a message might be infinitely spawned under some conditions.
We'll see, I'll investigate further.