TcpStream connect non-blocking

[jonas32]

Hello, I am working on a scenario where a lot of TcpStreams (up to 256) need to run concurrently.
While doing that, i am running into a strange issue.
As soon as the number of running TcpStreams hits the number of tokio core threads (in my case the number of cpu cores), the runtime deadlocks.
Adding timeouts to the TcpStream::connect operations also does not help, since the whole runtime is stuck and the timeout will never kick in.
There are no special properties etc. specified in the connection. To be exact, executing this line brings the deadlock:

let addr = "127.0.0.1:3000";
let stream = TcpStream::connect(addr).await?;

I am not sure if i am doing something wrong or if there are better ways to spawn that many connections.
Has anyone ever seen that behavior or knows a way how i can start the streams without blocking the core threads?

[Darksonn]

It sounds like your application is doing something somewhere that blocks the thread. What exactly that is is hard to say — The tokio::net::TcpStream::connect call does not do that.

[Noah-Kennedy]

If you can get an MCRE (minimum complete, reproducible example) showing this, it would help us find where exactly you are blocking.

[jonas32]

Thanks for asking back. I found a way around it already. I still cant really explain why this is happening, so might still be interesting for you.
I did not push the changes yet, so you can see the actual place where it happens.
https://github.com/asynos/aerospike-client-rust/blob/async/aerospike-core/src/net/connection_pool.rs#L63
In Line 63 a Mutex is being locked.
Line 78 creates a new connection. If you follow the trail there it will lead you to the 'new' function, where the TCP stream is opened.
https://github.com/asynos/aerospike-client-rust/blob/async/aerospike-core/src/net/connection.rs#L49
Here, the first instruction is to open the TCP Stream. That's where the execution stops and will not be resumed.
Looking back at the first link, the Mutex is staying locked while TCP tries to connect. This stops the program, since the Mutex never unlocks. If the Mutex is closed right before initiating the new connection, everything is fine.

To be more exact, the Mutex stays locked since the execution of the whole program will stop and be stuck here. That's the part i cant explain is: This only happens if the Mutex is still locked. If the Mutex unlocked before, it will open the stream.
https://github.com/jonas32/tokio/blob/master/tokio/src/net/tcp/stream.rs#L146

It only happens under the condition that the amount of open TCP Streams (fully opened or opening does not matter) reaches the amount of core threads.

The workaround by explicitly dropping the Mutex right before opening the stream works fine.

I marked the previous post as an answer, since this is not directly tokio related, even if i don't understand why the stream polling stops.