Pasta network establishing socket connection prematurely?

[bollhals]

Hi

I have a kind of a strange issue, that I'm not sure if this is expected or a misconfiguration somewhere on my side, but since it seems since I switched from slirp4netns to paste for my container network, have have an minor issue with my network setup.

I have e.g. 2 containers starting up at the same time, one is an RabbitMQ Broker, the other one is an example application that as part of the startup establishes an connection to the RabbitMQ.

Obviously the RabbitMQ container needs some time to successfully start, so the example application is faster in calling connect (TCP connection). With slirp4netns it waited until the RabbitMQ came online before it proceeded, or timed out doing so. With Pasta it seems that the connection is established almost immediately, and the example application as part of the connection setup to rabbitmq issues its first read on the socket, which fails, as the RabbitMQ server hasn't come up yet.

This left me wondering, is there a functional difference between how pasta and slirp4netns deal with connection establishment between containers when the server isn't accepting any requests yet or am I having some other strange issue that I do not see clearly yet?

Any help is appreciated!

[Luap99]

I am not aware of any meaningful difference between pasta and slirp4netns in that regard. I am not aware of slirp4nents waiting for connections, maybe it was just slower to set up which meant the container did already bound the port?

Maybe @sbrivio-rh has more insights in how a connection is established.

[sbrivio-rh]

• Currently we use 2024_12_11.09478d5, update to 2025_06_11.0293c6f is in the works.

Hah, then that should fix your issue.

• We've been using rootless podman with slirp4netns and now with pasta. But this issue only exist with pasta. And I've only tried with two containers in the same host.
  So with my limited understanding it sounds like this is expected (at least for now until the mentioned partial solution might get implemented), but I don't know/understand why it is different with slirp4netns?

What's expected is that inbound connections will be first accepted, and then reset (instead of just being reset right away), but outbound connections should be properly reset (which, I think, is your actual issue here).

For inbound connections (from external hosts, that is, not loopback connections):

• pasta needs to connect to the service in the container to know if the client can actually connect, and, in order to connect, it's sending an actual Layer-2 (Ethernet) frame, with an IP (Layer-3) header, and TCP (Layer-4) information. It's a SYN segment.

  For that, it needs to know the source address of the client (because it preserves the source address), and, for that, it needs to accept() the connection. No accept(), no information about the client.

• the port forwarder you use in slirp4netns mode is, by default, rootlesskit/rootlessport (matching the port_handler=rootlesskit option, which is the default). That port forwarder doesn't preserve the source address, and it will not actually build a Ethernet frame (it just connect()s to the service in the container instead of building packets).

  So it doesn't need to accept() a connection to get the source address before connecting, it can just call connect() and, if that fails, not call accept().

[bollhals]

• Currently we use 2024_12_11.09478d5, update to 2025_06_11.0293c6f is in the works.

Hah, then that should fix your issue.

We'll update and report back if it is fixed. 👍

• We've been using rootless podman with slirp4netns and now with pasta. But this issue only exist with pasta. And I've only tried with two containers in the same host.
  So with my limited understanding it sounds like this is expected (at least for now until the mentioned partial solution might get implemented), but I don't know/understand why it is different with slirp4netns?

What's expected is that inbound connections will be first accepted, and then reset (instead of just being reset right away), but outbound connections should be properly reset (which, I think, is your actual issue here).

For inbound connections (from external hosts, that is, not loopback connections):

• pasta needs to connect to the service in the container to know if the client can actually connect, and, in order to connect, it's sending an actual Layer-2 (Ethernet) frame, with an IP (Layer-3) header, and TCP (Layer-4) information. It's a SYN segment.
  For that, it needs to know the source address of the client (because it preserves the source address), and, for that, it needs to accept() the connection. No accept(), no information about the client.
• the port forwarder you use in slirp4netns mode is, by default, rootlesskit/rootlessport (matching the port_handler=rootlesskit option, which is the default). That port forwarder doesn't preserve the source address, and it will not actually build a Ethernet frame (it just connect()s to the service in the container instead of building packets).
  So it doesn't need to accept() a connection to get the source address before connecting, it can just call connect() and, if that fails, not call accept().

Let me see if I understood you.

For Pasta & an inbound connection, pasta first accept it and later if needed (no valid response from the target application inside the container) reset it.
For my case, since it's container A -> container B, it's outbound for container A, but inbound for B, so it goes through the same as stated before for container B. Hence in order to send information about the client, it accept it, and later if needed resets it. => Which is what I see in Container A, that it is able to establish a connection to B, but the read fails, as it gets (presumably) reset by the server (or pasta in this case)

And this isn't done for slirp4netns, as it doesn't preserve the source address.

[sbrivio-rh]

For Pasta & an inbound connection, pasta first accept it and later if needed (no valid response from the target application inside the container) reset it. For my case, since it's container A -> container B, it's outbound for container A, but inbound for B, so it goes through the same as stated before for container B. Hence in order to send information about the client, it accept it, and later if needed resets it.

Correct. However, "later" should be as soon as connect() returns error, not much later. That is,

=> Which is what I see in Container A, that it is able to establish a connection to B, but the read fails, as it gets (presumably) reset by the server (or pasta in this case)

right now you get a failure a while later (on the first data transfer). The connection should be closed right away instead, that's
what was fixed for https://bugs.passt.top/show_bug.cgi?id=93.

And this isn't done for slirp4netns, as it doesn't preserve the source address.

Somewhat confusingly: slirp4netns itself would preserve the source address, but the port forwarding implementation that's selected in slirp4netns mode, by default, is not slirp4netns, it's rootlesskit / rootlessport, and that one doesn't keep the source address.

On top of that, it's a direct connect() in the container (Layer-4), instead of sending a SYN packet (building packet at Layer-2) to it, so that will fail or succeed right away.

[bollhals]

Just a quick feedback: I've managed to update it to podman 5.6.0 and pasta 2025_06_11.0293c6f, but nothing changed in how it is visible from my logs. It still connects and upon the first read being issued on the connnected socket, it fails with EndOfStream. It retries several times until the RabbitMQ container is fully up and has accepted the incoming connection.

[sbrivio-rh]

Then I guess the issue in your case is really that we accept() the connection (and reset it a moment later, probably causing that EndOfStream) regardless of whether the service is ready.

Coincidentally, we recently happened to discuss with @EdDev, @dgibson, @nirdothan a possibly related feature, intended for rather different purposes (Kubernetes TCP liveness probes in KubeVirt, using the passt network binding there) where passt (and pasta, here) would periodically probe a list of separately configured ports and only open listening sockets as those ports are actually listening in the container/guest.

We already have many bits of the feature ready, as pasta already implements automatic port forwarding based on periodic scanning of /proc/net/ entries, but there's no way to configure that for a selected range of ports, and there's no way to actually probe for readiness. If we ever get to implement that, you would have a setting directly addressing the issue at hand.

Contributions in that sense are warmly welcome. It's not really complicated to do, and I can provide guidance if needed.

By the way, you might try to pass -t auto to pasta (that's the default in pasta, but Podman disables it, so you would need to do something like podman run --net=pasta:t,auto ...). It probably won't fit your usage in general, but it would be interesting to see if it takes care of the issue at hand.