Inter-container communication using pasta

[shunkica]

What is the proper way to allow two rootless containers, ran by two different users on the same host, to communicate with each other on some port?

The only way I have figured out is to publish a port to the host, eg. -p "127.0.0.1:12345:12345 and then run the other container with --network=pasta:-T,12345

Is there a way to bypass this need for publishing a port to the host, and have containers talk to each other directly?

[Luap99]

Use --network bridge or create a new network and use that then the contianers both have a ip on the same network and can communicate directly via that ip.

[shunkica]

Use --network bridge or create a new network and use that then the contianers both have a ip on the same network and can communicate directly via that ip.

Maybe I am missing something but for me this only works when the containers are started by the same user.

My requirement is that the containers are running under different users. I am not a network engineer, but my understanding is that this should be possible in pasta?

 -  avoids the need for a generic, full-fledged TCP/IP stack by coordinating TCP connection dynamics between sender and receiver
 -  offers a fast bypass path for local connections: if a process connects to another process on the same host across namespaces, data is directly forwarded using pairs of Layer-4 sockets
 - with default options, maps routing and addressing information to the namespace, avoiding any need for NAT

[sbrivio-rh]

The "fast bypass path for local connections" is socket "splicing" (in the sense of the splice() system call) between a socket on the "host" (outside the container) and one inside the container. It's the orange appendage in this diagram, top left. That's what -T,12345 does, in your case.

At the moment, what you're doing is pretty much the best solution pasta can offer. We're working on a more flexible model for forwarding, based on a full-fledged flow table.

At that point, we can consider supporting multiple containers (network namespaces) within the same instance of pasta, and pasta could splice between separate containers directly. But I wouldn't expect much better throughput: what you have in your example is already entirely zero-copy. Latency would probably benefit, though.

[Luap99]

Use --network bridge or create a new network and use that then the contianers both have a ip on the same network and can communicate directly via that ip.

Maybe I am missing something but for me this only works when the containers are started by the same user.

Ah yes, sorry I overlooked the two different user part.

[marinmo]

I want to voice my concern a little re: this topic since your first reply, @Luap99 kinda hits a bit home and on a sore toe.

In the blog post discussing the change to pasta (https://blog.podman.io/2024/03/podman-5-0-breaking-changes-in-detail/), you write "While I do not think it is necessarily a breaking change for many [...]" ... But I promise I'm not the only one running a reverse proxy with podman using the default podman network - which does break due to this change. This change is huge and the fact that it's not really explicitly documented anywhere just boggles my mind?

Like, "assuming you don't explicitly pass --network or --pod to containers, this change will break port forwarding between them" just seems like something which should've raised a red flag somewhere along the way? The neat thing - and even advertised as a kind of selling point - about rootless podman/slirp4netns was/is that you didn't need to fiddle with IPs - just publish ports to all or a specific internal interface and be done with it! Pasta breaks this expected behaviour, and that's not a small change imo.

I can't be the only one running a database and an application behind a reverse proxy? Is it such an unusual use case? I don't even (yet) run them as separate users (which, as seen here, makes things even more complicated with pasta!).

Could the documentation somehow be updated to be clearer about this?

[Luap99]

Could the documentation somehow be updated to be clearer about this?

Sure but which documentation? Where would you actually look for this?

Does this setting not work for you?

[network]
pasta_options = ["-a", "10.0.2.0", "-n", "24", "-g", "10.0.2.2", "--dns-forward", "10.0.2.3"]

[marinmo]

I'd expect that to go into the blog post first and foremost! Either the solution you posted, or a lengthier explanation of what the consequences of containers not being able to access the host ip currently are and how to work around them. I'd also add some notes about this here: https://github.com/containers/podman/blob/main/rootless.md - I'll get to work on a PR after I post this.

Maybe it's rather intuitive for you - having worked with this, but for me (a rather hobbyist home admin), it wasn't really clear as day! Instead, it took me several hours to figure out - what changed, when, why - and I'm rather good at googling! Maybe I'm not the primary target audience here, still I found it very hard to understand what broke, and why - and in the end I found an issue here on github where people solved their problems by switching back to slirp4netns from pasta which I also ended up doing.

I'll follow pasta:s development though since I can see what would make it potentially superior. But I do think this change was a bit hasty considering pastas current shortcomings.

[Luap99]

Well that part is in the blog post with a explanation of it works and a description of your problem and how to fix it.

By default it will not be possible to connect to the host via the eth0 (or whatever your main interface is called) ip as the exact same ip is used in the container and thus is not routed to the outside. This can cause problems for users of the host.containers.internal name entry as we rely on the host ip being reachable. For Podman 5.0.0 it is likely that this entry will contain a invalid ip but we are working on a fix for Podman 5.0.1. However, the underlying problem will stay if you only have a single host ip (excluding localhost), as there would be no way to route to that if the container always uses the same ip. One workaround for that is to tell pasta to use a different address in the container. In this case set something like this in containers.conf:
...

[marinmo]

Guess it's clear as day now, but it sure wasn't at first - especially since this is a breaking change in how to think about inter container communications. I've submitted a PR for the rootless part of the documentation which - for me! - would've saved a lot of time :)