Separate pod network and caddy network

[HighPriest]

Hello @eriksjolund
Thank you for this amazing resource. With your work, I was able to get up-to-speed with Podman in a day of work!

To the point.
I am surprised that when a Network= is added to a container, which is part of a pod, the container gets removed from the pod network. 127.0.0.1:db_port no longer works.

To get communication caddy container -> service container -> db container, I had to put all these containers into a single network.
I was hoping that if I put a container on a network, it receives a separate network interface to only communicate with devices on that network. But it seems to not work like this.

Does it mean that a separation of concerns like this is not possible? Or have I missed something and we could create Example5 which works like this?

[eriksjolund]

Thank you for this amazing resource. With your work, I was able to get up-to-speed with Podman in a day of work!

Good to hear!

Which podman version are you using?
Are you running podman directly on Linux?
(I haven't really explored using Podman Desktop so things might be different when using Podman Desktop)

I am surprised that when a Network= is added to a container, which is part of a pod, the container gets removed from the pod network. 127.0.0.1:db_port no longer works.

I think when using a pod, all network configuration needs to be applied to the pod instead of to the container.

In other words, use Network=

[Pod]
Network=...

instead of Network=

[Container]
Network=...

(I haven't really investigated using pods with custom networks, so the advice above is untested).

Instead of using a pod, you could also use multiple custom networks.
You can add them like this

[Container]
Network=mynet1.network
Network=mynet2.network

and use Options=isolate=true like this

file mynet1.network

[Network]
Options=isolate=true

file mynet2.network

[Network]
Options=isolate=true

One advantage of not using a pod, is that you then can specify different UID/GID mappings for each container. When using a pod, you need to use the same UID/GID mapping for all containers in the pod.

Specifying different UID/GID mappings for each container can be important in situations when
you bind-mount directories into containers and you would like to control which host UID/GID is used as owner for files written to those directories.

[eriksjolund]

Here is a skecth of how you could use multiple custom networks instead of a pod:

file service.network

[Network]
Options=isolate=true

file caddy.network

[Network]
Options=isolate=true

file caddy.container

[Container]
Network=caddy.network
...

file service.container

[Container]
Network=caddy.network
Network=service.network
...

file database.container

[Container]
Network=service.network
...

---

If the database container does not need to connect to the internet, you could add Internal=true to the service.network

file service.network

[Network]
Options=isolate=true
Internal=true

[HighPriest]

Thank you for such a thorough response.
You are right, when two containers are in a Pod, the only option to extend their networking is by adding the Network parameters to the Pod quadlet files.
What is unfortunate, adding the Pod to an external network, tears down the internal Pod network and makes the external Networks the only way to communicate between containers. This is the exact same behaviour that happens when one of the containers in a Pod is added to an external Network. The internal Pod Network is also torn down.

The only option i see here is to ... manually ... create separate networks for pods and for service communication with the proxy.

So, in configs it would look like:
caddy.container

[Container]
Network=caddy.network

service.container

[Container]
Network=caddy.network
Network=service.network

service.db.container

[Container]
Network=service.network

Mind that, I have not tested this approach.
It is a shame that individual containers in a Pod cannot be safely network to other containers. There must be a good reason for this, or we are missing something.

[redbeardymcgee]

I ran into the same issue with dozens of database and redis/valkey instances supporting various container services with web frontends. Every frontend needed to include Network=reverse-proxy.network and an additional Network=backend-xyz.network. Then, each database needed to only be configured with Network=backend.network. This enabled me to avoid port collisions on reverse-proxy.network, but resulted in losing the source address from incoming requests which would then break services like searxng because it relies on X-Forwarded-For to implement a bot limiter. Other services also expect real client IPs for various reasons.

I did not even try to approach pods yet because of the complication you described here. I switched to just Network=pasta to overcome the source address preservation issue, but it reintroduces the same problem as pod networks in this thread; Network=pasta is incompatible with Network=backend-abc.network such that the containers cannot start due to the conflict between pasta type and bridge type networks, resulting in every container sharing the same network and conflicting endlessly.

This is technically solved by configuring each database instance with a new listening port, but that has a hard limit of ~65000. That's probably fine, but it's also extremely tedious and feels like it defeats much of the purpose. Besides, many containers or apps are not designed with this in mind. searxng, for example, does not allow binding on a different interface or port with either its settings.yml or env vars when deployed behind a reverse proxy.

[eriksjolund]

@redbeardymcgee If you are running Linux (with systemd) then you could probably follow example4.
If you need more network isolation between containers on the custom network, you could create multiple custom networks.

In other words, instead of using only one custom network mynet, create mynet1, mynet2, mynet3 .... and so on.

In the file caddy.container add

Network=mynet1.network
Network=mynet2.network
Network=mynet3.network

under the [Container] section.

If you create myservice1.container and myservice2.container and let both of them be connected to
just one custom network (for instance mynet1), then it is almost as if the two containers are running in a pod.

There is one major difference though, the two containers could connect to each other by using each other's container name but not by using 127.0.0.1. Those names are resolved by the internal DNS.

[redbeardymcgee]

Thank you for your thoughtful response. I am trying to work through the example given already, because the socket-activation preserves the source address but doesn't have the same limitation of custom networks that pasta does (either loss of source address which can run, or conflict between pasta & bridge types which cannot run at all).

I would have liked to explore pods a little more for my project at redbeardymcgee/podbox but it will have to wait until I get more documentation written. I also find connecting by container name to be more intuitive and useful than the loopback address anyway. Is there something about this which is detrimental? Maybe a performance issue or something? I haven't quite fully understood why I would prefer a pod over a custom network yet.

[eriksjolund]

Is there something about this which is detrimental?

I am not aware of any.

Maybe a performance issue or something?

Yes, maybe. I am also curious if there is any performance advantage of using a pod compared to using a custom network.

On the other hand there is a performance advantage of using socket activation. For details, see Native network performance over the socket-activated socket

I haven't quite fully understood why I would prefer a pod over a custom network yet.

Here is a possible advantage: Using pods might be more convenient if you already have created such pods for running in kubernetes.

[HighPriest]

I don't have concrete benchmarks, but there is always going to be a difference between direct IP networking (requesting just 127.0.0.1 with a port) and requesting a connection through some DNS system (requesting an example.service.com and exchanging information through received link).

There is a whole movement against using localhost in place of 127.0.0.1, because it can result in performance degradation.