podman-compose based set-up on RHEL 9 looses network connectivity after some time without error messages

[EHJ-52n]

I have a NOT rootless set-up that is migrated from docker-compose to podman-compose.
The set-up is using one database service and two APIs using the database, hence nothing complicated in terms of networking and stuff, from my point of view.
The migration was triggered because the docker-compose set-up on RHEL9 lost network connectivity over time without any hint why.
Now, this is happening in the podman-compose set-up, too.
I found some old issues for podman and lost network connectivity over time, but they are outdated, and marked as fixed or dealing with old network plugins (e.g. #12495).
When the connectivity is lost and my services are not reachable anymore (pods still running), I just need to execute the following command, and the network is working again: sudo podman run -p 80:80 --rm nginx.
The nftables ruleset is not changed and journalctl does not show any errors for NetworkManager for example.

Do you have any hint, were to look for error messages or hints?

What additional information do you need?

[Luap99]

Are you using firewalld? If that is the case when you execute firewall-cmd --reload do you loose the network connectivity?

[EHJ-52n]

firewalld.service is disabled and dead.

[Luap99]

What exactly do you mean with "looses network connectivity"? Full ip connectivity, dns only, etc...? Did you try to setup packages captures and see where things get dropped?

If the nft ruleset really didn't change then I do not really see what else would trigger this. You can check for the proper interfaces in ip link (bridge and veth interfaces for the networks) and addresses with ip addr. But I don't see how they would be deleted other than doing that explicitly and not why it should work after a single podman run again.

Unless you can provide an exact reproducer I don't think I can help. Keep in mind that neither podman-compose or docker-compose are directly supported by us (though I would assume the way things are created shouldn't matter here anyway) If you are RHEL customer I suggest you context the Red Hat support with issues in RHEL.

[EHJ-52n]

@Luap99 thank you very much for your time on this very generic issue!

What exactly do you mean with "looses network connectivity"? Full ip connectivity, dns only, etc...?

I mean with "looses network connectivity", that the services are not reachable from outside the hosting VM.
On the VM itself, the services are still reachable.
Other processes running in podman pods cannot reach the internet and the aforementioned service.
There error messages are timeouts and nothing like network unreachable or cannot resolve hostname.

Did you try to setup packages captures and see where things get dropped?

How can I achieve this with podman on RHEL?

If the nft ruleset really didn't change then I do not really see what else would trigger this. You can check for the proper interfaces in ip link (bridge and veth interfaces for the networks) and addresses with ip addr. But I don't see how they would be deleted other than doing that explicitly and not why it should work after a single podman run again.

The adresses and interface/bridge names from the ruleset and ip addr show are matching.

My next step is to set-up a monitoring service on another VM to identify more exactly the time, when the connection is "lost".

[EHJ-52n]

It looks like it was caused by an ansible task, that disables net.ipv4.ip_forward every six hours. In addition, podman and its network plugins enable this without any error or warning message everytime you start a new pod or execute podman network reload --all (see #399 (comment)).

I think that it might be a good idea to make a log statement on any useful level, that a basic kernel parameter was adjusted on the fly.

[Luap99]

enable this without any error or warning message everytime

sure because by default that is off on first boot so throwing a warning by default makes no sense for the sysctl.

We do log Setting sysctl value for net.ipv4.ip_forward to 1 on debug level but that happens every time AFAIK and not just when the value is actually changed.