Number of network connections limited on Mac (apple silicon)?

[cschaible]

Hi, I'm developing a distributed application with a few microservices that are connected to databases (Apache Kafka, Jaeger, etc.). When I start the application on a Mac (with M2) and run a loadtest (inserting into / reading from a db-table) , I see after some seconds (when the load increases) failing requests with the error message Connection Refused.

The microservices are running on the host, the databases are running in a container in the podman machine.
When the loadtest reaches ~400 requests/second the errors start to occur. It doesn't really matter whether i use 3 or 10 concurrent users in the loadtest - always the same result. I could imagine that there are more active network connections (as I also send traces to Jaeger).
Not all requests are failing - i guess it's caused by the load that goes beyond than the ~400 requests/second.

On another mac (with m1) with podman, the same problem occurs. If the load stays below 400 req/sec on both macs the loadtests succeed.

When I look into PgAdmin on the Mac I also see network errors while the loadtest is running - once the loadtest is finished, PgAdmin recovers and displays data again.
I tested the application on a less powerful linux machine (with docker) generating higher load (800 req/sec) and didn't see any issues.

This leads me to the assumption that there must be a limit or an issue in podman (podman-machine or the interaction between the OS and the podman-machine) that causes these errors.
I'm using podman "rootless" and already tried to increase the "Max open files" limit from 1024 to 100000 without success.

[core@localhost ~]$ cat /proc/self/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             30350                30350                processes 
Max open files            100000               524288               files     
Max locked memory         8388608              8388608              bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       30350                30350                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

I would appreciate to know if there's a known limitation or issue / explanation how to solve the issue?

Thanks in advance!

[mheon]

Can you try with --net=host and see if you observe the same limitation? I suspect this could be slirp4netns causing the bottleneck.

[cschaible]

Thanks for the fast reply. I cannot start the containers with --net=host as it does not seem to be possible to set network_mode: host in podman-compose currently. According to podman-compose #530 a fix isn't released yet (it's only on devel branch) which I cannot install on my machine. Thank you for your help anyway.