Podman socket not answering

[gaeljw]

Hello folks,

I'm investigating a difference of behaviour between some of our servers regarding the usage of Podman socket and would appreciate any help on the topic.

(The full context is that we're using testcontainers library that tries to run some containers and fail to do it in some servers)

✅ On a server where everything works :

# As the user owning the socket (docker.sock is a symbolic link to podman/podman.sock)
curl -v --unix-socket /run/user/1000/docker.sock 'http://localhost:2375/v1.32/info'

Output:

*   Trying /run/user/1000/docker.sock:0...
* Connected to localhost (/run/user/1000/podman/podman.sock) port 2375 (#0)
> GET /v1.32/info HTTP/1.1
> Host: localhost:2375
> User-Agent: curl/7.76.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Api-Version: 1.41
< Content-Type: application/json
< Libpod-Api-Version: 4.6.1
< Server: Libpod/4.6.1 (linux)
< X-Reference-Id: 0xc0005c26c0
< Date: Tue, 14 Nov 2023 08:18:26 GMT
< Transfer-Encoding: chunked
<
{"ID":"770f2014-1ebe-4b79-ba98-1e90e65cfa9d","Containers":0,"ContainersRunning":0,"ContainersPaused":0,"ContainersStopped":0,"Images":137,"Driver":"overlay","DriverStatus":[["Native Overlay Diff","true"],["Using metacopy","false"],["Backing Filesystem","extfs"],["Supports d_type","true"]],"Plugins":{"Volume":["local"],"Network":["bridge","macvlan","ipvlan"],"Authorization":null,"Log":["k8s-file","none","passthrough","journald"]},"MemoryLimit":true,"SwapLimit":false,"CpuCfsPeriod":false,"CpuCfsQuota":false,"CPUShares":false,"CPUSet":false,"PidsLimit":true,"IPv4Forwarding":false,"BridgeNfIptables":false,"BridgeNfIp6tables":false,"Debug":false,"NFd":11,"OomKillDisable":false,"NGoroutines":13,"SystemTime":"2023-11-14T08:18:26.215129462Z","LoggingDriver":"","CgroupDriver":"systemd","NEventsListener":0,"KernelVersion":"5.14.0-373.el9.x86_64","OperatingSystem":"\"centos\"","OSVersion":"9","OSType":"linux","Architecture":"amd64","IndexServerAddress":"","RegistryConfig":{"AllowNondistributableArtifactsCIDRs":[],"AllowNondistributableArtifactsHostnames":[],"InsecureRegistryCIDRs":[],"IndexConfigs":{},"Mirrors":[]},"NCPU":8,"MemTotal":8045076480,"GenericResources":null,"DockerRootDir":"/opt/jenkins-agent/.local/share/containers/storage","HttpProxy":"","HttpsProxy":"","NoProxy":"","Name":"srvok.mycompany.net","Labels":null,"ExperimentalBuild":true,"ServerVersion":"4.6.1","Runtimes":{"crun":{"path":"/usr/bin/crun"},"crun-wasm":{"path":"/usr/bin/crun-wasm"},"kata":{"path":"/usr/bin/kata-runtime"},"krun":{"path":"/usr/bin/krun"},"ocijail":{"path":"/usr/local/bin/ocijail"},"runc":{"path":"/usr/bin/runc"},"runj":{"path":"/usr/local/bin/runj"},"runsc":{"path":"/usr/bin/runsc"},"youki":{"path":"/usr/local/bin/youki"}},"DefaultRuntime":"crun","Swarm":{"NodeID":"","NodeAddr":"","LocalNodeState":"inactive","ControlAvailable":false,"Error":"","RemoteManagers":null},"LiveRestoreEnabled":false,"Isolation":"","InitBinary":"","ContainerdCommit":{"ID":"","Expected":""},"RuncCommit":{"ID":"","Expected":""},"InitCommit":{"ID":"","Expected":""},"SecurityOptions":["name=seccomp,profile=default","name=rootless"],"ProductLicense":"Apache-2.0","Warnings":[],"BuildahVersion":"1.31.3","CPURealtimePeriod":false,"CPURealtimeRuntime":false,"CgroupVersion":"2","Rootless":true,"SwapFree":6189416448,"SwapTotal":6442446848,"Uptime":"695h 41m 45.00s (Approximately 28.96 days)"}
* Connection #0 to host localhost left intact
*

🟥 On a server where it doesn't work, the curl waits forever for a response:

# As the user owning the socket (docker.sock is a symbolic link to podman/podman.sock)
curl -v --unix-socket /run/user/1000/docker.sock 'http://localhost:2375/v1.32/info'

Output:

*   Trying /run/user/1000/docker.sock:0...
* Connected to localhost (/run/user/1000/podman/podman.sock) port 2375 (#0)
> GET /v1.32/info HTTP/1.1
> Host: localhost:2375
> User-Agent: curl/7.76.1
> Accept: */*
>
^C

I don't know where to investigate next, there's likely a difference in the installation between the two servers but how to find it?!

Thanks for any help on the matter,

---

On both servers:

podman --version
podman version 4.6.1

[gaeljw]

For anyone coming here, it seems this was caused by a mess in ownership (root user vs. a specific user).

And/or a container that seem to hog the socket for some reason (still investigating this).