I don't know how to move forward: dropped packets for localhost web server hosted on rootless container on AMD Macs, not observed on WSL Windows.

[JOSEPHGILBY]

Contaner architecture summary (details shared below):

• Rootless mode
• 1 pod
• 2 containers
  • 1 Microsoft sql-server
  • 1 Microsoft development container (allows VSCode to open inside the container and develop inside)

I've observed this issue on two separate Macs and not on podman running on Windows WSL. Basically, we're running a webserver over localhost inside the development container. Intermittently, but not always, the browser will fail to load a webpage because of net::ERR CONTENT LENGTH MISMATCH thrown for one of the static files (html, css, js). Taking a look at wireshark for one of these failed file downloads, the last pieces of data don't seem to have been sent at all. However, no indication of a failed download is sent and the FIN packets are sent and acknowledged in the transmission.

The webserver we run is PHP's webserver. To debug the issue further, I installed PHP's source and modified the source to check and make sure every byte of the static file is read (POSIX read) from the filestore and then sent (POSIX send) to the network with no errors (<0 returned). As such, no errors were thrown and every byte of the static file was read and sent during a failed file download.

With this information, I'm not sure how to move forward. Are there any additional tools you might suggest to continue debugging this issue? Trying to read past POSIX read() and send() commands I run into the container's kernel which could mean the issue lies with Microsoft's dev container implementation, podman container / pod / machine implementation, or something else. At this time, i suspect the issue could lie in podman since the packet drops haven't been observed in Windows over WSL, but I'm not sure what tools I could use to continue looking at this issue. Any ideas?

podman pod inspect (names scrambled)

{
     "Id": "759c4d63f3c8d1264f871790287c78f3082a671e7b23e440ad82f17418fc87cb",
     "Name": "dev-pod",
     "Created": "2022-05-11T11:12:37.310823419-05:00",
     "CreateCommand": [
          "podman",
          "pod",
          "create",
          "--name",
          "dev-pod",
          "--userns=keep-id",
          "--publish",
          "5434:1433"
     ],
     "State": "Running",
     "Hostname": "",
     "CreateCgroup": true,
     "CgroupParent": "user.slice",
     "CgroupPath": "user.slice/user-libpod_pod_759c4d63f3c8d1264f871790287c78f3082a671e7b23e440ad82f17418fc87cb.slice",
     "CreateInfra": true,
     "InfraContainerID": "8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d",
     "InfraConfig": {
          "PortBindings": {
               "1433/tcp": [
     {
          "HostIp": "",
          "HostPort": "5434"
     }
]
          },
          "HostNetwork": false,
          "StaticIP": "",
          "StaticMAC": "",
          "NoManageResolvConf": false,
          "DNSServer": null,
          "DNSSearch": null,
          "DNSOption": null,
          "NoManageHosts": false,
          "HostAdd": null,
          "Networks": [
               "podman"
          ],
          "NetworkOptions": null,
          "pid_ns": "private",
          "userns": "private"
     },
     "SharedNamespaces": [
          "user",
          "uts",
          "ipc",
          "net"
     ],
     "NumContainers": 3,
     "Containers": [
          {
               "Id": "399afeb6864e429a1d5155e39c7dfe0e0b90a4e2b6593122b710f7b3e5ab4d55",
               "Name": "mssql",
               "State": "running"
          },
          {
               "Id": "8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d",
               "Name": "759c4d63f3c8-infra",
               "State": "running"
          },
          {
               "Id": "979cbf0483ab90d3bf151f489ced8a5c04200cc04c27fc625cb715b15df7599c",
               "Name": "priceless_babbage",
               "State": "running"
          }
     ]
}

podman container inspect mssql

[
     {
          "Id": "399afeb6864e429a1d5155e39c7dfe0e0b90a4e2b6593122b710f7b3e5ab4d55",
          "Created": "2022-05-11T11:14:46.100623603-05:00",
          "Path": "/opt/mssql/bin/permissions_check.sh",
          "Args": [
               "/opt/mssql/bin/sqlservr"
          ],
          "State": {
               "OciVersion": "1.0.2-dev",
               "Status": "running",
               "Running": true,
               "Paused": false,
               "Restarting": false,
               "OOMKilled": false,
               "Dead": false,
               "Pid": 1215,
               "ConmonPid": 1211,
               "ExitCode": 0,
               "Error": "",
               "StartedAt": "2022-05-12T10:23:03.221725244-05:00",
               "FinishedAt": "0001-01-01T00:00:00Z",
               "Health": {
                    "Status": "",
                    "FailingStreak": 0,
                    "Log": null
               },
               "CgroupPath": "/user.slice/user-501.slice/[email protected]/user.slice/user-libpod_pod_759c4d63f3c8d1264f871790287c78f3082a671e7b23e440ad82f17418fc87cb.slice/libpod-399afeb6864e429a1d5155e39c7dfe0e0b90a4e2b6593122b710f7b3e5ab4d55.scope",
               "CheckpointedAt": "0001-01-01T00:00:00Z",
               "RestoredAt": "0001-01-01T00:00:00Z"
          },
          "Image": "d78e982c2f2bdabbf34c3fd1b898ade9a25a22a5872ef23ea0c4fde1e6e3d7c8",
          "ImageName": "mcr.microsoft.com/mssql/server:2019-latest",
          "Rootfs": "",
          "Pod": "759c4d63f3c8d1264f871790287c78f3082a671e7b23e440ad82f17418fc87cb",
          "ResolvConfPath": "/run/user/501/containers/overlay-containers/8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d/userdata/resolv.conf",
          "HostnamePath": "/run/user/501/containers/overlay-containers/399afeb6864e429a1d5155e39c7dfe0e0b90a4e2b6593122b710f7b3e5ab4d55/userdata/hostname",
          "HostsPath": "/run/user/501/containers/overlay-containers/8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d/userdata/hosts",
          "StaticDir": "/var/home/core/.local/share/containers/storage/overlay-containers/399afeb6864e429a1d5155e39c7dfe0e0b90a4e2b6593122b710f7b3e5ab4d55/userdata",
          "OCIConfigPath": "/var/home/core/.local/share/containers/storage/overlay-containers/399afeb6864e429a1d5155e39c7dfe0e0b90a4e2b6593122b710f7b3e5ab4d55/userdata/config.json",
          "OCIRuntime": "crun",
          "ConmonPidFile": "/run/user/501/containers/overlay-containers/399afeb6864e429a1d5155e39c7dfe0e0b90a4e2b6593122b710f7b3e5ab4d55/userdata/conmon.pid",
          "PidFile": "/run/user/501/containers/overlay-containers/399afeb6864e429a1d5155e39c7dfe0e0b90a4e2b6593122b710f7b3e5ab4d55/userdata/pidfile",
          "Name": "mssql",
          "RestartCount": 0,
          "Driver": "overlay",
          "MountLabel": "system_u:object_r:container_file_t:s0:c265,c814",
          "ProcessLabel": "system_u:system_r:container_t:s0:c265,c814",
          "AppArmorProfile": "",
          "EffectiveCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "BoundingCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "ExecIDs": [],
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/var/home/core/.local/share/containers/storage/overlay/9473945ff30ca5a3da0cb9205c5e1bbb71556b63ea3bab238744fe95da5edd0f/diff:/var/home/core/.local/share/containers/storage/overlay/5a080831a8ed585226b815c3e66e2748f5f1d7a5a83e19f01618094cfa96e7cb/diff:/var/home/core/.local/share/containers/storage/overlay/cfcd7279f715d639348cdfb562dcd5345db077c312bab6f2328d27d3b5e666a9/diff:/var/home/core/.local/share/containers/storage/overlay/2e6a05f12a8abe4312de2c37421439c8d858f34f9b44e250e5422a2bcf0132f1/diff:/var/home/core/.local/share/containers/storage/overlay/b52fe7d4db2edac566913df4417bf41656b5ea27c20c3dd41d164f2eeddbf397/diff:/var/home/core/.local/share/containers/storage/overlay/0eba131dffd015134cb310c284b776c1e44d330146cd2f0e30c4e464d0b76d24/diff",
                    "MergedDir": "/var/home/core/.local/share/containers/storage/overlay/2de87ba41b1a36efee8d7207842377548c70b2bf4b1eb4438774e8c29f2c1f9c/merged",
                    "UpperDir": "/var/home/core/.local/share/containers/storage/overlay/2de87ba41b1a36efee8d7207842377548c70b2bf4b1eb4438774e8c29f2c1f9c/diff",
                    "WorkDir": "/var/home/core/.local/share/containers/storage/overlay/2de87ba41b1a36efee8d7207842377548c70b2bf4b1eb4438774e8c29f2c1f9c/work"
               }
          },
          "Mounts": [
               {
                    "Type": "volume",
                    "Name": "mssql-pem",
                    "Source": "/var/home/core/.local/share/containers/storage/volumes/mssql-pem/_data",
                    "Destination": "/etc/ssl/certs",
                    "Driver": "local",
                    "Mode": "",
                    "Options": [
                         "nosuid",
                         "nodev",
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "volume",
                    "Name": "mssql-key",
                    "Source": "/var/home/core/.local/share/containers/storage/volumes/mssql-key/_data",
                    "Destination": "/etc/ssl/private",
                    "Driver": "local",
                    "Mode": "",
                    "Options": [
                         "nosuid",
                         "nodev",
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "volume",
                    "Name": "mssql-conf",
                    "Source": "/var/home/core/.local/share/containers/storage/volumes/mssql-conf/_data",
                    "Destination": "/var/opt/mssql",
                    "Driver": "local",
                    "Mode": "",
                    "Options": [
                         "nosuid",
                         "nodev",
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               }
          ],
          "Dependencies": [
               "8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d"
          ],
          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "10.88.0.1",
               "IPAddress": "10.88.0.2",
               "IPPrefixLen": 16,
               "IPv6Gateway": "",
               "GlobalIPv6Address": "",
               "GlobalIPv6PrefixLen": 0,
               "MacAddress": "f6:0e:44:2e:89:27",
               "Bridge": "",
               "SandboxID": "",
               "HairpinMode": false,
               "LinkLocalIPv6Address": "",
               "LinkLocalIPv6PrefixLen": 0,
               "Ports": {
                    "1433/tcp": [
                         {
                              "HostIp": "",
                              "HostPort": "5434"
                         }
                    ]
               },
               "SandboxKey": "/run/user/501/netns/netns-9153807d-7183-224d-49f2-1ad410ddcaa6",
               "Networks": {
                    "podman": {
                         "EndpointID": "",
                         "Gateway": "10.88.0.1",
                         "IPAddress": "10.88.0.2",
                         "IPPrefixLen": 16,
                         "IPv6Gateway": "",
                         "GlobalIPv6Address": "",
                         "GlobalIPv6PrefixLen": 0,
                         "MacAddress": "f6:0e:44:2e:89:27",
                         "NetworkID": "podman",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "8734a8a94c63"
                         ]
                    }
               }
          },
          "Namespace": "",
          "IsInfra": false,
          "Config": {
               "Hostname": "dev-pod",
               "Domainname": "",
               "User": "0:0",
               "AttachStdin": false,
               "AttachStdout": false,
               "AttachStderr": false,
               "Tty": false,
               "OpenStdin": false,
               "StdinOnce": false,
               "Env": [
                    "container=podman",
                    "ACCEPT_EULA=Y",
                    "MSSQL_SA_PASSWORD=Succ9ssREVKEEP",
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "TERM=xterm",
                    "HOME=/root",
                    "HOSTNAME=dev-pod"
               ],
               "Cmd": [
                    "/opt/mssql/bin/sqlservr"
               ],
               "Image": "mcr.microsoft.com/mssql/server:2019-latest",
               "Volumes": null,
               "WorkingDir": "/",
               "Entrypoint": "/opt/mssql/bin/permissions_check.sh",
               "OnBuild": null,
               "Labels": {
                    "com.microsoft.product": "Microsoft SQL Server",
                    "com.microsoft.version": "15.0.4198.2",
                    "vendor": "Microsoft"
               },
               "Annotations": {
                    "io.container.manager": "libpod",
                    "io.kubernetes.cri-o.ContainerType": "container",
                    "io.kubernetes.cri-o.Created": "2022-05-11T11:14:46.100623603-05:00",
                    "io.kubernetes.cri-o.SandboxID": "dev-pod",
                    "io.kubernetes.cri-o.TTY": "false",
                    "io.podman.annotations.autoremove": "FALSE",
                    "io.podman.annotations.init": "FALSE",
                    "io.podman.annotations.privileged": "FALSE",
                    "io.podman.annotations.publish-all": "FALSE",
                    "org.opencontainers.image.stopSignal": "15"
               },
               "StopSignal": 15,
               "CreateCommand": [
                    "podman",
                    "run",
                    "-e",
                    "ACCEPT_EULA=Y",
                    "-e",
                    "MSSQL_SA_PASSWORD=Succ9ssREVKEEP",
                    "--pod=dev-pod",
                    "-u",
                    "0:0",
                    "--name",
                    "mssql",
                    "-v",
                    "mssql-conf:/var/opt/mssql",
                    "-v",
                    "mssql-pem:/etc/ssl/certs",
                    "-v",
                    "mssql-key:/etc/ssl/private",
                    "-d",
                    "mcr.microsoft.com/mssql/server:2019-latest"
               ],
               "Umask": "0022",
               "Timeout": 0,
               "StopTimeout": 10,
               "Passwd": true
          },
          "HostConfig": {
               "Binds": [
                    "mssql-pem:/etc/ssl/certs:rw,rprivate,nosuid,nodev,rbind",
                    "mssql-key:/etc/ssl/private:rw,rprivate,nosuid,nodev,rbind",
                    "mssql-conf:/var/opt/mssql:rw,rprivate,nosuid,nodev,rbind"
               ],
               "CgroupManager": "systemd",
               "CgroupMode": "private",
               "ContainerIDFile": "",
               "LogConfig": {
                    "Type": "journald",
                    "Config": null,
                    "Path": "",
                    "Tag": "",
                    "Size": "0B"
               },
               "NetworkMode": "container:8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d",
               "PortBindings": {},
               "RestartPolicy": {
                    "Name": "",
                    "MaximumRetryCount": 0
               },
               "AutoRemove": false,
               "VolumeDriver": "",
               "VolumesFrom": null,
               "CapAdd": [],
               "CapDrop": [
                    "CAP_AUDIT_WRITE",
                    "CAP_MKNOD",
                    "CAP_NET_RAW"
               ],
               "Dns": [],
               "DnsOptions": [],
               "DnsSearch": [],
               "ExtraHosts": [],
               "GroupAdd": [],
               "IpcMode": "container:8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d",
               "Cgroup": "",
               "Cgroups": "default",
               "Links": null,
               "OomScoreAdj": 0,
               "PidMode": "private",
               "Privileged": false,
               "PublishAllPorts": false,
               "ReadonlyRootfs": false,
               "SecurityOpt": [],
               "Tmpfs": {},
               "UTSMode": "container:8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d",
               "UsernsMode": "container:8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d",
               "IDMappings": {
                    "UidMap": [
                         "0:1:501",
                         "501:0:1",
                         "502:502:999499"
                    ],
                    "GidMap": [
                         "0:1:1000",
                         "1000:0:1",
                         "1001:1001:999000"
                    ]
               },
               "ShmSize": 65536000,
               "Runtime": "oci",
               "ConsoleSize": [
                    0,
                    0
               ],
               "Isolation": "",
               "CpuShares": 0,
               "Memory": 0,
               "NanoCpus": 0,
               "CgroupParent": "user.slice/user-libpod_pod_759c4d63f3c8d1264f871790287c78f3082a671e7b23e440ad82f17418fc87cb.slice",
               "BlkioWeight": 0,
               "BlkioWeightDevice": null,
               "BlkioDeviceReadBps": null,
               "BlkioDeviceWriteBps": null,
               "BlkioDeviceReadIOps": null,
               "BlkioDeviceWriteIOps": null,
               "CpuPeriod": 0,
               "CpuQuota": 0,
               "CpuRealtimePeriod": 0,
               "CpuRealtimeRuntime": 0,
               "CpusetCpus": "",
               "CpusetMems": "",
               "Devices": [],
               "DiskQuota": 0,
               "KernelMemory": 0,
               "MemoryReservation": 0,
               "MemorySwap": 0,
               "MemorySwappiness": 0,
               "OomKillDisable": false,
               "PidsLimit": 2048,
               "Ulimits": [],
               "CpuCount": 0,
               "CpuPercent": 0,
               "IOMaximumIOps": 0,
               "IOMaximumBandwidth": 0,
               "CgroupConf": null
          }
     }
]

podman container inspect priceless_babbage

[
     {
          "Id": "979cbf0483ab90d3bf151f489ced8a5c04200cc04c27fc625cb715b15df7599c",
          "Created": "2022-05-11T11:33:19.364915268-05:00",
          "Path": "/bin/sh",
          "Args": [
               "-c",
               "echo Container started\ntrap \"exit 0\" 15\n\nexec \"$@\"\nwhile sleep 1 & wait $!; do :; done",
               "-"
          ],
          "State": {
               "OciVersion": "1.0.2-dev",
               "Status": "running",
               "Running": true,
               "Paused": false,
               "Restarting": false,
               "OOMKilled": false,
               "Dead": false,
               "Pid": 1543465,
               "ConmonPid": 1543461,
               "ExitCode": 0,
               "Error": "",
               "StartedAt": "2022-05-16T16:03:01.883057557-05:00",
               "FinishedAt": "2022-05-16T16:01:02.192211798-05:00",
               "Health": {
                    "Status": "",
                    "FailingStreak": 0,
                    "Log": null
               },
               "CgroupPath": "/user.slice/user-501.slice/[email protected]/user.slice/user-libpod_pod_759c4d63f3c8d1264f871790287c78f3082a671e7b23e440ad82f17418fc87cb.slice/libpod-979cbf0483ab90d3bf151f489ced8a5c04200cc04c27fc625cb715b15df7599c.scope",
               "CheckpointedAt": "0001-01-01T00:00:00Z",
               "RestoredAt": "0001-01-01T00:00:00Z"
          },
          "Image": "43ccfaddfedab2d9fc4cdc2c46303b278e466923c6c1543d2947007794f9c70b",
          "ImageName": "localhost/vsc-php-8.1.3-src-dev-b8388b615861eb5c89013a8fd00b03f5:latest",
          "Rootfs": "",
          "Pod": "759c4d63f3c8d1264f871790287c78f3082a671e7b23e440ad82f17418fc87cb",
          "ResolvConfPath": "/run/user/501/containers/overlay-containers/8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d/userdata/resolv.conf",
          "HostnamePath": "/run/user/501/containers/overlay-containers/979cbf0483ab90d3bf151f489ced8a5c04200cc04c27fc625cb715b15df7599c/userdata/hostname",
          "HostsPath": "/run/user/501/containers/overlay-containers/8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d/userdata/hosts",
          "StaticDir": "/var/home/core/.local/share/containers/storage/overlay-containers/979cbf0483ab90d3bf151f489ced8a5c04200cc04c27fc625cb715b15df7599c/userdata",
          "OCIConfigPath": "/var/home/core/.local/share/containers/storage/overlay-containers/979cbf0483ab90d3bf151f489ced8a5c04200cc04c27fc625cb715b15df7599c/userdata/config.json",
          "OCIRuntime": "crun",
          "ConmonPidFile": "/run/user/501/containers/overlay-containers/979cbf0483ab90d3bf151f489ced8a5c04200cc04c27fc625cb715b15df7599c/userdata/conmon.pid",
          "PidFile": "/run/user/501/containers/overlay-containers/979cbf0483ab90d3bf151f489ced8a5c04200cc04c27fc625cb715b15df7599c/userdata/pidfile",
          "Name": "priceless_babbage",
          "RestartCount": 0,
          "Driver": "overlay",
          "MountLabel": "system_u:object_r:container_file_t:s0:c265,c814",
          "ProcessLabel": "system_u:system_r:container_t:s0:c265,c814",
          "AppArmorProfile": "",
          "EffectiveCaps": [
               "CAP_SYS_PTRACE"
          ],
          "BoundingCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT",
               "CAP_SYS_PTRACE"
          ],
          "ExecIDs": [
               "03de326616874410f3f9bd8c01d4c801f78bd0566f027c2b67b0bfe0f7c879cb",
               "232990f6f3fb1161b367ff824c850404974ed24ff65d1d4afa745813468e8186",
               "b01c896a84c363584a64f2a36cb52b6cd1d1e8031d8b200231e190d979994b13",
               "b764c5ac8c5a22d0da890ff5e24a79a4fe4f26524274279bcb6853cecc8d3a3c",
               "c0c806882e1f6679222bf36c0dce6d19996a76f5a98e8dfc113ac4f7160c49f9",
               "1705de0d47d9ce453038ec82b03ebdf61ac93519a7d5b3aea9d08e602d657654",
               "4e041a65df81d36672cbf19edbfbf41c4984169750e804cb6a23cb040b34f48d",
               "4e8ce16c4c570f453ec631c50ac33bef6269fd5c759305ca81503e81a30ff1ac",
               "8355a11e03b02103acafc200682827f6182249afdcdc3f9f84425e40d28ddd4a",
               "a5db5b2220f1e8582ab0db61adab0b745063e40a681e3e2cae3f389972b78f5d"
          ],
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/var/home/core/.local/share/containers/storage/overlay/0a925ceb53ed8e40535e9398be42f338c1b49647c62dd6dab6bc6ab6509614b1/diff:/var/home/core/.local/share/containers/storage/overlay/0714fa8c72e2955c5681bf753add08ede66a375bdca553c3bba0b11bdbffb6ec/diff:/var/home/core/.local/share/containers/storage/overlay/f6048a819a5b33a299fcc5362d194265e730768e6786561a07ff679bed7e3905/diff:/var/home/core/.local/share/containers/storage/overlay/0144e44842f707bc8f9a048af0f629892559d9ade7d004307cbf61f27f9a356b/diff:/var/home/core/.local/share/containers/storage/overlay/452e4358dedc1b42acfdeab5bec445bda23f84ef16ba787b9fce560f5d1b0add/diff:/var/home/core/.local/share/containers/storage/overlay/29ef59936f898f42e76a1ebf0d7ef1410bf4b00c335001455f81cf9604fad6cd/diff:/var/home/core/.local/share/containers/storage/overlay/cefbac2f2f53e4007d056103b9639f43a7fc6839bd00d897a351be0a75eebc75/diff:/var/home/core/.local/share/containers/storage/overlay/cbd7b2150e6554ee3a901e64594258fef239d4c2aca7fc5a0a2189c0ec0c4a18/diff:/var/home/core/.local/share/containers/storage/overlay/b94fae10dd82bd67a1ba48ae41b0ca9c6ac78261426e7e716bb99e780888d266/diff:/var/home/core/.local/share/containers/storage/overlay/add3813d5c933943008d03a60d576b1f3ef31a9b6ed48060e87b3f0015b269a7/diff:/var/home/core/.local/share/containers/storage/overlay/4f061339358fa95c0d731be2bd78a107327a52c19bdd8199f8504961d868166c/diff:/var/home/core/.local/share/containers/storage/overlay/48cb7d45684e2930c75768aec4ac9126ba1b4bdfc99d83dd3416dca4f4362fc4/diff:/var/home/core/.local/share/containers/storage/overlay/a015a0a0d796e675053ac5c84ea4c36d9632af531dca8d3f2bc4918ca033f027/diff:/var/home/core/.local/share/containers/storage/overlay/54b8fdd1d3cb0dc2b7fa705d6e0db5abb9449ca159178903c32a9d52d3e830a3/diff:/var/home/core/.local/share/containers/storage/overlay/3dd9e99d8d92b0119386f0d4ca1a600fb5394067d4d91af2113a8bf4d4ed6bb8/diff:/var/home/core/.local/share/containers/storage/overlay/d25b217a6619652baaa10608f95b9f108650d05f34e256287c4fd56c82fecd21/diff:/var/home/core/.local/share/containers/storage/overlay/ac3a166b56986861bc12ed9f8ee859a554c0c3d4f9fc17f69e081c5ae8c218d7/diff:/var/home/core/.local/share/containers/storage/overlay/3e98d4eb129335929a35faebe410f7a76ab67f343d555819f2a7d29055fc111c/diff:/var/home/core/.local/share/containers/storage/overlay/6315da9c27ccb9bdbce85426f7458f0d75ca85e7f90facc97a710247bf36cf37/diff:/var/home/core/.local/share/containers/storage/overlay/24313ca28a08d6629e3da89aad806af541b3c9ba78c679cffb382e8977a0cf42/diff:/var/home/core/.local/share/containers/storage/overlay/4890fe800951b90390a3039e3a85222e57ec60203a2aaff7df4a75ee47812eed/diff:/var/home/core/.local/share/containers/storage/overlay/79d399db1bf3d95fda1474ad8fcb5c7f6bce4ae7a10e599c33889e08b402541d/diff:/var/home/core/.local/share/containers/storage/overlay/1f5a5955b7c438edb1478de0b8608f3e78771c6b98f7c2779de1b75928745571/diff:/var/home/core/.local/share/containers/storage/overlay/939f950dd7c2539f551e65e71da49f97825e6a3bfd08c17062afa1b2d505cbe5/diff:/var/home/core/.local/share/containers/storage/overlay/277707afd8a38244edd41e30e7cd9cc03b50584dba25f256efc137ce3a5628d6/diff:/var/home/core/.local/share/containers/storage/overlay/aa2ad6f36baf3b5e66421c57d00a6f7bf4eae01a083b69137d15b0807e4a2d37/diff:/var/home/core/.local/share/containers/storage/overlay/e0de967896516195aa4b0cae31ccc6228cac3854d5eaa6d3530722d92227b772/diff:/var/home/core/.local/share/containers/storage/overlay/054604538f7439be76f286591be099a25ca27006725d66a3f5b98a63d783faf3/diff:/var/home/core/.local/share/containers/storage/overlay/30fff2cc51bee8008fb4aa180036fdfb770c9823214f8b62bf54e6262e777bff/diff:/var/home/core/.local/share/containers/storage/overlay/289c11f0632879e0947405592ce21d880656ed2a5b9d2a4285df3ccdb5cdd00a/diff:/var/home/core/.local/share/containers/storage/overlay/ff970188222528b6524891c544b28401a4e93c2204b7bbb5d0d3b810a9511acd/diff:/var/home/core/.local/share/containers/storage/overlay/e4585f4fa03802080127ebdd89ea933dedca15cd0261fc08ac7b6c3fac34af5f/diff:/var/home/core/.local/share/containers/storage/overlay/585f117a5de0c3cf226987778dd2acb61620053891697eae2257c1c89755a4be/diff:/var/home/core/.local/share/containers/storage/overlay/42da69791a7215ac2c0e8723e09e2fc2a838c8e3d6ee730a7290f54603d92b98/diff:/var/home/core/.local/share/containers/storage/overlay/4521e6481936ce53f48854b9c87ebeb0018e2f0082977e3d600e77623e8d2969/diff:/var/home/core/.local/share/containers/storage/overlay/d6f7d05716e6c461485a38afce10383b667ccf8cdc861b88f9b019bedd8e8d91/diff:/var/home/core/.local/share/containers/storage/overlay/d99bc515c4192dcf98d3a0bb1ed2c6f96720dd1103b835476dad6c4781d9755a/diff:/var/home/core/.local/share/containers/storage/overlay/10ed59e5acb1a320be18a5c944a40a64a08c47f20cc3ff7936414089efba4a10/diff:/var/home/core/.local/share/containers/storage/overlay/38f26c79dbc71646f57ebfe3339195423ce70c033ffc41b08acf286bc234f414/diff:/var/home/core/.local/share/containers/storage/overlay/c35d1b3ef6b51b2b1cff1f7fdf962a2db9791ce4071f3df4b76e8b0d8acf56c8/diff:/var/home/core/.local/share/containers/storage/overlay/da502ce34d9b226eee8ac98f08fd5defd520c0977160b69e1b122b12fc7e8d9c/diff:/var/home/core/.local/share/containers/storage/overlay/79d59f2c0404bc2aa2a976f5b981b823d548d72bbad5b31dc4f276c98649f24f/diff:/var/home/core/.local/share/containers/storage/overlay/608f3a074261105f129d707e4d9ad3d41b5baa94887f092b7c2857f7274a2fce/diff",
                    "MergedDir": "/var/home/core/.local/share/containers/storage/overlay/b5376b0251b8d53acaecd5fdd58324a23931179b425d2f3dc9fa873dce50e9e7/merged",
                    "UpperDir": "/var/home/core/.local/share/containers/storage/overlay/b5376b0251b8d53acaecd5fdd58324a23931179b425d2f3dc9fa873dce50e9e7/diff",
                    "WorkDir": "/var/home/core/.local/share/containers/storage/overlay/b5376b0251b8d53acaecd5fdd58324a23931179b425d2f3dc9fa873dce50e9e7/work"
               }
          },
          "Mounts": [
               {
                    "Type": "volume",
                    "Name": "code-src",
                    "Source": "/var/home/core/.local/share/containers/storage/volumes/code-src/_data",
                    "Destination": "/workspaces/code",
                    "Driver": "local",
                    "Mode": "",
                    "Options": [
                         "nosuid",
                         "nodev",
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "volume",
                    "Name": "mssql-crt",
                    "Source": "/var/home/core/.local/share/containers/storage/volumes/mssql-crt/_data",
                    "Destination": "/usr/local/share/ca-certificates",
                    "Driver": "local",
                    "Mode": "",
                    "Options": [
                         "nosuid",
                         "nodev",
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "volume",
                    "Name": "vscode",
                    "Source": "/var/home/core/.local/share/containers/storage/volumes/vscode/_data",
                    "Destination": "/vscode",
                    "Driver": "local",
                    "Mode": "",
                    "Options": [
                         "nosuid",
                         "nodev",
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               }
          ],
          "Dependencies": [
               "8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d"
          ],
          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "10.88.0.1",
               "IPAddress": "10.88.0.2",
               "IPPrefixLen": 16,
               "IPv6Gateway": "",
               "GlobalIPv6Address": "",
               "GlobalIPv6PrefixLen": 0,
               "MacAddress": "f6:0e:44:2e:89:27",
               "Bridge": "",
               "SandboxID": "",
               "HairpinMode": false,
               "LinkLocalIPv6Address": "",
               "LinkLocalIPv6PrefixLen": 0,
               "Ports": {
                    "1433/tcp": [
                         {
                              "HostIp": "",
                              "HostPort": "5434"
                         }
                    ]
               },
               "SandboxKey": "/run/user/501/netns/netns-9153807d-7183-224d-49f2-1ad410ddcaa6",
               "Networks": {
                    "podman": {
                         "EndpointID": "",
                         "Gateway": "10.88.0.1",
                         "IPAddress": "10.88.0.2",
                         "IPPrefixLen": 16,
                         "IPv6Gateway": "",
                         "GlobalIPv6Address": "",
                         "GlobalIPv6PrefixLen": 0,
                         "MacAddress": "f6:0e:44:2e:89:27",
                         "NetworkID": "podman",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "8734a8a94c63"
                         ]
                    }
               }
          },
          "Namespace": "",
          "IsInfra": false,
          "Config": {
               "Hostname": "dev-pod",
               "Domainname": "",
               "User": "vscode",
               "AttachStdin": false,
               "AttachStdout": false,
               "AttachStderr": false,
               "Tty": false,
               "OpenStdin": false,
               "StdinOnce": false,
               "Env": [
                    "APACHE_ENVVARS=/etc/apache2/envvars",
                    "PHP_ASC_URL=https://www.php.net/distributions/php-8.1.4.tar.xz.asc",
                    "PHP_INI_DIR=/usr/local/etc/php",
                    "PHPIZE_DEPS=autoconf \t\tdpkg-dev \t\tfile \t\tg++ \t\tgcc \t\tlibc-dev \t\tmake \t\tpkg-config \t\tre2c",
                    "NVM_DIR=/usr/local/share/nvm",
                    "container=podman",
                    "APACHE_CONFDIR=/etc/apache2",
                    "GPG_KEYS=528995BFEDFBA7191D46839EF9BA0ADA31CBD89E 39B641343D8C104B2B146DC3F9C39DC0B9698544 F1F692238FBC1666E5A5CCD4199F9DFEF6FFBAFD",
                    "PHP_CPPFLAGS=-fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64",
                    "PHP_CFLAGS=-fstack-protector-strong -fpic -fpie -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64",
                    "TERM=xterm",
                    "PHP_SHA256=05a8c0ac30008154fb38a305560543fc172ba79fb957084a99b8d3b10d5bdb4b",
                    "PHP_URL=https://www.php.net/distributions/php-8.1.4.tar.xz",
                    "NVM_SYMLINK_CURRENT=true",
                    "PATH=/opt/mssql-tools18/bin:/usr/local/share/nvm/current/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "PHP_VERSION=8.1.4",
                    "PHP_LDFLAGS=-Wl,-O1 -pie",
                    "CONTAINER_NAME=php-8.1.3-src-dev",
                    "HOME=/home/vscode",
                    "HOSTNAME=dev-pod"
               ],
               "Cmd": [
                    "-c",
                    "echo Container started\ntrap \"exit 0\" 15\n\nexec \"$@\"\nwhile sleep 1 & wait $!; do :; done",
                    "-"
               ],
               "Image": "localhost/vsc-php-8.1.3-src-dev-b8388b615861eb5c89013a8fd00b03f5:latest",
               "Volumes": null,
               "WorkingDir": "/var/www/html",
               "Entrypoint": "/bin/sh",
               "OnBuild": null,
               "Labels": {
                    "com.visualstudio.code.devcontainers.id": "php",
                    "com.visualstudio.code.devcontainers.release": "v0.231.6",
                    "com.visualstudio.code.devcontainers.source": "https://github.com/microsoft/vscode-dev-containers/",
                    "com.visualstudio.code.devcontainers.timestamp": "Tue, 12 Apr 2022 23:30:01 GMT",
                    "com.visualstudio.code.devcontainers.variant": "8.1-apache-bullseye",
                    "io.buildah.version": "1.26.1",
                    "version": "0.203.3",
                    "vsch.local.folder": "/Users/myuser/dev-devcontainers/devcontainers/php-8.1.3-src-dev",
                    "vsch.quality": "stable"
               },
               "Annotations": {
                    "io.container.manager": "libpod",
                    "io.kubernetes.cri-o.ContainerType": "container",
                    "io.kubernetes.cri-o.Created": "2022-05-11T11:33:19.364915268-05:00",
                    "io.kubernetes.cri-o.SandboxID": "dev-pod",
                    "io.kubernetes.cri-o.TTY": "false",
                    "io.podman.annotations.autoremove": "FALSE",
                    "io.podman.annotations.init": "FALSE",
                    "io.podman.annotations.privileged": "FALSE",
                    "io.podman.annotations.publish-all": "FALSE",
                    "org.opencontainers.image.base.digest": "sha256:2fd4fb51b340f3dce04e9258fe7b38a84780c0ac1700f9b2bc50b152a716d911",
                    "org.opencontainers.image.base.name": "mcr.microsoft.com/vscode/devcontainers/php:0-8.1",
                    "org.opencontainers.image.stopSignal": "28"
               },
               "StopSignal": 28,
               "CreateCommand": [
                    "podman",
                    "run",
                    "--sig-proxy=false",
                    "-a",
                    "STDOUT",
                    "-a",
                    "STDERR",
                    "--mount",
                    "source=code-src,target=/workspaces/code,type=volume",
                    "--mount",
                    "source=mssql-crt,target=/usr/local/share/ca-certificates,type=volume",
                    "--mount",
                    "type=volume,src=vscode,dst=/vscode",
                    "-l",
                    "vsch.local.folder=/Users/myuser/dev-devcontainers/devcontainers/php-8.1.3-src-dev",
                    "-l",
                    "vsch.quality=stable",
                    "-e",
                    "CONTAINER_NAME=php-8.1.3-src-dev",
                    "-u",
                    "vscode",
                    "--pod=dev-pod",
                    "--cap-add=SYS_PTRACE",
                    "--entrypoint",
                    "/bin/sh",
                    "vsc-php-8.1.3-src-dev-b8388b615861eb5c89013a8fd00b03f5",
                    "-c",
                    "echo Container started\ntrap \"exit 0\" 15\n\nexec \"$@\"\nwhile sleep 1 & wait $!; do :; done",
                    "-"
               ],
               "Umask": "0022",
               "Timeout": 0,
               "StopTimeout": 10,
               "Passwd": true
          },
          "HostConfig": {
               "Binds": [
                    "code-src:/workspaces/dev:rw,rprivate,nosuid,nodev,rbind",
                    "mssql-crt:/usr/local/share/ca-certificates:rw,rprivate,nosuid,nodev,rbind",
                    "vscode:/vscode:rw,rprivate,nosuid,nodev,rbind"
               ],
               "CgroupManager": "systemd",
               "CgroupMode": "private",
               "ContainerIDFile": "",
               "LogConfig": {
                    "Type": "journald",
                    "Config": null,
                    "Path": "",
                    "Tag": "",
                    "Size": "0B"
               },
               "NetworkMode": "container:8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d",
               "PortBindings": {},
               "RestartPolicy": {
                    "Name": "",
                    "MaximumRetryCount": 0
               },
               "AutoRemove": false,
               "VolumeDriver": "",
               "VolumesFrom": null,
               "CapAdd": [
                    "CAP_SYS_PTRACE"
               ],
               "CapDrop": [
                    "CAP_AUDIT_WRITE",
                    "CAP_MKNOD",
                    "CAP_NET_RAW"
               ],
               "Dns": [],
               "DnsOptions": [],
               "DnsSearch": [],
               "ExtraHosts": [],
               "GroupAdd": [],
               "IpcMode": "container:8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d",
               "Cgroup": "",
               "Cgroups": "default",
               "Links": null,
               "OomScoreAdj": 0,
               "PidMode": "private",
               "Privileged": false,
               "PublishAllPorts": false,
               "ReadonlyRootfs": false,
               "SecurityOpt": [],
               "Tmpfs": {},
               "UTSMode": "container:8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d",
               "UsernsMode": "container:8734a8a94c636454914fb6556d87a0aac46680d25e64e08ca18fdbbe3ef7418d",
               "IDMappings": {
                    "UidMap": [
                         "0:1:501",
                         "501:0:1",
                         "502:502:999499"
                    ],
                    "GidMap": [
                         "0:1:1000",
                         "1000:0:1",
                         "1001:1001:999000"
                    ]
               },
               "ShmSize": 65536000,
               "Runtime": "oci",
               "ConsoleSize": [
                    0,
                    0
               ],
               "Isolation": "",
               "CpuShares": 0,
               "Memory": 0,
               "NanoCpus": 0,
               "CgroupParent": "user.slice/user-libpod_pod_759c4d63f3c8d1264f871790287c78f3082a671e7b23e440ad82f17418fc87cb.slice",
               "BlkioWeight": 0,
               "BlkioWeightDevice": null,
               "BlkioDeviceReadBps": null,
               "BlkioDeviceWriteBps": null,
               "BlkioDeviceReadIOps": null,
               "BlkioDeviceWriteIOps": null,
               "CpuPeriod": 0,
               "CpuQuota": 0,
               "CpuRealtimePeriod": 0,
               "CpuRealtimeRuntime": 0,
               "CpusetCpus": "",
               "CpusetMems": "",
               "Devices": [],
               "DiskQuota": 0,
               "KernelMemory": 0,
               "MemoryReservation": 0,
               "MemorySwap": 0,
               "MemorySwappiness": 0,
               "OomKillDisable": false,
               "PidsLimit": 2048,
               "Ulimits": [],
               "CpuCount": 0,
               "CpuPercent": 0,
               "IOMaximumIOps": 0,
               "IOMaximumBandwidth": 0,
               "CgroupConf": null
          }
     }
]

Output of podman version:

Client:       Podman Engine
Version:      4.0.3
API Version:  4.0.3
Go Version:   go1.18
Built:        Fri Apr  1 10:28:59 2022
OS/Arch:      darwin/amd64

Server:       Podman Engine
Version:      4.1.0
API Version:  4.1.0
Go Version:   go1.16.15
Built:        Fri May  6 11:13:30 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "35"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 5.17.5-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 813867008
  memTotal: 4110102528
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.4-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 124h 15m 7.58s (Approximately 5.17 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 29
  runRoot: /run/user/501/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.1.0
  Built: 1651853610
  BuiltTime: Fri May  6 11:13:30 2022
  GitCommit: ""
  GoVersion: go1.16.15
  OsArch: linux/amd64
  Version: 4.1.0

Package info (e.g. output of rpm -q podman or apt list podman):
I ran brew list podman since the other two are not installed on my Mac.

/usr/local/Cellar/podman/4.0.3/bin/podman
/usr/local/Cellar/podman/4.0.3/bin/podman-mac-helper
/usr/local/Cellar/podman/4.0.3/bin/podman-remote
/usr/local/Cellar/podman/4.0.3/etc/bash_completion.d/podman
/usr/local/Cellar/podman/4.0.3/libexec/gvproxy
/usr/local/Cellar/podman/4.0.3/share/fish/vendor_completions.d/podman.fish
/usr/local/Cellar/podman/4.0.3/share/man/ (161 files)
/usr/local/Cellar/podman/4.0.3/share/zsh/site-functions/_podman

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

I haven't personally tested on the latest Podman version, but someone else checked on the latest Podman version and the issue still occurred.

Additional environment details (AWS, VirtualBox, physical, etc.):

[rhatdan]

I believe his is more todo with the VM running the container, although it could be an issue with rootless podman versus rootful podman, Have you tried this with rootful?
@Luap99 @mheon @baude WDYT?

[Luap99]

Likely a problem with gvproxy (the process which provides network connectivity between the host and VM)
Maybe related: #12495

[JOSEPHGILBY]

I'll take a look at running our setup in rootful mode today and update you if there's a change.

[JOSEPHGILBY]

I ran the webserver in rootful mode and am still getting the same CONTENT_MISMATCH errors.

[JOSEPHGILBY]

% podman info --debug
host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpuUtilization:
    idlePercent: 75.12
    systemPercent: 11.46
    userPercent: 13.42
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "35"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.17.5-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 892473344
  memTotal: 4110102528
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.4-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 2h 13m 0.07s (Approximately 0.08 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 106825756672
  graphRootUsed: 16185917440
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 29
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.1.0
  Built: 1651853610
  BuiltTime: Fri May  6 11:13:30 2022
  GitCommit: ""
  GoVersion: go1.16.15
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.0

[JOSEPHGILBY]

A quick update, but when I run the setup on docker (no pod + using a network), we don't have these issues. In addition to the network mismatch errors is a significant performance issue (10-30s to load a page). On docker, the performance is significantly better (up to a couple of seconds at most to load a page).

Are there any other ideas on what could be causing these mismatch errors / significant performance issues?