How to use netem inside podman container in MacOS Host?

[bc-lee]

Issue Description

How to use netem inside podman container in MacOS Host?

From what I understand,

1. Podman MacOS runs a QEMU VM
2. Podman MacOS runs Fedora CoreOS (As of writing, I am using fedora-coreos-38.20230625.2.0-qemu.aarch64.qcow2)

Also, when I run $ podman machine ssh lsmod | grep sch_netem in host, I got nothing.

So, in my configuration, it seems that sch_netem is not loaded in the VM.

I'd like to use tc (or netlink) to simulate various network conditions inside a container. How can I do that? Do I need to build my own VM image with sch_netem enabled? Or is there any other way to do that?

podman info output

$ podman version
Client:       Podman Engine
Version:      4.5.1
API Version:  4.5.1
Go Version:   go1.20.4
Git Commit:   9eef30051c83f62816a1772a743e5f1271b196d7
Built:        Sat May 27 00:10:12 2023
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.5.1
API Version:  4.5.1
Go Version:   go1.20.4
Built:        Sat May 27 02:58:19 2023
OS/Arch:      linux/arm64

$ podman info
host:
  arch: arm64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 97.48
    systemPercent: 0.9
    userPercent: 1.62
  cpus: 1
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.3.8-200.fc38.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 242446336
  memTotal: 2048552960
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.5-1.fc38.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.aarch64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 1h 17m 18.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 8007327744
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 9
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 1685123899
  BuiltTime: Sat May 27 02:58:19 2023
  GitCommit: ""
  GoVersion: go1.20.4
  Os: linux
  OsArch: linux/arm64
  Version: 4.5.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[Luap99]

Most kernel modules are loaded on demand when they are used the first time. You can load it manually with sudo modprobe sch_netem or configure it in modules-load.d so it is there for each boot. The fedora kernel should have that module enabled.

But is not clear what you are trying to do, is there an actual error you encounter? In most cases loading modules manually should not be necessary.

[bc-lee]

Unfortunately, I can't find the sch_netem kernel module in my machine's podman. When I ssh into the QEMU VM using $ podman machine ssh, I get the following result:

# find /lib/modules -name "sch_netem*"
# cat /etc/*-release
Fedora release 38 (Thirty Eight)
NAME="Fedora Linux"
VERSION="38.20230709.2.0 (CoreOS)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora CoreOS 38.20230709.2.0"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:38"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='38.20230709.2.0'
Fedora release 38 (Thirty Eight)
Fedora release 38 (Thirty Eight)

In contrast, on my other laptop with Fedora 38 (not Core), I found the following:

$ find /lib/modules -name "sch_netem*"
/lib/modules/6.3.11-200.fc38.x86_64/kernel/net/sched/sch_netem.ko.xz
/lib/modules/6.3.12-200.fc38.x86_64/kernel/net/sched/sch_netem.ko.xz
/lib/modules/6.3.8-100.fc37.x86_64/kernel/net/sched/sch_netem.ko.xz

I conclude that the sch_netem module doesn't exist in Fedora Core images, so it doesn't seem to be a podman issue.

Maybe using a different image for podman will work, so I tried to make a new podman machine with the following commands, but it didn't boot ($ podman machine start never finishes):

$ podman machine init --rootful --username user --image-path /path/to/Fedora-Server-KVM-38-1.6.aarch64.qcow2

I'm not sure the above command for using different images is okay and if using different images for QEMU is a supported use-case for podman.

Thanks in advance.

[bc-lee]

After digging some time, I found that I can install kernel-modules-extra package inside podman machine to get the sch_netem module.

# rpm-ostree install --skip-validation kernel-modules-extra
(after reboot)
# lsmod | grep sch_netem
sch_netem              24576  0
# find /lib/modules -name "sch_netem*"
/lib/modules/6.3.11-200.fc38.aarch64/kernel/net/sched/sch_netem.ko.xz

I'm closing this discussion as I think this is not a bug in podman.