How does the machine based on applehv access the ports of the container?

[BlackHole1]

This question is not about reporting a bug, but arises out of curiosity about how things

I understand that under applehv, podman creates virtual machines based on gvproxy and vfkit.

While developing podman locally, I was surprised to find that the host computer can directly access the ports exposed by the container without any need to do anything. In my memory, it was necessary to request the /services/forwarder/expose interface of gvproxy to do the exposing (which is my current practice).

I am currently implementing a minimal virtual machine (less than 100M) that can run podman. However, I encountered a problem with port exposure. I can’t manage to automatically export ports to the host computer. Here is a simplified command for launching the virtual machine:

./gvproxy -ssh-port 2223 \
    -listen vsock://:1024 \
    -listen unix:///tmp/socket/vfkit.sock \
    -listen unix:///tmp/socket/network.sock

./vfkit \
    --restful-uri unix:///tmp/socket/vfkit-restful.sock \
    --device virtio-vsock,port=1024,socketURL=/tmp/socket/vfkit.sock

In my original implementation, I started vfkit based on --device virtio-vsock, and in the virtual machine, I started a gvforwarder to get the virtual machine online normally. Then I used the /services/forwarder/expose interface to expose the container ports to the host machine.

Not until the day before yesterday did I find out that podman can automate port exporting. So I tried it locally, but I haven’t succeeded until now (this is why I’m asking for help here). I simulated the commands by which podman starts gvproxy and vfkit (based on gvproxy’s listen-vfkit and vfkit’s --device virtio-net), but it had no effect in my rootfs.

Through testing, I found out that the automatic port exporting does not happen when a socket is executed in podman machine to listen to a specific port. It only happens when I start a container and listen to a port.

In my actual investigation, I found that podman now uses netavark instead of cni, but in my rootfs, cni is used. I am uncertain if the automatic port export is due to netavark (I looked through the recent CHANGLOG of netavark but found no clues).

Below is the information in my virtual machine.

# podman inspect podman
[
     {
          "name": "podman",
          "id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
          "driver": "bridge",
          "network_interface": "cni-podman0",
          "created": "2023-11-23T18:06:10+08:00",
          "subnets": [
               {
                    "subnet": "10.88.0.0/16",
                    "gateway": "10.88.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": false,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]


# podman info
host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 99.09
    systemPercent: 0.53
    userPercent: 0.38
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: buildroot
    version: 2023.08.3
  eventLogger: journald
  freeLocks: 2047
  hostname: tiny
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.1.50
  linkmode: dynamic
  logDriver: journald
  memFree: 1555320832
  memTotal: 2044211200
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 1.8.4
      commit: 5a8fa99a5e41facba2eda4af12fa26313918805b
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 0h 41m 4.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: true
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
    PullFromMirror: ""
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 8796093022208
  graphRootUsed: 65294336
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 0
  BuiltTime: Thu Jan  1 08:00:00 1970
  GitCommit: 4.6.2
  GoVersion: go1.20.11
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2


# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 5a:94:ef:e4:0c:ea brd ff:ff:ff:ff:ff:ff
    inet 169.254.107.165/16 brd 169.254.255.255 scope global noprefixroute enp0s1
       valid_lft forever preferred_lft forever
    inet6 fe80::f148:44af:a06b:e071/64 scope link
       valid_lft forever preferred_lft forever
3: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
5: ip6_vti0@NONE: <NOARP> mtu 1332 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd :: permaddr 2a5c:22a3:657c::
6: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
7: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd :: permaddr f216:61f2:4dd6::

PTAL @baude @cfergeau 🙇‍♂️🙇‍♂️

[BlackHole1]

To avoid any confusion in my description, let me clarify what the term 'export port' in the above context means: When running podman run -d -p 8080:80 caddy in a virtual machine, the host machine can access localhost:8080 without manually sending a request to gvproxy to forward the host's port 8080 to port 8080 in the virtual machine.

[Luap99]

podman just talks to the gvproxy API endpoint:
https://github.com/containers/podman/blob/main/libpod/networking_machine.go#L122

[BlackHole1]

Thanks!!!

The reason why it didn’t take effect in my rootfs is because I didn’t create the /etc/containers/podman-machine file. After creating it just now, the automatic port forwarding succeeded!