Unable to use Avahi in Ubuntu Xenial running in podman

[KEO-RSP]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When running Ubuntu Xenial in podman, announcing a service with Avahi appears to do nothing although a success message appears. The service cannot be found - inside the same container - with avahi-browse.

Note that I am absolutely uncertain whether this is a bug in podman or whether it's simply related to the sheer age of Ubuntu Xenial.

Steps to reproduce the issue:

$ podman run -ti ubuntu:xenial

then in the running container:

root@94d60fa09c1c:/# apt update && apt install avahi-utils -y
[...]
root@94d60fa09c1c:/# service dbus start
* Starting system message bus dbus                         [OK]
root@94d60fa09c1c:/# service avahi-daemon start
* Starting Avahi mDNS/DNS-SD Daemon avahi-daemon           [OK]
root@94d60fa09c1c:/# avahi-publish-service somename _foo._tcp 54321
Established under name 'somename'

then back in the host system, in a separate terminal:

podman exec -ti 94d60fa09c1c /bin/bash
root@94d60fa09c1c:/# avahi-browse -a

There is no output.

Describe the results you received:

avahi-browse does not discover the announced service.

Describe the results you expected:

avahi-browse should discover the announced service.

Additional information you deem important (e.g. issue happens only occasionally):

It works perfectly with Ubuntu Bionic or later.

Output of podman version:

Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.6
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 4
  distribution:
    codename: una
    distribution: linuxmint
    version: "20.3"
  eventLogger: journald
  hostname: rsp-dev
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-110-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 7756664832
  memTotal: 16585023488
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 8588881920
  swapTotal: 8588881920
  uptime: 2h 32m 37.39s (Approximately 0.08 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/rsp/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 1.5
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/rsp/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 11
  runRoot: /run/user/1000/containers
  volumePath: /home/rsp/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/unknown,now 100:3.4.2-1 amd64 [installed]
podman/unknown 100:3.4.2-1 arm64
podman/unknown 100:3.4.2-1 armhf
podman/unknown 100:3.4.2-1 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Tested with the latest version of Podman: No
Checked the troubleshooting guide: Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Podman runs inside a VirtualBox VM on Linux Mint 20.3 Una, which is based on Ubuntu Focal.

[mheon]

Avahi is multicast, if I'm correct? I don't believe slirp4netns handles multicast, so you'll probably need to use --net=host.

[KEO-RSP]

Indeed, we are talking about multicast traffic. However, doing the exact same thing with Bionic or later works as expected. I've prepared two Dockerfiles. One is for Xenial:

FROM ubuntu:xenial

RUN apt update && apt install avahi-utils net-tools -y --no-install-recommends

WORKDIR /root
ADD entrypoint.sh /root

ENTRYPOINT ["bash", "entrypoint.sh"]

The other is for Bionic:

FROM ubuntu:bionic

RUN apt update && apt install avahi-utils net-tools -y --no-install-recommends

WORKDIR /root
ADD entrypoint.sh /root

ENTRYPOINT ["bash", "entrypoint.sh"]

Then there is entrypoint.sh:

#!/bin/bash

set -e

service dbus start
service avahi-daemon start

avahi-publish-service myservice _foo._tcp 54321

Let's say that I build images with tags. The first one gets tagged as avahi_xenial, the second one gets tagged as avahi_bionic. Here are the results of running the images and executing avahi_browse and ifconfig:

Image avahi_xenial

$ podman run --rm localhost/avahi_xenial
 * Starting system message bus dbus
   ...done.
 * Starting Avahi mDNS/DNS-SD Daemon avahi-daemon
   ...done.
Established under name 'myservice'

From another terminal:

$ podman exec -ti ecstatic_clarke /bin/bash
root@41699dac1f15:~# avahi-browse -av
Server version: avahi 0.6.32-rc; Host name: 41699dac1f15.local
E Ifce Prot Name                                          Type                 Domain
: Cache exhausted
: All for now
^CGot SIGINT, quitting.
root@41699dac1f15:~# ifconfig
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

tap0      Link encap:Ethernet  HWaddr 7a:30:54:b2:1f:76  
          inet addr:10.0.2.100  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::7830:54ff:feb2:1f76/64 Scope:Link
          UP BROADCAST RUNNING  MTU:65520  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:796 (796.0 B)

Here's the network settings portion of podman inspect's output:

"NetworkSettings": {
        "EndpointID": "",
        "Gateway": "",
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "MacAddress": "",
        "Bridge": "",
        "SandboxID": "",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {},
        "SandboxKey": "/run/user/1000/netns/cni-a227342b-6b03-4bfb-b30a-62b1a38f42c3"
}

Image avahi_bionic

$ podman run --rm localhost/avahi_bionic
 * Starting system message bus dbus
   ...done.
 * Starting Avahi mDNS/DNS-SD Daemon avahi-daemon
   ...done.
Established under name 'myservice'

From another terminal:

$ podman exec -ti dreamy_colden /bin/bash
root@ed8706070ab2:~# avahi-browse -av
Server version: avahi 0.7; Host name: ed8706070ab2.local
E Ifce Prot Name                                          Type                 Domain
+     lo IPv4 myservice                                     _foo._tcp            local
: Cache exhausted
: All for now
^CGot SIGINT, quitting.
root@ed8706070ab2:~# ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 18  bytes 2787 (2.7 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 18  bytes 2787 (2.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap0: flags=67<UP,BROADCAST,RUNNING>  mtu 65520
        inet 10.0.2.100  netmask 255.255.255.0  broadcast 10.0.2.255
        inet6 fe80::dc58:fbff:fe8d:23f8  prefixlen 64  scopeid 0x20<link>
        ether de:58:fb:8d:23:f8  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9  bytes 726 (726.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Major difference: the service gets discovered on the loopback interface!

Here's the network settings portion of podman inspect's output:

"NetworkSettings": {
        "EndpointID": "",
        "Gateway": "",
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "MacAddress": "",
        "Bridge": "",
        "SandboxID": "",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {},
        "SandboxKey": "/run/user/1000/netns/cni-58fe7484-e42b-810b-6621-90582aa12d28"
}

Differences between Xenial and Bionic

Of course, the two have different versions of about everything. Here are the versions of dbus and Avahi:

Xenial dbus: 1.10.6, Avahi: 0.6.32-rc
Bionic dbus: 1.12.2, Avahi: 0.7

Apart from that, dbus- and Avahi-related config files look the same.

Findings so far

Using dbus-monitor, I was able to identify a difference that occurs when avahi-browse looks for services. Here's the output from within the Xenial container:

method call time=1652856161.319944 sender=:1.3 -> destination=org.freedesktop.DBus serial=1 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=Hello
method return time=1652856161.319965 sender=org.freedesktop.DBus -> destination=:1.3 serial=1 reply_serial=1
   string ":1.3"
signal time=1652856161.319970 sender=org.freedesktop.DBus -> destination=(null destination) serial=8 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameOwnerChanged
   string ":1.3"
   string ""
   string ":1.3"
signal time=1652856161.319987 sender=org.freedesktop.DBus -> destination=:1.3 serial=2 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameAcquired
   string ":1.3"
method call time=1652856161.320232 sender=:1.3 -> destination=org.freedesktop.DBus serial=2 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=AddMatch
   string "type='signal', interface='org.freedesktop.Avahi.Server', sender='org.freedesktop.Avahi', path='/'"
method return time=1652856161.320251 sender=org.freedesktop.DBus -> destination=:1.3 serial=3 reply_serial=2
method call time=1652856161.320423 sender=:1.3 -> destination=org.freedesktop.DBus serial=3 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=AddMatch
   string "type='signal', interface='org.freedesktop.DBus', sender='org.freedesktop.DBus', path='/org/freedesktop/DBus'"
method return time=1652856161.320509 sender=org.freedesktop.DBus -> destination=:1.3 serial=4 reply_serial=3
method call time=1652856161.320722 sender=:1.3 -> destination=org.freedesktop.DBus serial=4 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=AddMatch
   string "type='signal', interface='org.freedesktop.DBus.Local'"
method return time=1652856161.320742 sender=org.freedesktop.DBus -> destination=:1.3 serial=5 reply_serial=4
method call time=1652856161.320904 sender=:1.3 -> destination=org.freedesktop.Avahi serial=5 path=/; interface=org.freedesktop.DBus.Peer; member=Ping
method return time=1652856161.321054 sender=:1.0 -> destination=:1.3 serial=16 reply_serial=5
method call time=1652856161.321236 sender=:1.3 -> destination=org.freedesktop.Avahi serial=6 path=/; interface=org.freedesktop.Avahi.Server; member=GetAPIVersion
method return time=1652856161.321365 sender=:1.0 -> destination=:1.3 serial=17 reply_serial=6
   uint32 515
method call time=1652856161.321551 sender=:1.3 -> destination=org.freedesktop.Avahi serial=7 path=/; interface=org.freedesktop.Avahi.Server; member=GetState
method return time=1652856161.321675 sender=:1.0 -> destination=:1.3 serial=18 reply_serial=7
   int32 2
method call time=1652856161.321866 sender=:1.3 -> destination=org.freedesktop.Avahi serial=8 path=/; interface=org.freedesktop.Avahi.Server; member=GetVersionString
method return time=1652856161.321988 sender=:1.0 -> destination=:1.3 serial=19 reply_serial=8
   string "avahi 0.6.32-rc"
method call time=1652856161.322169 sender=:1.3 -> destination=org.freedesktop.Avahi serial=9 path=/; interface=org.freedesktop.Avahi.Server; member=GetHostNameFqdn
method return time=1652856161.322280 sender=:1.0 -> destination=:1.3 serial=20 reply_serial=9
   string "f40eaa5fa97b.local"
method call time=1652856161.322497 sender=:1.3 -> destination=org.freedesktop.Avahi serial=10 path=/; interface=org.freedesktop.Avahi.Server; member=ServiceTypeBrowserNew
   int32 -1
   int32 -1
   string ""
   uint32 0
method return time=1652856161.322613 sender=:1.0 -> destination=:1.3 serial=21 reply_serial=10
   object path "/Client1/ServiceTypeBrowser1"
signal time=1652856161.322628 sender=:1.0 -> destination=:1.3 serial=22 path=/Client1/ServiceTypeBrowser1; interface=org.freedesktop.Avahi.ServiceTypeBrowser; member=CacheExhausted
signal time=1652856161.322630 sender=:1.0 -> destination=:1.3 serial=23 path=/Client1/ServiceTypeBrowser1; interface=org.freedesktop.Avahi.ServiceTypeBrowser; member=AllForNow

Here's the output from within the Bionic container:

method call time=1652856234.728625 sender=:1.3 -> destination=org.freedesktop.DBus serial=1 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=Hello
method return time=1652856234.728676 sender=org.freedesktop.DBus -> destination=:1.3 serial=1 reply_serial=1
   string ":1.3"
signal time=1652856234.728688 sender=org.freedesktop.DBus -> destination=(null destination) serial=8 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameOwnerChanged
   string ":1.3"
   string ""
   string ":1.3"
signal time=1652856234.728707 sender=org.freedesktop.DBus -> destination=:1.3 serial=2 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameAcquired
   string ":1.3"
method call time=1652856234.729208 sender=:1.3 -> destination=org.freedesktop.DBus serial=2 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=AddMatch
   string "type='signal', interface='org.freedesktop.Avahi.Server', sender='org.freedesktop.Avahi', path='/'"
method return time=1652856234.729258 sender=org.freedesktop.DBus -> destination=:1.3 serial=3 reply_serial=2
method call time=1652856234.730035 sender=:1.3 -> destination=org.freedesktop.DBus serial=3 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=AddMatch
   string "type='signal', interface='org.freedesktop.DBus', sender='org.freedesktop.DBus', path='/org/freedesktop/DBus'"
method return time=1652856234.730052 sender=org.freedesktop.DBus -> destination=:1.3 serial=4 reply_serial=3
method call time=1652856234.730418 sender=:1.3 -> destination=org.freedesktop.DBus serial=4 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=AddMatch
   string "type='signal', interface='org.freedesktop.DBus.Local'"
method return time=1652856234.730435 sender=org.freedesktop.DBus -> destination=:1.3 serial=5 reply_serial=4
method call time=1652856234.730710 sender=:1.3 -> destination=org.freedesktop.Avahi serial=5 path=/; interface=org.freedesktop.DBus.Peer; member=Ping
method return time=1652856234.730999 sender=:1.0 -> destination=:1.3 serial=16 reply_serial=5
method call time=1652856234.731447 sender=:1.3 -> destination=org.freedesktop.Avahi serial=6 path=/; interface=org.freedesktop.Avahi.Server; member=GetAPIVersion
method return time=1652856234.731783 sender=:1.0 -> destination=:1.3 serial=17 reply_serial=6
   uint32 515
method call time=1652856234.732223 sender=:1.3 -> destination=org.freedesktop.Avahi serial=7 path=/; interface=org.freedesktop.Avahi.Server; member=GetState
method return time=1652856234.732258 sender=:1.0 -> destination=:1.3 serial=18 reply_serial=7
   int32 2
method call time=1652856234.732851 sender=:1.3 -> destination=org.freedesktop.Avahi serial=8 path=/; interface=org.freedesktop.Avahi.Server; member=GetVersionString
method return time=1652856234.732865 sender=:1.0 -> destination=:1.3 serial=19 reply_serial=8
   string "avahi 0.7"
method call time=1652856234.733159 sender=:1.3 -> destination=org.freedesktop.Avahi serial=9 path=/; interface=org.freedesktop.Avahi.Server; member=GetHostNameFqdn
method return time=1652856234.733363 sender=:1.0 -> destination=:1.3 serial=20 reply_serial=9
   string "066becb0e363.local"
method call time=1652856234.733798 sender=:1.3 -> destination=org.freedesktop.Avahi serial=10 path=/; interface=org.freedesktop.Avahi.Server; member=ServiceTypeBrowserNew
   int32 -1
   int32 -1
   string ""
   uint32 0
method return time=1652856234.734094 sender=:1.0 -> destination=:1.3 serial=21 reply_serial=10
   object path "/Client1/ServiceTypeBrowser1"
signal time=1652856234.734131 sender=:1.0 -> destination=:1.3 serial=22 path=/Client1/ServiceTypeBrowser1; interface=org.freedesktop.Avahi.ServiceTypeBrowser; member=ItemNew
   int32 1
   int32 0
   string "_foo._tcp"
   string "local"
   uint32 5
signal time=1652856234.734147 sender=:1.0 -> destination=:1.3 serial=23 path=/Client1/ServiceTypeBrowser1; interface=org.freedesktop.Avahi.ServiceTypeBrowser; member=CacheExhausted
method call time=1652856234.734539 sender=:1.3 -> destination=org.freedesktop.Avahi serial=11 path=/; interface=org.freedesktop.Avahi.Server; member=ServiceBrowserNew
   int32 -1
   int32 -1
   string "_foo._tcp"
   string "local"
   uint32 0
method return time=1652856234.734923 sender=:1.0 -> destination=:1.3 serial=24 reply_serial=11
   object path "/Client1/ServiceBrowser2"
signal time=1652856234.734956 sender=:1.0 -> destination=:1.3 serial=25 path=/Client1/ServiceBrowser2; interface=org.freedesktop.Avahi.ServiceBrowser; member=ItemNew
   int32 1
   int32 0
   string "myservice"
   string "_foo._tcp"
   string "local"
   uint32 13
signal time=1652856234.734974 sender=:1.0 -> destination=:1.3 serial=26 path=/Client1/ServiceBrowser2; interface=org.freedesktop.Avahi.ServiceBrowser; member=CacheExhausted
signal time=1652856235.736800 sender=:1.0 -> destination=:1.3 serial=27 path=/Client1/ServiceTypeBrowser1; interface=org.freedesktop.Avahi.ServiceTypeBrowser; member=AllForNow
signal time=1652856235.736824 sender=:1.0 -> destination=:1.3 serial=28 path=/Client1/ServiceBrowser2; interface=org.freedesktop.Avahi.ServiceBrowser; member=AllForNow

Obviously, in the Xenial container, the service type browser "finds nothing". In contrast, in the Bionic container, it gets "ItemNew" signals and in turn goes forward to browse for actual services matching the found service type.

To exclude any kind of weird situation like "Avahi never worked on Xenial", I installed Xenial as a VirtualBox VM and tried Avahi there, which works absolutely fine.

After a lot of debugging, reading Avahi code, building Avahi from sources in the respective container etc., it turns out that Avahi generally ignores loopback interfaces and interfaces that can't do multicast. It does mention this in the documentation somewhere. However, starting with Bionic, it adds lo to the multicast group 224.0.0.251. An Avahi daemon built directly from v0.7 sources taken from Github does not do so. This brought me to the final result of this deep dive expedition: Avahi 0.7 for Ubuntu contains a patch that enables loopback interfaces to announce services. If the Ubuntu team hadn't applied this patch, we would have observed the same behaviour on Xenial and Bionic, which is that both lo and tap0 are ignored by the Avahi daemon. To some extent, the expection that the daemon would behave the same on Xenial and Bionic was wrong.

The good news: there's nothing wrong with Podman. Thanks for thinking about the topic though, I appreciate it 👍