IPv6 MLD listener report going to wrong interface

[martijncoenen]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

I'm running opnsense virtualized inside proxmox; LAN/WAN are both virtio interfaces, which are each mapped to a bridge in proxmox, and the bridges map to two physical interfaces. My network has several unifi switches, and I have IGMP snooping configured on several VLANs on the internal LAN. This results in one of the switches to send out MLD queries for IPv6, to determine what multicast addresses a host/port is interested in. I confirmed that these multicast queries arrive correctly at the bridge in proxmox, and in turn in the virtio interface in opnsense. This is captured on the vtnet1_vlan100 interface, which corresponds to vlan100 on the LAN:

9:47:15.748139 IP6 fe80::f692:bfff:fe81:337c > ff02::1: HBH ICMP6, multicast listener querymax resp delay: 10000 addr: ::, length 24

this interface has IPv6 configured; my ISP gives out a prefix over pppoe, and I use "track interface" with WAN as upstream.

Interestingly, when pppoe is disconnected, everything looks fine, and the mld listener reports are sent back (:9380 is the vtnet1_vlan100 interface) correctly:

19:47:18.462474 IP6 fe80::9ca3:3dff:fea4:9380 > ff05::1:3: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff05::1:3, length 24
19:47:19.071261 IP6 fe80::9ca3:3dff:fea4:9380 > ff02::2:ff49:d8be: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2:ff49:d8be, length 24
19:47:19.271584 IP6 fe80::9ca3:3dff:fea4:9380 > ff02::1:ffa4:9380: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ffa4:9380, length 24
19:47:19.683804 IP6 fe80::9ca3:3dff:fea4:9380 > ff02::2:49d8:bec6: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2:49d8:bec6, length 24
19:47:22.090143 IP6 fe80::9ca3:3dff:fea4:9380 > ff02::1:2: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:2, length 24

but when I setup a connection to my ISP over pppoe, the queries still arrive on vtnet1_vlan100, but suddenly, the reports go out over the pppoe0 interface, instead of the vtnet1_vlan100 interface.

# tcpdump -i pppoe0 | grep -i multi
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes
19:54:20.570018 IP6 fe80::9ca3:3dff:fea4:9380 > ff02::2:49d8:bec6: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2:49d8:bec6, length 24
19:54:21.370026 IP6 fe80::9ca3:3dff:fea4:9380 > ff05::1:3: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff05::1:3, length 24
19:54:21.576229 IP6 fe80::9ca3:3dff:fea4:9380 > ff02::1:ffa4:9380: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ffa4:9380, length 24
...

I think this probably happens because, as soon as PPPoE is connected, a default route for the pppoe0 interface is added to the routing table, and the MLD report goes over the default route. That seems wrong though - I think the report should at least go back over the interface where it came from (eg, vtnet1_vlan100) in this case.

Anyway, the result of this is that the linux bridge doesn't know that opnsense is interested in the "all routers" multicast address at ff02::2, and so when a client on the LAN sends an IPv6 router solicitation, that doesn't end up at my opnsense box, and things break.

I've only just set up IPv6 locally, so I'm not sure if this issue is specific to this version or not.

To Reproduce

I described the environment in the description above. Not sure how much of that is relevant to reproduce; I suspect at least some of it is, because otherwise I would expect many more users to hit this issue.

Expected behavior

Multicast MLD reports should be sent back out over the interface from which the query was received.

Describe alternatives you considered

I can workaround this problem by disabling IGMP snooping internally, or by configuring the proxmox bridge to just forward all multicast packets to all ports. Neither is optimal.

Screenshots

N/A

Relevant log files

N/A

Additional context

N/A

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 22.7.10_2 (amd64, OpenSSL), virtualized in Proxmox
Ryzen 3700X
virtio network adapters (backed by proxmox bridge)

[OPNsense-bot]

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository,
please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue,
just let us know, so we can reopen the issue and assign an owner to it.

[T-X]

Hi, I stumbled over this exact same issue, also with an OPNsense 25.1.5_5 (amd64) as a firewall/router VM inside a Proxmox 8.4.1 server. The MLD queries are incoming on vtnet0 and OPNsense then sends the MLD reports out on the wrong interface, on pppoe0. This breaks IPv6 in our setup, leading to IPv6 packet loss.

Any chance to increase the priority of this? Proper MLD handling is mandatory by the IPv6 standard.

[martijncoenen]

Interesting that you ran into the same issue. I forever wanted to try and debug this, but also thought it must have been something on my side. Now that you ran into the same thing, I might try building a debug kernel to figure it out. That being said, for some reason I can't get my unifi gear to start sending MLD queries again, despite IGMP snooping being enabled. Are you also using Unifi gear by any chance?

[T-X]

Interesting that you ran into the same issue. I forever wanted to try and debug this, but also thought it must have been something on my side. Now that you ran into the same thing, I might try building a debug kernel to figure it out. That being said, for some reason I can't get my unifi gear to start sending MLD queries again, despite IGMP snooping being enabled. Are you also using Unifi gear by any chance?

We have two Unifi 6 lite WiFi APs with OpenWrt, but they are not sending MLD queries: https://wiki.chaotikum.org/hackspace:netz:hosts#wifi.

Originally it was our modem sending MLD queries (DrayTek Vigor 167). But since I've updated its firmware today, it's finally not sending them anymore and behaving more like a modem. Instead MLD queries are now send from an OpenWrt powered switch, a ZyXEL GS1900-24HP v1, which in turn thanks to DSA uses the Linux kernel bridge's MLD snooping+querier implementation.

I tried to reproduce this in a vanilla FreeBSD 14.2 VM but wasn't successful so far.

I now also copied the running OPNsense VM from our Proxmox, booted that with KVM on Linux with basically nearly the same qemu command on my laptop as Proxmox does. But somehow can't reproduce it with that either... One of the few differences that come to my mind that I don't have a PPPoE server running here on my laptop. Also noticed that pppoe0 in my copy had a different IPv6 link-local address than the original VM. The original VM has the same link-local IPv6 address on both vtnet0/LAN and pppoe0. Tried adjusting that in my OPNsense copy with "ifconfig" manually but still can't reproduce the issue in my VM, MLD queries + MLD reports both are on vtnet0 and not pppoe0 there.

Also using MLDv1 vs. MLDv2 for the querier does not seem to make a difference. In either case on our producetion OPNsense instance I see MLDv1 or MLDv2 reports on the wrong interface on pppoe0.

Also started looking into the FreeBSD kernel code, but couldn't immediately spot what could cause the issue in there. @martijncoenen do you happen to know if I can somehow enable and print the "CTR3(KTR_MLD, ...)" log messages on FreeBSD, without needing to replace the kernel in OPNsense? Sorry, I'm rather new to FreeBSD.

[T-X]

Regarding workarounds, using something like this on Proxmox helps to avoid IPv6 packetloss, basically installing listeners for and forwarding the IPv6 multicast groups which OPNsense uses:

bridge mdb add dev vmbr0 port tap100i0 grp ff02::1:ffbb:2ed0 permanent
ip -6 addr add ff02::1:ffbb:2ed0 dev vmbr0 autojoin 
bridge mdb add dev vmbr0 port tap100i0 grp ff05::1:3 permanent
ip -6 addr add ff05::1:3 dev vmbr0 autojoin
bridge mdb add dev vmbr0 port tap100i0 grp ff02::2 permanent
ip -6 addr add ff02::2 dev vmbr0 autojoin
bridge mdb add dev vmbr0 port tap100i0 grp ff05::1:3 permanent
ip -6 addr add ff05::1:3 dev vmbr0 autojoin
bridge mdb add dev vmbr0 port tap100i0 grp ff02::1:2 permanent
ip -6 addr add ff02::1:2 dev vmbr0 autojoin
bridge mdb add dev vmbr0 port tap100i0 grp ff02::1:ff00:1 permanent
ip -6 addr add ff02::1:ff00:1 dev vmbr0 autojoin
bridge mdb add dev vmbr0 port tap100i0 grp ff02::2:326f:31ed permanent
ip -6 addr add ff02::2:326f:31ed dev vmbr0 autojoin
bridge mdb add dev vmbr0 port tap100i0 grp ff02::2:ff32:6f31 permanent
ip -6 addr add ff02::2:ff32:6f31 dev vmbr0 autojoin

(Other installations would need to adjust the interface names and IPv6 addresses, check "ifmcstat".)

I was also wondering if as a workaround one could somehow redirect these MLD reports from pppoe0 to vtnet0 with a firewall rule addition. Anyone having an idea of a command I could try on the console (or the OPNsense web-ui) for that?

[T-X]

@martijncoenen next to a Linux bridge as a querier, via "/sys/class/net/br0/bridge/multicast_{snooping,querier}", there is also a userspace implementation for an MLD querier here: fgont/ipv6toolkit#80

[martijncoenen]

@T-X thanks for all the details. I spent last night digging into this, and I think I found the issue; I will do a much longer writeup of this, but TLDR, it is a combination of the pppoe and LAN interfaces having the same link-local IPv6 address (which you also found), and a set of rules that OPNSense writes to the pf filter, to force packets that originate from the pppoe address out on pppoe0. The MLD reports have the same IPv6 link-local address as pppoe, so this rule pushes them out over the pppoe interface instead. Try enabling 'Firewall -> Settings -> Advanced -> Disable automatic rules which force local services to use the assigned interface gateway.`. This setting removes these rules, and fixes the issue for me.

[martijncoenen]

Despite the LLD reports now going to the correct interface, I did notice that IPv6 is broken on my clients with that setting enabled though, so there's more digging to do I guess...
(edit) Nevermind, this turned out to be due to a different configuration change. I think it's all working now. More detailed write-up coming.

[T-X]

Try enabling 'Firewall -> Settings -> Advanced -> Disable automatic rules which force local services to use the assigned interface gateway.`. This setting removes these rules, and fixes the issue for me.

Yaiy, nice catch! Can confirm, toggling this checkbox fixes the issue for me, too!

And yes, IPv6 in general with MLD snooping enabled on our switches seems to be working fine now for me as well.

(Still don't quite know why I can't reproduce it in a VM, but looking forward to the details later.)

[martijncoenen]

Great to hear that also fixed it for you! Will update more details here later.

[martijncoenen]

I crreated https://forum.opnsense.org/index.php?topic=47327.0 for this. Maybe this issue can get reopened. If you're interested in more details, let me know!