[multicast] implicit group lifecycle with IP pool integration (#9450)

This PR also addresses permission models, object deletion, and error handling questions related to reserved addresses presented in @askfongjojo's testing Google Doc (default IP Pools are covered in mainline already).

In thinking through the *Groups* API, permission scopes, and flexibility, @rcgoodfellow mentioned this consideration:

> Do we need an explicit notion of a group object at all? Or can instances simply allocate/deallocate group IPs from pools, and there is no explicit management of group objects.

With Fleet admins having access control to create pools and link silos to a pool, we arrived at the idea of replacing the current explicit multicast group CRUD with an implicit lifecycle, where groups are created upon the first member join and deleted when the last member leaves.

**Auth Model:**
  - Discovery (fleet-scoped):
    - Read/list groups and list members: any authenticated user in the same fleet.
  - Membership (project-scoped):
    - Join/leave requires Instance::Modify on the specific instance.
  - Creation control:
    - Implicit group creation only when the caller's silo is linked to a suitable multicast pool (by name or by explicit IP in that pool).

**Behavior:**
  - Implicit lifecycle:
    - Create on first join (idempotent); delete when last member leaves (atomic mark-for-removal, reconciler schedules cleanup).
  - Addressing and validation:
    - Implicit allocation from the caller's linked multicast pools.
    - SSM/ASM semantics enforced:
      - IPv4 SSM 232/8 and IPv6 ff3x::/32 require ≥1 source IP.
      - ASM groups may optionally specify sources (can be `None`).
    - When joining by explicit IP: resolve the pool containing the IP, verify the silo link before creating.
  - Error handling:
    - Reserved/invalid multicast ranges rejected at pool/range add time.

**API:**
  - Primary flows:
    - Group-centric member management: POST/DELETE /v1/multicast-groups/{group}/members
    - Instance-centric join/leave: PUT/DELETE /v1/instances/{instance}/multicast-groups/{group}
  - Discovery endpoints remain for list/view; there is no explicit group create/update/delete.
  - This is a *breaking* change, but multicast is not yet enabled or available in production.

**Key changes:**
  - Implicit group model; groups exist while they have members.
  - IP pool integration for multicast allocation with silo link gating.
  - Simplified API centered on join/leave flows.
  - Add multicast_ip to the member table for responses.
  - For consistency, move to `Instant` type over `SystemTime` for mcast-related caches.

This also fixes the flaky test issue in #9588.

Author: zeeshanlakhani

Cargo.lock

@@ -629,7 +629,7 @@ oxide-client = { path = "clients/oxide-client" }
 oxide-tokio-rt = "0.1.2"
 oxide-vpc = { git = "https://github.com/oxidecomputer/opte", rev = "795a1e0aeefb7a2c6fe4139779fdf66930d09b80", features = [ "api", "std" ] }
 oxlog = { path = "dev-tools/oxlog" }
-oxnet = "0.1.3"
+oxnet = "0.1.4"
 once_cell = "1.21.3"
 openapi-lint = { git = "https://github.com/oxidecomputer/openapi-lint", branch = "main" }
 openapiv3 = "2.2.0"

Cargo.toml

@@ -25,82 +25,186 @@ pub const SLED_PREFIX: u8 = 64;
 
 // Multicast constants
 
-/// IPv4 Source-Specific Multicast (SSM) subnet as defined in RFC 4607:
-/// <https://tools.ietf.org/html/rfc4607>.
+/// IPv4 Source-Specific Multicast (SSM) subnet.
 ///
-/// RFC 4607 Section 3 allocates 232.0.0.0/8 as the IPv4 SSM address range.
+/// See [RFC 4607 §3] for the IPv4 SSM address range allocation (232.0.0.0/8).
 /// This is a single contiguous block, unlike IPv6 which has per-scope ranges.
-pub const IPV4_SSM_SUBNET: oxnet::Ipv4Net =
-    oxnet::Ipv4Net::new_unchecked(Ipv4Addr::new(232, 0, 0, 0), 8);
+///
+/// [RFC 4607 §3]: https://www.rfc-editor.org/rfc/rfc4607#section-3
+pub const IPV4_SSM_SUBNET: Ipv4Net =
+    Ipv4Net::new_unchecked(Ipv4Addr::new(232, 0, 0, 0), 8);
 
-/// IPv6 Source-Specific Multicast (SSM) subnet as defined in RFC 4607:
-/// <https://tools.ietf.org/html/rfc4607>.
+/// IPv6 Source-Specific Multicast (SSM) subnet.
 ///
-/// RFC 4607 Section 3 specifies "FF3x::/32 for each scope x" - meaning one
-/// /32 block per scope (FF30::/32, FF31::/32, ..., FF3F::/32).
+/// See [RFC 4607 §3] for SSM scope allocation. The RFC specifies "ff3x::/32
+/// for each scope x" - meaning one /32 block per scope (ff30::/32, ff31::/32,
+/// ..., ff3f::/32).
 ///
 /// We use /12 as an implementation convenience to match all these blocks with
 /// a single subnet. This works because all SSM addresses share the same first
 /// 12 bits:
-/// - Bits 0-7:  11111111 (0xFF, multicast prefix)
+/// - Bits 0-7:  11111111 (0xff, multicast prefix)
 /// - Bits 8-11: 0011 (flag field = 3, indicating SSM)
-/// - Bits 12-15: xxxx (scope field, any value 0-F)
+/// - Bits 12-15: xxxx (scope field, any value 0-f)
 ///
-/// Thus FF30::/12 efficiently matches FF30:: through FF3F:FFFF:...:FFFF,
+/// Thus ff30::/12 efficiently matches ff30:: through ff3f:ffff:...:ffff,
 /// covering all SSM scopes.
-pub const IPV6_SSM_SUBNET: oxnet::Ipv6Net = oxnet::Ipv6Net::new_unchecked(
-    Ipv6Addr::new(0xff30, 0, 0, 0, 0, 0, 0, 0),
-    12,
-);
+///
+/// This superset is used only for contains-based classification and validation
+/// (e.g., `contains()` checks). It is not an allocation boundary.
+///
+/// [RFC 4607 §3]: https://www.rfc-editor.org/rfc/rfc4607#section-3
+pub const IPV6_SSM_SUBNET: Ipv6Net =
+    Ipv6Net::new_unchecked(Ipv6Addr::new(0xff30, 0, 0, 0, 0, 0, 0, 0), 12);
+
+/// Maximum source IPs per SSM group member (per [RFC 3376] IGMPv3).
+///
+/// [RFC 3376]: https://www.rfc-editor.org/rfc/rfc3376
+pub const MAX_SSM_SOURCE_IPS: usize = 64;
+
+/// Check if an IP is in the SSM (Source-Specific Multicast) range.
+///
+/// SSM ranges per [RFC 4607 §3]:
+/// - IPv4: 232.0.0.0/8
+/// - IPv6: ff3x::/32 (all SSM scopes)
+///
+/// [RFC 4607 §3]: https://www.rfc-editor.org/rfc/rfc4607#section-3
+pub fn is_ssm_address(ip: std::net::IpAddr) -> bool {
+    match ip {
+        IpAddr::V4(addr) => IPV4_SSM_SUBNET.contains(addr),
+        IpAddr::V6(addr) => IPV6_SSM_SUBNET.contains(addr),
+    }
+}
 
 /// IPv4 multicast address range (224.0.0.0/4).
-/// See RFC 5771 (IPv4 Multicast Address Assignments):
-/// <https://www.rfc-editor.org/rfc/rfc5771>
+///
+/// See [RFC 5771] for IPv4 multicast address assignments.
+///
+/// [RFC 5771]: https://www.rfc-editor.org/rfc/rfc5771
 pub const IPV4_MULTICAST_RANGE: Ipv4Net =
     Ipv4Net::new_unchecked(Ipv4Addr::new(224, 0, 0, 0), 4);
 
 /// IPv4 link-local multicast subnet (224.0.0.0/24).
+///
 /// This range is reserved for local network control protocols and should not
 /// be routed beyond the local link. Includes addresses for protocols like
 /// OSPF (224.0.0.5), RIPv2 (224.0.0.9), and other local routing protocols.
-/// See RFC 5771 Section 4:
-/// <https://www.rfc-editor.org/rfc/rfc5771#section-4>
+///
+/// See [RFC 5771 §4] for link-local multicast address assignments. The IANA
+/// IPv4 Multicast Address Space registry is the canonical source for
+/// assignments.
+///
+/// [RFC 5771 §4]: https://www.rfc-editor.org/rfc/rfc5771#section-4
 pub const IPV4_LINK_LOCAL_MULTICAST_SUBNET: Ipv4Net =
     Ipv4Net::new_unchecked(Ipv4Addr::new(224, 0, 0, 0), 24);
 
 /// IPv6 multicast address range (ff00::/8).
-/// See RFC 4291 (IPv6 Addressing Architecture):
-/// <https://www.rfc-editor.org/rfc/rfc4291>
+///
+/// See [RFC 4291] for IPv6 addressing architecture.
+///
+/// [RFC 4291]: https://www.rfc-editor.org/rfc/rfc4291
 pub const IPV6_MULTICAST_RANGE: Ipv6Net =
     Ipv6Net::new_unchecked(Ipv6Addr::new(0xff00, 0, 0, 0, 0, 0, 0, 0), 8);
 
 /// IPv6 multicast prefix (ff00::/8) mask/value for scope checking.
+///
+/// See [RFC 4291 §2.7] for multicast address format.
+///
+/// [RFC 4291 §2.7]: https://www.rfc-editor.org/rfc/rfc4291#section-2.7
 pub const IPV6_MULTICAST_PREFIX: u16 = 0xff00;
 
-/// Admin-scoped IPv6 multicast prefix (ff04::/16) as u16 for address
+/// Admin-local IPv6 multicast prefix (ff04::/16) as u16 for address
 /// construction and normalization of underlay multicast addresses.
+///
+/// See [RFC 4291 §2.7] and [RFC 7346] for IPv6 multicast address format
+/// and scope definitions.
+///
+/// [RFC 4291 §2.7]: https://www.rfc-editor.org/rfc/rfc4291#section-2.7
+/// [RFC 7346]: https://www.rfc-editor.org/rfc/rfc7346
 pub const IPV6_ADMIN_SCOPED_MULTICAST_PREFIX: u16 = 0xff04;
 
+/// Fixed underlay admin-local IPv6 multicast subnet (ff04::/64) used for
+/// internal multicast group allocation and external→underlay mapping.
+///
+/// Admin-local scope (4) is defined in [RFC 7346] as "the smallest scope that
+/// must be administratively configured."
+///
+/// Static for consistency across racks. The XOR-fold algorithm maps external
+/// multicast IPs into this /64 with an 8-bit salt, guaranteeing 256 unique
+/// addresses per external IP for collision retries. IP pool validation rejects
+/// ranges overlapping this prefix.
+///
+/// External pools may use other admin-local prefixes (e.g., `ff04:0:0:1::/64`)
+/// outside this range or other administratively configured scopes
+/// (e.g., site-local `ff05::/16`).
+///
+/// [RFC 7346]: https://www.rfc-editor.org/rfc/rfc7346
+// TODO: Expose this subnet via rack API (e.g., in `Rack` view or a dedicated
+// networking info endpoint) so operators can see reserved address ranges.
+pub const UNDERLAY_MULTICAST_SUBNET: Ipv6Net = Ipv6Net::new_unchecked(
+    Ipv6Addr::new(IPV6_ADMIN_SCOPED_MULTICAST_PREFIX, 0, 0, 0, 0, 0, 0, 0),
+    64,
+);
+
+/// Last address in the underlay multicast subnet (ff04::ffff:ffff:ffff:ffff).
+pub const UNDERLAY_MULTICAST_SUBNET_LAST: Ipv6Addr = Ipv6Addr::new(
+    IPV6_ADMIN_SCOPED_MULTICAST_PREFIX,
+    0,
+    0,
+    0,
+    0xffff,
+    0xffff,
+    0xffff,
+    0xffff,
+);
+
 /// IPv6 interface-local multicast subnet (ff01::/16).
+///
 /// These addresses are not routable and should not be added to IP pools.
-/// See RFC 4291 Section 2.7 (multicast scope field):
-/// <https://www.rfc-editor.org/rfc/rfc4291#section-2.7>
-pub const IPV6_INTERFACE_LOCAL_MULTICAST_SUBNET: oxnet::Ipv6Net =
-    oxnet::Ipv6Net::new_unchecked(
-        Ipv6Addr::new(0xff01, 0, 0, 0, 0, 0, 0, 0),
-        16,
-    );
+///
+/// See [RFC 4291 §2.7] for multicast scope field definitions.
+///
+/// [RFC 4291 §2.7]: https://www.rfc-editor.org/rfc/rfc4291#section-2.7
+pub const IPV6_INTERFACE_LOCAL_MULTICAST_SUBNET: Ipv6Net =
+    Ipv6Net::new_unchecked(Ipv6Addr::new(0xff01, 0, 0, 0, 0, 0, 0, 0), 16);
+
+/// Last address in the IPv6 interface-local multicast subnet.
+pub const IPV6_INTERFACE_LOCAL_MULTICAST_LAST: Ipv6Addr = Ipv6Addr::new(
+    0xff01, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff,
+);
 
 /// IPv6 link-local multicast subnet (ff02::/16).
+///
 /// These addresses are not routable beyond the local link and should not be
 /// added to IP pools.
-/// See RFC 4291 Section 2.7 (multicast scope field):
-/// <https://www.rfc-editor.org/rfc/rfc4291#section-2.7>
-pub const IPV6_LINK_LOCAL_MULTICAST_SUBNET: oxnet::Ipv6Net =
-    oxnet::Ipv6Net::new_unchecked(
-        Ipv6Addr::new(0xff02, 0, 0, 0, 0, 0, 0, 0),
-        16,
-    );
+///
+/// See [RFC 4291 §2.7] for multicast scope field definitions.
+///
+/// [RFC 4291 §2.7]: https://www.rfc-editor.org/rfc/rfc4291#section-2.7
+pub const IPV6_LINK_LOCAL_MULTICAST_SUBNET: Ipv6Net =
+    Ipv6Net::new_unchecked(Ipv6Addr::new(0xff02, 0, 0, 0, 0, 0, 0, 0), 16);
+
+/// Last address in the IPv6 link-local multicast subnet.
+pub const IPV6_LINK_LOCAL_MULTICAST_LAST: Ipv6Addr = Ipv6Addr::new(
+    0xff02, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff,
+);
+
+/// IPv6 reserved-scope multicast subnet (ff00::/16).
+///
+/// Scope 0 is reserved - packets with this scope must not be originated and
+/// must be silently dropped if received. These addresses should not be added
+/// to IP pools.
+///
+/// See [RFC 4291 §2.7] for multicast scope field definitions.
+///
+/// [RFC 4291 §2.7]: https://www.rfc-editor.org/rfc/rfc4291#section-2.7
+pub const IPV6_RESERVED_SCOPE_MULTICAST_SUBNET: Ipv6Net =
+    Ipv6Net::new_unchecked(Ipv6Addr::new(0xff00, 0, 0, 0, 0, 0, 0, 0), 16);
+
+/// Last address in the IPv6 reserved-scope multicast subnet.
+pub const IPV6_RESERVED_SCOPE_MULTICAST_LAST: Ipv6Addr = Ipv6Addr::new(
+    0xff00, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff, 0xffff,
+);
 
 /// maximum possible value for a tcp or udp port
 pub const MAX_PORT: u16 = u16::MAX;
@@ -254,8 +358,9 @@ pub static NTP_OPTE_IPV6_SUBNET: LazyLock<Ipv6Net> = LazyLock::new(|| {
 // Anycast is a mechanism in which a single IP address is shared by multiple
 // devices, and the destination is located based on routing distance.
 //
-// This is covered by RFC 4291 in much more detail:
-// <https://datatracker.ietf.org/doc/html/rfc4291#section-2.6>
+// See [RFC 4291 §2.6] for anycast address allocation.
+//
+// [RFC 4291 §2.6]: https://www.rfc-editor.org/rfc/rfc4291#section-2.6
 //
 // Anycast addresses are always the "zeroeth" address within a subnet.  We
 // always explicitly skip these addresses within our network.

common/src/address.rs

@@ -2558,6 +2558,11 @@ impl Vni {
     pub fn random_system() -> Self {
         Self(rand::rng().random_range(0..Self::MIN_GUEST_VNI))
     }
+
+    /// Returns the VNI as a raw u32.
+    pub const fn as_u32(&self) -> u32 {
+        self.0
+    }
 }
 
 impl From<Vni> for u32 {

common/src/api/external/mod.rs

@@ -713,7 +713,7 @@ task: "multicast_reconciler"
   configured period: every <REDACTED_DURATION>m
   last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
     started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
-warning: unknown background task: "multicast_reconciler" (don't know how to interpret details: Object {"disabled": Bool(false), "errors": Array [], "groups_created": Number(0), "groups_deleted": Number(0), "groups_verified": Number(0), "members_deleted": Number(0), "members_processed": Number(0)})
+warning: unknown background task: "multicast_reconciler" (don't know how to interpret details: Object {"disabled": Bool(false), "empty_groups_marked": Number(0), "errors": Array [], "groups_created": Number(0), "groups_deleted": Number(0), "groups_verified": Number(0), "members_deleted": Number(0), "members_processed": Number(0)})
 
 task: "phantom_disks"
   configured period: every <REDACTED_DURATION>s
@@ -1281,7 +1281,7 @@ task: "multicast_reconciler"
   configured period: every <REDACTED_DURATION>m
   last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
     started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
-warning: unknown background task: "multicast_reconciler" (don't know how to interpret details: Object {"disabled": Bool(false), "errors": Array [], "groups_created": Number(0), "groups_deleted": Number(0), "groups_verified": Number(0), "members_deleted": Number(0), "members_processed": Number(0)})
+warning: unknown background task: "multicast_reconciler" (don't know how to interpret details: Object {"disabled": Bool(false), "empty_groups_marked": Number(0), "errors": Array [], "groups_created": Number(0), "groups_deleted": Number(0), "groups_verified": Number(0), "members_deleted": Number(0), "members_processed": Number(0)})
 
 task: "phantom_disks"
   configured period: every <REDACTED_DURATION>s

dev-tools/omdb/tests/successes.out

@@ -91,7 +91,8 @@ struct RouteSet {
 pub struct MulticastGroupCfg {
     /// The multicast group IP address (IPv4 or IPv6).
     pub group_ip: IpAddr,
-    /// For Source-Specific Multicast (SSM), list of source addresses.
+    /// Source addresses for source-filtered multicast (optional for ASM,
+    /// required for SSM).
     pub sources: Vec<IpAddr>,
 }
 
@@ -752,7 +753,7 @@ impl PortManager {
     ///
     /// TODO: Once OPTE kernel module supports multicast group APIs, this
     /// method should be updated to configure OPTE port-level multicast
-    /// group membership. Note: multicast groups are fleet-wide and can span
+    /// group membership. Note: multicast groups are fleet-scoped and can span
     /// across VPCs.
     pub fn multicast_groups_ensure(
         &self,

illumos-utils/src/opte/port_manager.rs

@@ -10,9 +10,7 @@ use anyhow::anyhow;
 use camino::{Utf8Path, Utf8PathBuf};
 use dropshot::ConfigDropshot;
 use dropshot::ConfigLogging;
-use ipnet::Ipv6Net;
 use nexus_types::deployment::ReconfiguratorConfig;
-use omicron_common::address::IPV6_ADMIN_SCOPED_MULTICAST_PREFIX;
 use omicron_common::address::Ipv6Subnet;
 pub use omicron_common::address::MAX_VPC_IPV4_SUBNET_PREFIX;
 pub use omicron_common::address::MIN_VPC_IPV4_SUBNET_PREFIX;
@@ -31,7 +29,6 @@ use serde_with::serde_as;
 use std::collections::HashMap;
 use std::fmt;
 use std::net::IpAddr;
-use std::net::Ipv6Addr;
 use std::net::SocketAddr;
 use std::time::Duration;
 use uuid::Uuid;
@@ -944,15 +941,6 @@ impl Default for FmTasksConfig {
     }
 }
 
-/// Fixed underlay admin-scoped IPv6 multicast network (ff04::/64) used for
-/// internal multicast group allocation and external→underlay mapping.
-/// This /64 subnet within the admin-scoped space provides 2^64 host addresses
-/// (ample for collision resistance) and is not configurable.
-pub const DEFAULT_UNDERLAY_MULTICAST_NET: Ipv6Net = Ipv6Net::new_assert(
-    Ipv6Addr::new(IPV6_ADMIN_SCOPED_MULTICAST_PREFIX, 0, 0, 0, 0, 0, 0, 0),
-    64,
-);
-
 /// Configuration for multicast options.
 #[derive(Clone, Debug, Default, Deserialize, Eq, PartialEq, Serialize)]
 pub struct MulticastConfig {