Attached (External) Subnets #911

FelixMcFelix · 2025-12-31T18:45:21Z

This PR reworks certain aspects of how NAT and gateway layers are constructed to enable attached (external) subnets to function. The primary aim here is that traffic to/from these subnets (owned by the host) do not undergo NAT, and bypass spoof detection as transit IPs would. A few wider changes have been necessary to ensure that these can be attached/detached without breaking any existing transit IPs, and to ensure that traffic originated from an external subnet cannot be directed towards a private VPC recipient.

To be compatible with add/remove of transit IPs, we have to store both transit/attached subnets consistently in VpcCfg rather than working directly on the port's ruleset. This also gives us the opportunity to set these on port create rather than using a followup ioctl.
Reconfiguring the gateway layer requires that we store DhcpCfg in VpcCfg now.
Keeping separate copies of VpcCfg in XdeDev and the port itself seemed... misleading, at best. When XDE needs that, we now reach through port.network().
StatefulActions should never have had mutable access to a packet's ActionMeta (the string-to-string map), because they may never be rerun after an LFT is created on subsequent slowpath walks. It is now up to the generated ActionDescs to update metadata -- this trait's role is similar to StaticAction, which is allowed mutable access to this metadata.
Use of V2B/V2P mappings is now explicitly restricted to sending traffic from external/internal IPs. I put in some rework of ActionMeta ergonomics to make this easier to express and implement.

As far as the control plane is concerned, it should need to only:

Attach external/VPC subnets as they arrive.
Ensure that for attached VPC subnets, the system router is updated s.t. that subnet has an Instance(<name>) target (in turn resolved to an IP target when reifying rules for the port, as happens with instance targets today).

Answers the functional dataplane requirements of #890. Closes #703.

This PR reworks certain aspects of how NAT and gateway layers are constructed to enable attached (external) subnets to function. The primary aim here is that traffic to/from these subnets (owned by the host) do not undergo NAT, and bypass spoof detection as transit IPs would. A few wider changes have been necessary to ensure that these can be attached/detached without breaking any existing transit IPs, and to ensure that traffic originated from an external subnet cannot be directed towards a private VPC recipient.

FelixMcFelix · 2026-01-06T12:44:22Z

lib/opte/src/engine/snat.rs

    use opte_api::Protocol;

-    pub trait Ip: Into<super::IpAddr> {
+    pub trait Ip: Into<super::IpAddr> + Send + Sync {


Followup from #883 -- this moves the Send + Sync bounds up so that we don't need to explicitly specify them in as many spots.

FelixMcFelix · 2026-01-06T13:28:04Z

lib/oxide-vpc/src/engine/nat.rs

+pub fn attach_subnet(
+    port: &Port<VpcNetwork>,
+    inet_gw_map: Option<&InternetGatewayMap>,
+    vpc_mappings: &Arc<VpcMappings>,
+    req: AttachSubnetReq,
+) -> Result<(), OpteError> {


I had a bit of uncertainty as to whether this and detach_subnet should live here, gateway, or some other module since the feature crosses multiple layers.

Living in its own module seems to make the most sense to me. I would not think to come to the nat module to look for attached subnet machinery.

Okay, moved out in f488213.

bnaecker

I've not looked through the entire PR yet, but this seems solid. Nice interface for attaching subnets, and some helpful cleanup along the way. Thanks!

zeeshanlakhani

one Q atm, but this is looking really good. I just want to do some realigning with the RFD before approval.

zeeshanlakhani · 2026-01-13T07:24:22Z

bench/src/packet.rs

                };

-                encap_external(inner_pkt, bsvc_phys, guest_phys)
+                encap_external(inner_pkt, *BSVC_PHYS, guest_phys)


Where is this const being defined now? Or is it generated?

It's defined in opte_test_utils, here:

opte/lib/opte-test-utils/src/lib.rs

Lines 1000 to 1004 in eb15888

pub static BSVC_PHYS: LazyLock<TestIpPhys> = LazyLock::new(|| TestIpPhys {

ip: BS_IP_ADDR,

mac: BS_MAC_ADDR,

vni: Vni::new(BOUNDARY_SERVICES_VNI).unwrap(),

});

zeeshanlakhani

This looks good at the MVP level, until we do further integrated testing.

rcgoodfellow

Thanks Kyle! Comments/questions follow

bin/opteadm/src/bin/opteadm.rs

rcgoodfellow · 2026-01-19T07:12:03Z

lib/oxide-vpc/src/api.rs

+/// be received/sent.
+#[derive(Debug, Clone, Serialize, Deserialize, Default, Eq, PartialEq)]
+pub struct AttachedSubnetConfig {
+    /// Denotes whether this attached subnet is an external IP block,


So if it's not external, then NAT will be applied? What is that being used for?

Again, attached VPC subnets. The instance owns, say, 172.30.0.0/22 and sends trafffic from 172.30.2.128 to 1.1.1.1 -- NAT still needs to be applied, since we're in the VPC's view of that private address space.

The mechanics in OPTE are simple enough for this, since we just don't touch the NAT layer. I know we've punted on the control plane logic for attached VPC subnets to the next release, but I figured it'd be better to avoid more churn.

In that case we'd be implementing an N:1 NAT in that case where N is the size of the subnet.

I think we'll need to make sure this NAT-from-attached-vpc-subnet path is only on VPC subnets attached to the primary interface so if the guest sends non-VPC traffic out a secondary instance for whatever reason, we don't scoop that up into a NAT vortex and replys come back on the primary interface.

That tracks. I know at least today the limit on any kind of normal external IP (SNAT/EIP/FIP) is that those must be bound to the primary interface.

lib/oxide-vpc/src/engine/nat.rs

rcgoodfellow · 2026-01-19T07:44:46Z

lib/oxide-vpc/src/engine/nat.rs

+pub fn attach_subnet(
+    port: &Port<VpcNetwork>,
+    inet_gw_map: Option<&InternetGatewayMap>,
+    vpc_mappings: &Arc<VpcMappings>,
+    req: AttachSubnetReq,
+) -> Result<(), OpteError> {


Living in its own module seems to make the most sense to me. I would not think to come to the nat module to look for attached subnet machinery.

rcgoodfellow · 2026-01-19T07:55:52Z

lib/oxide-vpc/src/engine/overlay.rs

+                // strong isolation which some customers require.
+                //
+                // In future we want this to be a tunable property of the VPC. In this
+                // case we would require an extra table/poptrie per VPC, containing all


This works out nicely for attached subnets. But I think regular floating/ephemeral IPs it will be trickier for large pools since we have no subnet aggregation and the routing table will essentially be a bunch of /32s or /128s for v4/v6 respectively.

I'll put some of this commentary in, you're right that this hypothetical is a bit thornier for individual EIPs. I think it's a little worse in that you can't often aggregate /32s and /128s here -- the value type would be the PhysAddr or (IpAddr, VID) corresponding to the target instance. Although I wouldn't expect the table size to be any larger than the V2P map?

Extended this a little in c647128.

Yeah that's what I was trying to say, is that we cannot really aggregate here. Well ... we could with a clever aggregating routing protocol, but we'd need to disaggregate on migration and would continuously be recomputing aggregations as instances are created/destroyed. It seems like it'd be tricky to get right.

bnaecker mentioned this pull request Dec 31, 2025

Nexus background task for managing external subnets in OPTE oxidecomputer/omicron#9582

Open

FelixMcFelix force-pushed the attached-subnets-mvp branch 7 times, most recently from 374d344 to a832697 Compare January 5, 2026 20:38

FelixMcFelix changed the title ~~Attached External Subnets~~ Attached (External) Subnets Jan 5, 2026

FelixMcFelix force-pushed the attached-subnets-mvp branch from a832697 to 5dc70c8 Compare January 6, 2026 12:15

FelixMcFelix force-pushed the attached-subnets-mvp branch from 5dc70c8 to 2e7cf69 Compare January 6, 2026 12:24

Self-review

eb15888

FelixMcFelix commented Jan 6, 2026

View reviewed changes

FelixMcFelix marked this pull request as ready for review January 6, 2026 13:44

FelixMcFelix requested review from rcgoodfellow and zeeshanlakhani January 8, 2026 13:59

rmustacc requested a review from bnaecker January 8, 2026 17:10

bnaecker reviewed Jan 8, 2026

View reviewed changes

zeeshanlakhani reviewed Jan 15, 2026

View reviewed changes

morlandi7 added this to the 18 milestone Jan 15, 2026

zeeshanlakhani approved these changes Jan 17, 2026

View reviewed changes

rcgoodfellow reviewed Jan 19, 2026

View reviewed changes

FelixMcFelix added 3 commits January 19, 2026 10:35

Merge branch 'master' into attached-subnets-mvp

89f16dc

Review feedback: module for attached subnet ops

f488213

Review feedback: caveat on future EIP hairpin prevention

c647128

rcgoodfellow approved these changes Jan 19, 2026

View reviewed changes

	pub static BSVC_PHYS: LazyLock<TestIpPhys> = LazyLock::new(\|\| TestIpPhys {
	ip: BS_IP_ADDR,
	mac: BS_MAC_ADDR,
	vni: Vni::new(BOUNDARY_SERVICES_VNI).unwrap(),
	});

Attached (External) Subnets #911

Are you sure you want to change the base?

Attached (External) Subnets #911

Uh oh!

Conversation

FelixMcFelix commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bnaecker left a comment

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FelixMcFelix Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani left a comment

Choose a reason for hiding this comment

Uh oh!

rcgoodfellow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

FelixMcFelix commented Dec 31, 2025 •

edited

Loading

FelixMcFelix Jan 15, 2026 •

edited

Loading