Break EarlyNetworkConfig up to support cleaner bootstore type changes (#9944)

This PR is pretty widespread and touches some easily-broken bits, so
I'll go into some background here. The replicated bootstore (soon to be
backed by trust quorum instead, but that's not relevant to the context
here) gives an eventually-consistent copy of data to all sleds. The type
exposed by the bootstore is
https://github.com/oxidecomputer/omicron/blob/7c56d21f5fb8127a619cb8557cbcc286e8ae0af7/bootstore/src/schemes/v0/storage.rs#L119-L126

`generation` ensures any write requests from an out-of-date Nexus can't
overwrite newer changes. Prior to this PR, `blob` contains a
JSON-serialized `EarlyNetworkConfig`, defined as
https://github.com/oxidecomputer/omicron/blob/7c56d21f5fb8127a619cb8557cbcc286e8ae0af7/sled-agent/types/versions/src/bgp_v6/early_networking.rs#L23-L35

`EarlyNetworkConfigBody` contains a variety of networking configuration
information required to bring up connectivity on a rack cold boot,
including BGP details, which switches have transceivers and in which
slots, etc. This type is quite complex, and historically changing it has
been quite painful - addressing that is the point of this PR and #9801
in general. This PR does not change `NetworkConfig` or
`EarlyNetworkConfigBody` - the focus is on `EarlyNetworkConfig`. It has
a couple of problems:

* `schema_version` is supposed to tell us what version of `body` is
present, but it's defined _in line_ with the body. That means any time
we rev a new version of `EarlyNetworkConfigBody`, we also have to rev a
new version of `EarlyNetworkConfig`, and don't have an opportunity to
inspect `schema_version` before already needing to know how to
deserialize `body`.
* `generation` is duplicated with the `generation` in `NetworkConfig`.
(This is not nearly as much of a problem in practice as the previous
point, but is an opportunity for illegal states - what would it mean for
a `NetworkConfig` at generation N to hold a blobified
`EarlyNetworkConfig` with a different generation?)

`EarlyNetworkConfig` is used in three places on `main`:

* Serialized as `blob` in the bootstore as described above
* Serialized as `data` in the `bootstore_config` CRDB table (this has
the same "duplicated generation" problem as `NetworkConfig` - this table
also stores `generation` as a separate column next to `data`)
* As the type for the `write_network_bootstore_config()` sled-agent
OpenAPI

This PR breaks `EarlyNetworkConfig` up into two types to address the
problems above. The serialized form is now `EarlyNetworkConfigEnvelope`:
https://github.com/oxidecomputer/omicron/blob/a8de498858cbeb2a58ebf835ccb11b21536de59b/sled-agent/types/versions/src/bootstore_versioning/early_networking.rs#L45-L55

This has `schema_version` and an _opaque_ `body` (typed as
`serde_json::Value`). This fixes both problems with
`EarlyNetworkConfig`:

* `EarlyNetworkConfigEnvelope` does not have to have a new type revved
any time `EarlyNetworkConfigBody`, because `body` is opaque. We can
deserialize the envelope, then inspect `schema_version` to know which
version of `EarlyNetworkConfigBody` it contains. (My hope is we _never_
have to rev this type.)
* No duplicated `generation`.

This is fully backwards-compatible as far as _deserialization_ is
concerned: any existing `EarlyNetworkConfig` can be safely deserialized
as an `EarlyNetworkConfigEnvelope`:

* `schema_version` is unchanged
* `generation` will be ignored
* `body` will be read as a `serde_json::Value` instead of an
`EarlyNetworkConfigBody`

`EarlyNetworkConfigEnvelope` does not implement `JsonSchema`, because it
should not be used in HTTP / OpenAPI contexts; it's only meant to be a
serialization wrapper. That brings us to the second type added in this
PR, `WriteNetworkConfigRequest`:
https://github.com/oxidecomputer/omicron/blob/a8de498858cbeb2a58ebf835ccb11b21536de59b/sled-agent/types/versions/src/bootstore_versioning/early_networking.rs#L38-L42

This is now the type accepted by sled-agent's
`write_network_bootstore_config()` endpoint. sled-agent will convert
this request into a `bootstore::NetworkConfig` in the straightforward
way:

1. Wrap `body` in an `EarlyNetworkConfigEnvelope`
2. Serialize the now-wrapped `body` as `NetworkConfig::blob`
3. Include `generation` as `NetworkConfig::generation`

I believe this gets us most of the way to #9801. I want to do an actual
rev of `EarlyNetworkConfigBody` to confirm this puts us in a good place
(and also work on the mechanics of ensuring that any revs made are
required to update the relevant bits of implementation that need to be
updated, such as `EarlyNetworkConfigEnvelope` knowing how to deserialize
the new body type) before closing the issue. I'll also do some update
testing on a racklette to confirm I didn't break anything w.r.t.
backwards compatibility, but I believe all of these changes should be
safe.

This PR also fixes #9943: sled-agent should never "upconvert"
`EarlyNetworkConfigBody` requests from Nexus into a newer version; it
needs to replicate the exact version it's given.

Author: jgallagher

Cargo.lock

@@ -29,17 +29,7 @@ progenitor::generate_api!(
     }),
     derives = [schemars::JsonSchema, PartialEq],
     patch = {
-        BfdPeerConfig = { derives = [Eq, Hash] },
-        BgpConfig = { derives = [Eq, Hash] },
-        BgpPeerConfig = { derives = [Eq, Hash] },
-        MaxPathConfig = { derives = [Eq, Hash] },
-        LldpPortConfig = { derives = [Eq, Hash, PartialOrd, Ord] },
-        TxEqConfig = { derives = [Eq, Hash] },
         OmicronPhysicalDiskConfig = { derives = [Eq, Hash, PartialOrd, Ord] },
-        PortConfig = { derives = [Eq, Hash] },
-        RouteConfig = { derives = [Eq, Hash] },
-        RouterLifetimeConfig = { derives = [Eq, Hash] },
-        UplinkAddressConfig = { derives = [Eq, Hash] },
         VirtualNetworkInterfaceHost = { derives = [Eq, Hash] },
     },
     crates = {
@@ -50,6 +40,9 @@ progenitor::generate_api!(
         Attestation = sled_agent_types_versions::latest::rot::Attestation,
         Baseboard = sled_agent_types_versions::latest::inventory::Baseboard,
         BaseboardId = sled_hardware_types::BaseboardId,
+        BfdPeerConfig = sled_agent_types_versions::latest::early_networking::BfdPeerConfig,
+        BgpConfig = sled_agent_types_versions::latest::early_networking::BgpConfig,
+        BgpPeerConfig = sled_agent_types_versions::latest::early_networking::BgpPeerConfig,
         ByteCount = omicron_common::api::external::ByteCount,
         CertificateChain = sled_agent_types_versions::latest::rot::CertificateChain,
         CommitRequest = trust_quorum_types::messages::CommitRequest,
@@ -62,6 +55,7 @@ progenitor::generate_api!(
         DiskManagementStatus = omicron_common::disk::DiskManagementStatus,
         DiskManagementError = omicron_common::disk::DiskManagementError,
         DiskVariant = omicron_common::disk::DiskVariant,
+        EarlyNetworkConfigBody = sled_agent_types_versions::latest::early_networking::EarlyNetworkConfigBody,
         Epoch = trust_quorum_types::types::Epoch,
         ExternalIpGatewayMap = omicron_common::api::internal::shared::ExternalIpGatewayMap,
         ExternalIpConfig = omicron_common::api::internal::shared::ExternalIpConfig,
@@ -73,8 +67,11 @@ progenitor::generate_api!(
         Inventory = sled_agent_types_versions::latest::inventory::Inventory,
         InventoryDisk = sled_agent_types_versions::latest::inventory::InventoryDisk,
         InventoryZpool = sled_agent_types_versions::latest::inventory::InventoryZpool,
+        LldpAdminStatus = sled_agent_types_versions::latest::early_networking::LldpAdminStatus,
+        LldpPortConfig = sled_agent_types_versions::latest::early_networking::LldpPortConfig,
         LrtqUpgradeMsg = trust_quorum_types::messages::LrtqUpgradeMsg,
         MacAddr = omicron_common::api::external::MacAddr,
+        MaxPathConfig = sled_agent_types_versions::latest::early_networking::MaxPathConfig,
         Measurement = sled_agent_types_versions::latest::rot::Measurement,
         MeasurementLog = sled_agent_types_versions::latest::rot::MeasurementLog,
         MupdateOverrideBootInventory = sled_agent_types_versions::latest::inventory::MupdateOverrideBootInventory,
@@ -89,24 +86,31 @@ progenitor::generate_api!(
         OmicronZoneImageSource = sled_agent_types_versions::latest::inventory::OmicronZoneImageSource,
         OmicronZoneType = sled_agent_types_versions::latest::inventory::OmicronZoneType,
         OmicronZonesConfig = sled_agent_types_versions::latest::inventory::OmicronZonesConfig,
+        PortConfig = sled_agent_types_versions::latest::early_networking::PortConfig,
         PortFec = sled_agent_types_versions::latest::early_networking::PortFec,
         PortSpeed = sled_agent_types_versions::latest::early_networking::PortSpeed,
         PrepareAndCommitRequest = trust_quorum_types::messages::PrepareAndCommitRequest,
+        RackNetworkConfig = sled_agent_types_versions::latest::early_networking::RackNetworkConfig,
         ReconfigureMsg = trust_quorum_types::messages::ReconfigureMsg,
         ResolvedVpcFirewallRule = omicron_common::api::internal::shared::ResolvedVpcFirewallRule,
         ResolvedVpcRoute = omicron_common::api::internal::shared::ResolvedVpcRoute,
         ResolvedVpcRouteSet = omicron_common::api::internal::shared::ResolvedVpcRouteSet,
         Rot = sled_agent_types_versions::latest::rot::Rot,
+        RouteConfig = sled_agent_types_versions::latest::early_networking::RouteConfig,
         RouterId = omicron_common::api::internal::shared::RouterId,
+        RouterLifetimeConfig = sled_agent_types_versions::latest::early_networking::RouterLifetimeConfig,
         RouterTarget = omicron_common::api::internal::shared::RouterTarget,
         RouterVersion = omicron_common::api::internal::shared::RouterVersion,
         Sha3_256Digest = sled_agent_types_versions::latest::rot::Sha3_256Digest,
         SledRole = sled_agent_types_versions::latest::inventory::SledRole,
         SourceNatConfigGeneric = omicron_common::api::internal::shared::SourceNatConfigGeneric,
         SwitchLocation = sled_agent_types_versions::latest::early_networking::SwitchLocation,
         Threshold = trust_quorum_types::types::Threshold,
+        TxEqConfig = sled_agent_types_versions::latest::early_networking::TxEqConfig,
+        UplinkAddressConfig = sled_agent_types_versions::latest::early_networking::UplinkAddressConfig,
         Vni = omicron_common::api::external::Vni,
         VpcFirewallIcmpFilter = omicron_common::api::external::VpcFirewallIcmpFilter,
+        WriteNetworkConfigRequest = sled_agent_types_versions::latest::early_networking::WriteNetworkConfigRequest,
         ZpoolKind = omicron_common::zpool_name::ZpoolKind,
         ZpoolName = omicron_common::zpool_name::ZpoolName,
     }

clients/sled-agent-client/src/lib.rs

@@ -50,11 +50,20 @@ impl From<sled_agent_types::early_networking::BfdMode> for BfdMode {
     fn from(value: sled_agent_types::early_networking::BfdMode) -> Self {
         match value {
             sled_agent_types::early_networking::BfdMode::SingleHop => {
-                BfdMode::SingleHop
+                Self::SingleHop
             }
             sled_agent_types::early_networking::BfdMode::MultiHop => {
-                BfdMode::MultiHop
+                Self::MultiHop
             }
         }
     }
 }
+
+impl From<BfdMode> for sled_agent_types::early_networking::BfdMode {
+    fn from(value: BfdMode) -> Self {
+        match value {
+            BfdMode::SingleHop => Self::SingleHop,
+            BfdMode::MultiHop => Self::MultiHop,
+        }
+    }
+}

nexus/db-model/src/bfd.rs

@@ -13,8 +13,14 @@ use nexus_types::identity::Resource;
 use omicron_common::api::external::Error;
 use omicron_common::api::external::IdentityMetadataCreateParams;
 use serde::{Deserialize, Serialize};
+use sled_agent_types::early_networking::BgpPeerConfig;
+use sled_agent_types::early_networking::ImportExportPolicy;
 use sled_agent_types::early_networking::MaxPathConfig;
+use sled_agent_types::early_networking::RouterLifetimeConfig;
+use sled_agent_types::early_networking::RouterLifetimeConfigError;
 use slog_error_chain::InlineErrorChain;
+use std::net::IpAddr;
+use std::net::Ipv6Addr;
 use uuid::Uuid;
 
 #[derive(
@@ -160,3 +166,60 @@ pub struct BgpPeerView {
     pub vlan_id: Option<SqlU16>,
     pub router_lifetime: SqlU16,
 }
+
+#[derive(Debug, thiserror::Error)]
+pub enum BgpPeerConfigDataError {
+    #[error("database contains illegal router lifetime value")]
+    RouterLifetime(#[source] RouterLifetimeConfigError),
+    #[error("database contains illegal min_ttl value: {0}")]
+    MinTtl(u32),
+}
+
+impl TryFrom<BgpPeerView> for BgpPeerConfig {
+    type Error = BgpPeerConfigDataError;
+
+    fn try_from(value: BgpPeerView) -> Result<Self, Self::Error> {
+        // For unnumbered peers (addr is None), use UNSPECIFIED
+        let addr = match value.addr {
+            None => IpAddr::V6(Ipv6Addr::UNSPECIFIED),
+            Some(addr) => addr.ip(),
+        };
+
+        // TODO-correctness We should have db constraints to ensure these can't
+        // fail.
+        let router_lifetime =
+            RouterLifetimeConfig::new(value.router_lifetime.0)
+                .map_err(BgpPeerConfigDataError::RouterLifetime)?;
+        let min_ttl = value
+            .min_ttl
+            .map(|val| {
+                u8::try_from(*val)
+                    .map_err(|_| BgpPeerConfigDataError::MinTtl(*val))
+            })
+            .transpose()?;
+
+        Ok(Self {
+            asn: *value.asn,
+            port: value.port_name,
+            addr,
+            hold_time: Some(value.hold_time.0.into()),
+            idle_hold_time: Some(value.idle_hold_time.0.into()),
+            delay_open: Some(value.delay_open.0.into()),
+            connect_retry: Some(value.connect_retry.0.into()),
+            keepalive: Some(value.keepalive.0.into()),
+            enforce_first_as: value.enforce_first_as,
+            local_pref: value.local_pref.map(|x| x.into()),
+            md5_auth_key: value.md5_auth_key,
+            min_ttl,
+            multi_exit_discriminator: value
+                .multi_exit_discriminator
+                .map(|x| x.into()),
+            remote_asn: value.remote_asn.map(|x| x.into()),
+            communities: Vec::new(),
+            allowed_export: ImportExportPolicy::NoFiltering,
+            allowed_import: ImportExportPolicy::NoFiltering,
+            vlan_id: value.vlan_id.map(|x| x.0),
+            router_lifetime,
+        })
+    }
+}

nexus/db-model/src/bgp.rs

@@ -52,5 +52,6 @@ rand.workspace = true
 repo-depot-api.workspace = true
 sled-agent-api.workspace = true
 sled-agent-types.workspace = true
+sled-agent-types-versions.workspace = true
 sled-diagnostics.workspace = true
 sp-sim.workspace = true

nexus/mgs-updates/Cargo.toml

@@ -177,7 +177,6 @@ impl HostPhase2SledAgentContext {
 struct HostPhase2SledAgentImpl;
 
 mod api_impl {
-
     use super::HostPhase2SledAgentContext;
     use super::HostPhase2SledAgentImpl;
     use camino::Utf8PathBuf;
@@ -224,7 +223,6 @@ mod api_impl {
     use sled_agent_types::diagnostics::SledDiagnosticsLogsDownloadQueryParam;
     use sled_agent_types::disk::DiskEnsureBody;
     use sled_agent_types::disk::DiskPathParam;
-    use sled_agent_types::early_networking::EarlyNetworkConfig;
     use sled_agent_types::firewall_rules::VpcFirewallRulesEnsureBody;
     use sled_agent_types::instance::InstanceEnsureBody;
     use sled_agent_types::instance::InstanceExternalIpBody;
@@ -269,6 +267,9 @@ mod api_impl {
     use sled_agent_types::zone_bundle::ZoneBundleId;
     use sled_agent_types::zone_bundle::ZoneBundleMetadata;
     use sled_agent_types::zone_bundle::ZonePathParam;
+    use sled_agent_types_versions::v1;
+    use sled_agent_types_versions::v20;
+    use sled_agent_types_versions::v25;
     use sled_diagnostics::SledDiagnosticsQueryOutput;
     use std::collections::BTreeMap;
     use std::collections::BTreeSet;
@@ -761,13 +762,30 @@ mod api_impl {
 
         async fn read_network_bootstore_config_cache(
             _rqctx: RequestContext<Self::Context>,
-        ) -> Result<HttpResponseOk<EarlyNetworkConfig>, HttpError> {
+        ) -> Result<
+            HttpResponseOk<v20::early_networking::EarlyNetworkConfig>,
+            HttpError,
+        > {
+            unimplemented!()
+        }
+
+        async fn write_network_bootstore_config_v25(
+            _rqctx: RequestContext<Self::Context>,
+            _body: TypedBody<v25::early_networking::WriteNetworkConfigRequest>,
+        ) -> Result<HttpResponseUpdatedNoContent, HttpError> {
+            unimplemented!()
+        }
+
+        async fn write_network_bootstore_config_v20(
+            _rqctx: RequestContext<Self::Context>,
+            _body: TypedBody<v20::early_networking::EarlyNetworkConfig>,
+        ) -> Result<HttpResponseUpdatedNoContent, HttpError> {
             unimplemented!()
         }
 
-        async fn write_network_bootstore_config(
+        async fn write_network_bootstore_config_v1(
             _rqctx: RequestContext<Self::Context>,
-            _body: TypedBody<EarlyNetworkConfig>,
+            _body: TypedBody<v1::early_networking::EarlyNetworkConfig>,
         ) -> Result<HttpResponseUpdatedNoContent, HttpError> {
             unimplemented!()
         }