[1/5] User data export: DB models and queries (#8471)

This set of PRs implements the endpoints and functionality proposed in
RFD 563, and the set starts with adding the database parts: the models,
the schema change, and the query routines.

A "user data export" object records the attachment of a read-only volume
to a particular Pantry for the purpose of exporting data. These objects
will be created and deleted using sagas invoked by a background task,
and not managed by a user or operator. As of right now the only
read-only user resources that can be exported are snapshots and images.

A user data export object will be created for each non-deleted snapshot
and image. A "changeset" representing the work required to manage this
will be queried for by the associated background task. If a snapshot or
image is deleted, the associated user data export object will
automatically be marked for deletion, and cleaned up by a saga.

In order not to create unbounded work, a similar pattern of using a
state plus "locking" with an operating saga id is used. Otherwise the
background task's periodic activation could initiate an unbounded amount
of sagas based on the computed changeset.

During Pantry expungement, that Pantry will no longer be returned by a
DNS lookup for all in-service Pantry addresses. The list of in-service
addresses returned from a DNS lookup functions similarly to a rendezvous
table, and is used to mark user data export objects affected by
expungements for deletion so they can be recreated by the associated
background task.

Even if a Pantry is not expunged, a restart of the service (or a bounce
of the sled) will cause all volume attachments to be lost. The user data
export logic handles this by detecting when this occurs and marking any
affected user data export as deleted so that another attachment can be
created. This will be seen in future commits of this set.

Author: jmpesp

nexus/db-model/src/lib.rs

@@ -118,6 +118,7 @@ mod typed_uuid;
 mod unsigned;
 mod upstairs_repair;
 mod user_builtin;
+mod user_data_export;
 mod utilization;
 mod virtual_provisioning_collection;
 mod virtual_provisioning_resource;
@@ -241,6 +242,7 @@ pub use typed_uuid::DbTypedUuid;
 pub use typed_uuid::to_db_typed_uuid;
 pub use upstairs_repair::*;
 pub use user_builtin::*;
+pub use user_data_export::*;
 pub use utilization::*;
 pub use v2p_mapping::*;
 pub use virtual_provisioning_collection::*;

nexus/db-model/src/schema_versions.rs

@@ -16,7 +16,7 @@ use std::{collections::BTreeMap, sync::LazyLock};
 ///
 /// This must be updated when you change the database schema.  Refer to
 /// schema/crdb/README.adoc in the root of this repository for details.
-pub const SCHEMA_VERSION: Version = Version::new(156, 0, 0);
+pub const SCHEMA_VERSION: Version = Version::new(157, 0, 0);
 
 /// List of all past database schema versions, in *reverse* order
 ///
@@ -28,6 +28,7 @@ static KNOWN_VERSIONS: LazyLock<Vec<KnownVersion>> = LazyLock::new(|| {
         // |  leaving the first copy as an example for the next person.
         // v
         // KnownVersion::new(next_int, "unique-dirname-with-the-sql-files"),
+        KnownVersion::new(157, "user-data-export"),
         KnownVersion::new(156, "boot-partitions-inventory"),
         KnownVersion::new(155, "vpc-firewall-icmp"),
         KnownVersion::new(154, "add-pending-mgs-updates"),

nexus/db-model/src/user_data_export.rs

@@ -0,0 +1,282 @@
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this
+// file, You can obtain one at https://mozilla.org/MPL/2.0/.
+
+use super::impl_enum_type;
+use crate::SqlU16;
+use crate::ipv6;
+use crate::typed_uuid::DbTypedUuid;
+use nexus_db_schema::schema::user_data_export;
+use omicron_uuid_kinds::UserDataExportKind;
+use omicron_uuid_kinds::UserDataExportUuid;
+use omicron_uuid_kinds::VolumeKind;
+use omicron_uuid_kinds::VolumeUuid;
+use serde::Deserialize;
+use serde::Serialize;
+use std::net::SocketAddrV6;
+use uuid::Uuid;
+
+impl_enum_type!(
+    UserDataExportResourceTypeEnum:
+
+    #[derive(Copy, Clone, Debug, AsExpression, FromSqlRow, Serialize, Deserialize, PartialEq)]
+    pub enum UserDataExportResourceType;
+
+    // Enum values
+    Snapshot => b"snapshot"
+    Image => b"image"
+);
+
+// FromStr impl required for use with clap (aka omdb)
+impl std::str::FromStr for UserDataExportResourceType {
+    type Err = String;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        match s {
+            "snapshot" => Ok(UserDataExportResourceType::Snapshot),
+            "image" => Ok(UserDataExportResourceType::Image),
+            _ => Err(format!("unrecognized value {} for enum", s)),
+        }
+    }
+}
+
+impl UserDataExportResourceType {
+    pub fn to_string(&self) -> String {
+        String::from(match self {
+            UserDataExportResourceType::Snapshot => "snapshot",
+            UserDataExportResourceType::Image => "image",
+        })
+    }
+}
+
+impl_enum_type!(
+    UserDataExportStateEnum:
+
+    #[derive(Copy, Clone, Debug, AsExpression, FromSqlRow, Serialize, Deserialize, PartialEq)]
+    pub enum UserDataExportState;
+
+    // Enum values
+    Requested => b"requested"
+    Assigning => b"assigning"
+    Live => b"live"
+    Deleting => b"deleting"
+    Deleted => b"deleted"
+);
+
+/// Instead of working directly with the UserDataExportRecord, callers can use
+/// this enum instead, where the call site only cares of the record is live or
+/// not.
+pub enum UserDataExport {
+    NotLive,
+
+    Live { pantry_address: SocketAddrV6, volume_id: VolumeUuid },
+}
+
+/// A "user data export" object represents an attachment of a read-only volume
+/// to a Pantry for the purpose of exporting data. As of this writing only
+/// snapshots and images are able to be exported this way. Management of these
+/// objects is done automatically by a background task.
+///
+/// Note that read-only volumes should never directly be constructed (read: be
+/// passed to Volume::construct). Copies should be created so that the
+/// appropriate reference counting for the read-only volume targets can be
+/// maintained. The user data export object stores that copied Volume, among
+/// other things.
+///
+/// The record transitions through the following states:
+///
+/// ```text
+///     Requested   <--              ---
+///                   |              |
+///         |         |              |
+///         v         |              |  responsibility of user
+///                   |              |  export create saga
+///     Assigning   --               |
+///                                  |
+///         |                        |
+///         v                        ---
+///                                  ---
+///       Live      <--              |
+///                   |              |
+///         |         |              |
+///         v         |              | responsibility of user
+///                   |              | export delete saga
+///     Deleting    --               |
+///                                  |
+///         |                        |
+///         v                        |
+///                                  ---
+///      Deleted
+/// ```
+///
+/// which are captured in the UserDataExportState enum. Annotated on the right
+/// are which sagas are responsible for which state transitions. The state
+/// transitions themselves are performed by these sagas and all involve a query
+/// that:
+///
+///  - checks that the starting state (and other values as required) make sense
+///  - updates the state while setting a unique operating_saga_id id (and any
+///    other fields as appropriate)
+///
+/// As multiple background tasks will be waking up, checking to see what sagas
+/// need to be triggered, and requesting that these sagas run, this is meant to
+/// block multiple sagas from running at the same time in an effort to cut down
+/// on interference - most will unwind at the first step of performing this
+/// state transition instead of somewhere in the middle. This is not required
+/// for correctness as each saga node can deal with this type of interference.
+#[derive(Queryable, Insertable, Selectable, Clone, Debug)]
+#[diesel(table_name = user_data_export)]
+pub struct UserDataExportRecord {
+    id: DbTypedUuid<UserDataExportKind>,
+
+    state: UserDataExportState,
+    operating_saga_id: Option<Uuid>,
+    generation: i64,
+
+    resource_id: Uuid,
+    resource_type: UserDataExportResourceType,
+    resource_deleted: bool,
+
+    pantry_ip: Option<ipv6::Ipv6Addr>,
+    pantry_port: Option<SqlU16>,
+    volume_id: Option<DbTypedUuid<VolumeKind>>,
+}
+
+impl UserDataExportRecord {
+    pub fn new(
+        id: UserDataExportUuid,
+        resource: UserDataExportResource,
+    ) -> Self {
+        let (resource_type, resource_id) = match resource {
+            UserDataExportResource::Snapshot { id } => {
+                (UserDataExportResourceType::Snapshot, id)
+            }
+
+            UserDataExportResource::Image { id } => {
+                (UserDataExportResourceType::Image, id)
+            }
+        };
+
+        Self {
+            id: id.into(),
+
+            state: UserDataExportState::Requested,
+            operating_saga_id: None,
+            generation: 0,
+
+            resource_type,
+            resource_id,
+            resource_deleted: false,
+
+            pantry_ip: None,
+            pantry_port: None,
+            volume_id: None,
+        }
+    }
+
+    pub fn id(&self) -> UserDataExportUuid {
+        self.id.into()
+    }
+
+    pub fn state(&self) -> UserDataExportState {
+        self.state
+    }
+
+    pub fn operating_saga_id(&self) -> Option<Uuid> {
+        self.operating_saga_id
+    }
+
+    pub fn generation(&self) -> i64 {
+        self.generation
+    }
+
+    pub fn resource(&self) -> UserDataExportResource {
+        match self.resource_type {
+            UserDataExportResourceType::Snapshot => {
+                UserDataExportResource::Snapshot { id: self.resource_id }
+            }
+
+            UserDataExportResourceType::Image => {
+                UserDataExportResource::Image { id: self.resource_id }
+            }
+        }
+    }
+
+    pub fn deleted(&self) -> bool {
+        self.resource_deleted
+    }
+
+    pub fn pantry_address(&self) -> Option<SocketAddrV6> {
+        match (&self.pantry_ip, &self.pantry_port) {
+            (Some(pantry_ip), Some(pantry_port)) => Some(SocketAddrV6::new(
+                (*pantry_ip).into(),
+                (*pantry_port).into(),
+                0,
+                0,
+            )),
+
+            (_, _) => None,
+        }
+    }
+
+    pub fn volume_id(&self) -> Option<VolumeUuid> {
+        self.volume_id.map(|i| i.into())
+    }
+
+    pub fn is_live(&self) -> Result<UserDataExport, &'static str> {
+        match self.state {
+            UserDataExportState::Requested
+            | UserDataExportState::Assigning
+            | UserDataExportState::Deleting
+            | UserDataExportState::Deleted => Ok(UserDataExport::NotLive),
+
+            UserDataExportState::Live => {
+                let Some(pantry_ip) = self.pantry_ip else {
+                    return Err("pantry_ip is None!");
+                };
+
+                let Some(pantry_port) = self.pantry_port else {
+                    return Err("pantry_port is None!");
+                };
+
+                let Some(volume_id) = self.volume_id else {
+                    return Err("volume_id is None!");
+                };
+
+                Ok(UserDataExport::Live {
+                    pantry_address: SocketAddrV6::new(
+                        pantry_ip.into(),
+                        *pantry_port,
+                        0,
+                        0,
+                    ),
+
+                    volume_id: volume_id.into(),
+                })
+            }
+        }
+    }
+}
+
+#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq)]
+pub enum UserDataExportResource {
+    Snapshot { id: Uuid },
+
+    Image { id: Uuid },
+}
+
+impl UserDataExportResource {
+    pub fn type_string(&self) -> String {
+        String::from(match self {
+            UserDataExportResource::Snapshot { .. } => "snapshot",
+            UserDataExportResource::Image { .. } => "image",
+        })
+    }
+
+    pub fn id(&self) -> Uuid {
+        match self {
+            UserDataExportResource::Snapshot { id } => *id,
+            UserDataExportResource::Image { id } => *id,
+        }
+    }
+}

nexus/db-queries/src/db/datastore/mod.rs

@@ -107,6 +107,7 @@ mod target_release;
 #[cfg(test)]
 pub(crate) mod test_utils;
 mod update;
+mod user_data_export;
 mod utilization;
 mod v2p_mapping;
 mod virtual_provisioning_collection;
@@ -137,6 +138,7 @@ pub use sled::SledTransition;
 pub use sled::TransitionError;
 pub use support_bundle::SupportBundleExpungementReport;
 pub use switch_port::SwitchPortSettingsCombinedResult;
+pub use user_data_export::*;
 pub use virtual_provisioning_collection::StorageType;
 pub use vmm::VmmStateUpdateResult;
 pub use volume::*;