TQ: Support upgrade from LRTQ (#9716)

This commit introduces a Nexus lockstep API and omdb support for
triggering upgrade from LRTQ.

In order to do so, we need to enable the `trust_quorum::NodeTask` to
read the lrtq state from disk if exists and there is no existing trust
quorum state on disk.

Notably, LRTQ upgrade did not require modifying the trust quorum
background task as prepare and commit operations are polled identically
between a normal reconfiguration and an LRTQ upgrade.

This was tested on a4x2. However, testing it required a slight deviation
from what would be required on a real upgrade in production. There are
no LRTQ ledgers until RSS runs, and the `trust_quorum::NodeTask` starts
before RSS. Therefore it doesn't see the shares because it only loads
them on startup. On a real system in the field the shares would exist as
soon as the sled-agent was ugpraded and the `NodeTask` would see them.
To rectify this via manual testing, I waited until RSS completed and
then restarted sled-agent on all nodes. I then issued an lrtq-upgrade
via omdb and it worked.

Note that for the above strategy to work, trust quorum RSS must be
disabled so that a real trust quorum configuration is not generated
during RSS. This is done via the following constant, which remains
unset:

```rust
pub const TRUST_QUORUM_INTEGRATION_ENABLED: bool = false;
```

Since that constant doesn't stop trust quorum from running, but just
prohibits RSS setup initializing a trust quorum, we are allowed to
upgrade out of LRTQ. At that point we have a real trust quorum
configuration and can also proceed to add sleds. I did that successfully
as well. What follows are some logged commands to show all this working
in a4x2. It looks like I lost the scrollback for issuing the
lrtq-upgrade, but it was done with the following command in omdb on
`g0`:

```
omdb nexus trust-quorum lrtq-upgrade -w

```

Here is the trust quorum configuration after the upgrade committed.

```
root@oxz_switch:~# omdb nexus trust-quorum get-config 62c2f638-c330-421e-8b4a-7f097a22281e latest
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:17:1:d01::6]:12232
TrustQuorumConfig {
    rack_id: 62c2f638-c330-421e-8b4a-7f097a22281e (rack),
    epoch: Epoch(
        2,
    ),
    last_committed_epoch: None,
    state: Committed,
    threshold: Threshold(
        2,
    ),
    commit_crash_tolerance: 0,
    coordinator: BaseboardId {
        part_number: "913-0000019",
        serial_number: "20000000",
    },
    encrypted_rack_secrets: Some(
        EncryptedRackSecrets {
            salt: Salt(
                [
                    163,
                    19,
                    118,
                    99,
                    229,
                    14,
                    116,
                    81,
                    210,
                    117,
                    180,
                    69,
                    101,
                    181,
                    254,
                    44,
                    38,
                    169,
                    149,
                    63,
                    59,
                    40,
                    63,
                    189,
                    164,
                    106,
                    222,
                    196,
                    112,
                    25,
                    179,
                    107,
                ],
            ),
            data: [
                241,
                144,
                251,
                158,
                0,
                19,
                155,
                183,
                228,
                30,
                218,
                227,
                212,
                100,
                159,
                158,
                160,
                13,
                199,
                185,
                20,
                142,
                61,
                26,
                217,
                92,
                247,
                170,
                110,
                38,
                238,
                91,
                75,
                78,
                71,
                65,
                54,
                93,
                208,
                90,
                44,
                2,
                185,
                10,
                62,
                167,
                222,
                57,
                198,
                217,
                174,
                172,
                70,
                145,
                22,
                206,
            ],
        },
    ),
    members: {
        BaseboardId {
            part_number: "913-0000019",
            serial_number: "20000000",
        }: TrustQuorumMemberData {
            state: Committed,
            share_digest: Some(
                sha3 digest: 2a25477bc8d7623c81ac24e970f381636e3f84c46e41f6f36ca14e7f6011cf1,
            ),
            time_prepared: Some(
                2026-01-23T21:52:19.347850Z,
            ),
            time_committed: Some(
                2026-01-23T21:54:27.774935Z,
            ),
        },
        BaseboardId {
            part_number: "913-0000019",
            serial_number: "20000001",
        }: TrustQuorumMemberData {
            state: Committed,
            share_digest: Some(
                sha3 digest: 5c670dba76c8d5248eac3b1cd6e4bfb88ca12c57bc090ad43ce155b11dcc74,
            ),
            time_prepared: Some(
                2026-01-23T21:52:19.324154Z,
            ),
            time_committed: Some(
                2026-01-23T21:54:27.774935Z,
            ),
        },
        BaseboardId {
            part_number: "913-0000019",
            serial_number: "20000003",
        }: TrustQuorumMemberData {
            state: Committed,
            share_digest: Some(
                sha3 digest: d21a25bbd233b5dab31469659e4efa8a79a3ca7face94719f2df3e7565877c36,
            ),
            time_prepared: Some(
                2026-01-23T21:52:19.337036Z,
            ),
            time_committed: Some(
                2026-01-23T21:55:13.633304Z,
            ),
        },
    },
    time_created: 2026-01-23T21:51:17.122318Z,
    time_committing: Some(
        2026-01-23T21:52:19.360465Z,
    ),
    time_committed: Some(
        2026-01-23T21:55:13.679196Z,
    ),
    time_aborted: None,
    abort_reason: None,
}
```

And here is the oxide CLI command to add a sled:

```
➜  oxide.rs git:(main) ✗ echo '{"sled_ids": [{"part_number": "913-0000019","serial_number": "20000002"}]}' |  target/debug/oxide --profile recovery api /v1/system/hardware/racks/62c2f638-c330-421e-8b4a-7f097a22281e/membership/add --method POST --input -
{
  "members": [
    {
      "part_number": "913-0000019",
      "serial_number": "20000000"
    },
    {
      "part_number": "913-0000019",
      "serial_number": "20000001"
    },
    {
      "part_number": "913-0000019",
      "serial_number": "20000002"
    },
    {
      "part_number": "913-0000019",
      "serial_number": "20000003"
    }
  ],
  "rack_id": "62c2f638-c330-421e-8b4a-7f097a22281e",
  "state": "in_progress",
  "time_aborted": null,
  "time_committed": null,
  "time_created": "2026-01-23T22:15:53.974119Z",
  "unacknowledged_members": [
    {
      "part_number": "913-0000019",
      "serial_number": "20000000"
    },
    {
      "part_number": "913-0000019",
      "serial_number": "20000001"
    },
    {
      "part_number": "913-0000019",
      "serial_number": "20000002"
    },
    {
      "part_number": "913-0000019",
      "serial_number": "20000003"
    }
  ],
  "version": 3
}
```

Polling for a bit, gives the committed status of this add-sled in the
CLI:

```
➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api '/v1/system/hardware/racks/62c2f638-c330-421e-8b4a-7f097a22281e/membership'
{
  "members": [
    {
      "part_number": "913-0000019",
      "serial_number": "20000000"
    },
    {
      "part_number": "913-0000019",
      "serial_number": "20000001"
    },
    {
      "part_number": "913-0000019",
      "serial_number": "20000002"
    },
    {
      "part_number": "913-0000019",
      "serial_number": "20000003"
    }
  ],
  "rack_id": "62c2f638-c330-421e-8b4a-7f097a22281e",
  "state": "committed",
  "time_aborted": null,
  "time_committed": "2026-01-23T22:17:33.936086Z",
  "time_created": "2026-01-23T22:15:53.974119Z",
  "unacknowledged_members": [],
  "version": 3
}
```

And here is the same thing with more detail in omdb:

```
root@oxz_switch:~# omdb nexus trust-quorum get-config 62c2f638-c330-421e-8b4a-7f097a22281e latest
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:17:1:d01::6]:12232
TrustQuorumConfig {
    rack_id: 62c2f638-c330-421e-8b4a-7f097a22281e (rack),
    epoch: Epoch(
        3,
    ),
    last_committed_epoch: Some(
        Epoch(
            2,
        ),
    ),
    state: Committed,
    threshold: Threshold(
        3,
    ),
    commit_crash_tolerance: 1,
    coordinator: BaseboardId {
        part_number: "913-0000019",
        serial_number: "20000001",
    },
    encrypted_rack_secrets: Some(
        EncryptedRackSecrets {
            salt: Salt(
                [
                    68,
                    134,
                    154,
                    136,
                    2,
                    76,
                    247,
                    184,
                    235,
                    215,
                    228,
                    69,
                    93,
                    48,
                    142,
                    161,
                    133,
                    127,
                    137,
                    173,
                    52,
                    16,
                    184,
                    194,
                    114,
                    38,
                    73,
                    215,
                    80,
                    207,
                    255,
                    114,
                ],
            ),
            data: [
                6,
                67,
                96,
                7,
                231,
                106,
                134,
                234,
                229,
                116,
                209,
                76,
                162,
                172,
                175,
                139,
                200,
                74,
                202,
                28,
                127,
                55,
                44,
                61,
                166,
                60,
                135,
                156,
                53,
                42,
                66,
                189,
                92,
                56,
                7,
                93,
                205,
                125,
                98,
                20,
                233,
                99,
                128,
                208,
                223,
                134,
                64,
                32,
                137,
                248,
                119,
                159,
                192,
                57,
                142,
                127,
                109,
                162,
                254,
                177,
                86,
                112,
                21,
                115,
                251,
                94,
                51,
                24,
                135,
                242,
                113,
                127,
                71,
                241,
                50,
                32,
                185,
                218,
                240,
                1,
                178,
                200,
                71,
                173,
                88,
                120,
                254,
                177,
                146,
                205,
                16,
                133,
                246,
                184,
                212,
                118,
            ],
        },
    ),
    members: {
        BaseboardId {
            part_number: "913-0000019",
            serial_number: "20000000",
        }: TrustQuorumMemberData {
            state: Committed,
            share_digest: Some(
                sha3 digest: 829cb1c74cde3390a3d8e0abd81399357fc4fd2d19e7cc4cb6ca582d5c1792d,
            ),
            time_prepared: Some(
                2026-01-23T22:16:29.159832Z,
            ),
            time_committed: Some(
                2026-01-23T22:17:33.885064Z,
            ),
        },
        BaseboardId {
            part_number: "913-0000019",
            serial_number: "20000001",
        }: TrustQuorumMemberData {
            state: Committed,
            share_digest: Some(
                sha3 digest: 97f0b02f4ab2e1fa54d34c939658e5869c97f3ef417e4a7aecf314697cfa,
            ),
            time_prepared: Some(
                2026-01-23T22:16:28.146229Z,
            ),
            time_committed: Some(
                2026-01-23T22:17:32.825838Z,
            ),
        },
        BaseboardId {
            part_number: "913-0000019",
            serial_number: "20000002",
        }: TrustQuorumMemberData {
            state: Committed,
            share_digest: Some(
                sha3 digest: 1c33b428dfb3f345f497868a79a4b73e7252c848ab8ae8d56f2a9137120f8d7,
            ),
            time_prepared: Some(
                2026-01-23T22:16:27.764304Z,
            ),
            time_committed: Some(
                2026-01-23T22:17:32.825838Z,
            ),
        },
        BaseboardId {
            part_number: "913-0000019",
            serial_number: "20000003",
        }: TrustQuorumMemberData {
            state: Committed,
            share_digest: Some(
                sha3 digest: ad6dc4ccc51bf2aeef6fd49bc4c7930d43a946fbc781ecb703428f14f66f11d,
            ),
            time_prepared: Some(
                2026-01-23T22:16:28.837407Z,
            ),
            time_committed: Some(
                2026-01-23T22:17:32.825838Z,
            ),
        },
    },
    time_created: 2026-01-23T22:15:53.974119Z,
    time_committing: Some(
        2026-01-23T22:16:29.183462Z,
    ),
    time_committed: Some(
        2026-01-23T22:17:33.936086Z,
    ),
    time_aborted: None,
    abort_reason: None,
}
```

Author: andrewjstone

Cargo.lock

@@ -11,12 +11,14 @@
 //!    2. A network config blob required for pre-rack-unlock configuration
 //!
 
+use crate::schemes::v0::SharePkgCommon;
+
 use super::{Fsm, FsmConfig, State};
 use camino::Utf8PathBuf;
 use omicron_common::ledger::{Ledger, Ledgerable};
 use serde::{Deserialize, Serialize};
 use sled_hardware_types::Baseboard;
-use slog::{Logger, info};
+use slog::{Logger, info, warn};
 
 /// A persistent version of `Fsm::State`
 #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
@@ -84,6 +86,34 @@ impl PersistentFsmState {
             (Fsm::new_uninitialized(node_id, config), 0)
         }
     }
+
+    /// Load the ledger for use by trust quorum
+    pub async fn load_for_trust_quorum_upgrade(
+        log: &Logger,
+        paths: Vec<Utf8PathBuf>,
+    ) -> Option<SharePkgCommon> {
+        let ledger = Ledger::<PersistentFsmState>::new(&log, paths).await?;
+        let persistent_state = ledger.into_inner();
+        info!(
+            log,
+            "Loaded LRTQ PersistentFsmState from ledger in state {} with generation {}",
+            persistent_state.state.name(),
+            persistent_state.generation
+        );
+
+        match persistent_state.state {
+            State::Uninitialized | State::Learning => {
+                warn!(
+                    log,
+                    "Unexpected LRTQ state: {}. No share available.",
+                    persistent_state.state.name()
+                );
+                None
+            }
+            State::InitialMember { pkg, .. } => Some(pkg.common.clone()),
+            State::Learned { pkg } => Some(pkg.common.clone()),
+        }
+    }
 }
 
 /// Network configuration required to bring up the control plane

bootstore/src/schemes/v0/storage.rs

@@ -67,6 +67,7 @@ progenitor::generate_api!(
         ReconfiguratorConfigView = nexus_types::deployment::ReconfiguratorConfigView,
         RecoverySiloConfig = sled_agent_types_versions::latest::rack_init::RecoverySiloConfig,
         SledAgentUpdateStatus = nexus_types::internal_api::views::SledAgentUpdateStatus,
+        TrustQuorumConfig = nexus_types::trust_quorum::TrustQuorumConfig,
         UpdateStatus = nexus_types::internal_api::views::UpdateStatus,
         ZoneStatus = nexus_types::internal_api::views::ZoneStatus,
         ZpoolName = omicron_common::zpool_name::ZpoolName,

clients/nexus-lockstep-client/src/lib.rs

@@ -69,6 +69,7 @@ progenitor::generate_api!(
         Inventory = sled_agent_types_versions::latest::inventory::Inventory,
         InventoryDisk = sled_agent_types_versions::latest::inventory::InventoryDisk,
         InventoryZpool = sled_agent_types_versions::latest::inventory::InventoryZpool,
+        LrtqUpgradeMsg = trust_quorum_types::messages::LrtqUpgradeMsg,
         MacAddr = omicron_common::api::external::MacAddr,
         MupdateOverrideBootInventory = sled_agent_types_versions::latest::inventory::MupdateOverrideBootInventory,
         Name = omicron_common::api::external::Name,

clients/sled-agent-client/src/lib.rs

@@ -85,6 +85,7 @@ supports-color.workspace = true
 tabled.workspace = true
 textwrap.workspace = true
 tokio = { workspace = true, features = ["full"] }
+trust-quorum-types.workspace = true
 tufaceous-artifact.workspace = true
 unicode-width.workspace = true
 update-engine.workspace = true

dev-tools/omdb/Cargo.toml

@@ -85,6 +85,7 @@ use omicron_uuid_kinds::DemoSagaUuid;
 use omicron_uuid_kinds::GenericUuid;
 use omicron_uuid_kinds::ParseError;
 use omicron_uuid_kinds::PhysicalDiskUuid;
+use omicron_uuid_kinds::RackUuid;
 use omicron_uuid_kinds::SledUuid;
 use omicron_uuid_kinds::SupportBundleUuid;
 use quiesce::QuiesceArgs;
@@ -97,6 +98,7 @@ use slog_error_chain::InlineErrorChain;
 use std::collections::BTreeMap;
 use std::collections::BTreeSet;
 use std::fs::OpenOptions;
+use std::num::ParseIntError;
 use std::os::unix::fs::PermissionsExt;
 use std::str::FromStr;
 use std::sync::Arc;
@@ -108,6 +110,7 @@ use tabled::settings::Padding;
 use tabled::settings::object::Columns;
 use tokio::io::AsyncWriteExt;
 use tokio::sync::OnceCell;
+use trust_quorum_types::types::Epoch;
 use update_engine::EventBuffer;
 use update_engine::ExecutionStatus;
 use update_engine::ExecutionTerminalInfo;
@@ -165,6 +168,8 @@ enum NexusCommands {
     /// interact with support bundles
     #[command(visible_alias = "sb")]
     SupportBundles(SupportBundleArgs),
+    /// interact with the trust quorum
+    TrustQuorum(TrustQuorumArgs),
     /// show running artifact versions
     UpdateStatus(UpdateStatusArgs),
 }
@@ -566,6 +571,43 @@ enum SupportBundleCommands {
     Inspect(SupportBundleInspectArgs),
 }
 
+#[derive(Debug, Args)]
+struct TrustQuorumArgs {
+    #[command(subcommand)]
+    command: TrustQuorumCommands,
+}
+
+#[derive(Debug, Subcommand)]
+enum TrustQuorumCommands {
+    GetConfig(TrustQuorumConfigArgs),
+    LrtqUpgrade,
+}
+
+#[derive(Debug, Clone, Copy, Args)]
+struct TrustQuorumConfigArgs {
+    rack_id: RackUuid,
+    epoch: TrustQuorumEpochOrLatest,
+}
+
+#[derive(Debug, Clone, Copy)]
+pub(crate) enum TrustQuorumEpochOrLatest {
+    Latest,
+    Epoch(Epoch),
+}
+
+impl FromStr for TrustQuorumEpochOrLatest {
+    type Err = ParseIntError;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        if matches!(s, "latest" | "current") {
+            Ok(Self::Latest)
+        } else {
+            let i: u64 = s.parse()?;
+            Ok(Self::Epoch(Epoch(i)))
+        }
+    }
+}
+
 #[derive(Debug, Args)]
 struct SupportBundleDeleteArgs {
     id: SupportBundleUuid,
@@ -860,6 +902,15 @@ impl NexusArgs {
             NexusCommands::SupportBundles(SupportBundleArgs {
                 command: SupportBundleCommands::Inspect(args),
             }) => cmd_nexus_support_bundles_inspect(&client, args).await,
+            NexusCommands::TrustQuorum(TrustQuorumArgs {
+                command: TrustQuorumCommands::GetConfig(args),
+            }) => cmd_nexus_trust_quorum_get_config(&client, args).await,
+            NexusCommands::TrustQuorum(TrustQuorumArgs {
+                command: TrustQuorumCommands::LrtqUpgrade,
+            }) => {
+                let token = omdb.check_allow_destructive()?;
+                cmd_nexus_trust_quorum_lrtq_upgrade(&client, token).await
+            }
             NexusCommands::UpdateStatus(args) => {
                 cmd_nexus_update_status(&client, args).await
             }
@@ -4452,6 +4503,55 @@ async fn cmd_nexus_support_bundles_list(
     Ok(())
 }
 
+async fn cmd_nexus_trust_quorum_get_config(
+    client: &nexus_lockstep_client::Client,
+    args: &TrustQuorumConfigArgs,
+) -> Result<(), anyhow::Error> {
+    let config = match args.epoch {
+        TrustQuorumEpochOrLatest::Latest => client
+            .trust_quorum_get_config(&args.rack_id.as_untyped_uuid(), None)
+            .await
+            .with_context(|| {
+                format!(
+                    "getting latest trust quorum config for rack {}",
+                    args.rack_id
+                )
+            })?,
+        TrustQuorumEpochOrLatest::Epoch(epoch) => client
+            .trust_quorum_get_config(
+                &args.rack_id.as_untyped_uuid(),
+                Some(epoch.0),
+            )
+            .await
+            .with_context(|| {
+                format!(
+                    "getting trust quorum config for rack {}, epoch {}",
+                    args.rack_id, epoch
+                )
+            })?,
+    }
+    .into_inner();
+
+    println!("{config:#?}");
+
+    Ok(())
+}
+
+async fn cmd_nexus_trust_quorum_lrtq_upgrade(
+    client: &nexus_lockstep_client::Client,
+    _destruction_token: DestructiveOperationToken,
+) -> Result<(), anyhow::Error> {
+    let epoch = client
+        .trust_quorum_lrtq_upgrade()
+        .await
+        .context("lrtq upgrade")?
+        .into_inner();
+
+    println!("Started LRTQ upgrade at epoch {epoch}");
+
+    Ok(())
+}
+
 /// Runs `omdb nexus support-bundles create`
 async fn cmd_nexus_support_bundles_create(
     client: &nexus_lockstep_client::Client,

dev-tools/omdb/src/bin/omdb/nexus.rs

@@ -903,6 +903,7 @@ Commands:
   sagas                  view sagas, create and complete demo sagas
   sleds                  interact with sleds
   support-bundles        interact with support bundles [aliases: sb]
+  trust-quorum           interact with the trust quorum
   update-status          show running artifact versions
   help                   Print this message or the help of the given subcommand(s)
 

dev-tools/omdb/tests/usage_errors.out

@@ -21,6 +21,7 @@ use nexus_db_model::DbTypedUuid;
 use nexus_db_model::HwBaseboardId;
 use nexus_db_model::TrustQuorumConfiguration as DbTrustQuorumConfiguration;
 use nexus_db_model::TrustQuorumMember as DbTrustQuorumMember;
+use nexus_types::trust_quorum::IsLrtqUpgrade;
 use nexus_types::trust_quorum::ProposedTrustQuorumConfig;
 use nexus_types::trust_quorum::{
     TrustQuorumConfig, TrustQuorumConfigState, TrustQuorumMemberData,
@@ -435,10 +436,17 @@ impl DataStore {
         )
         .await?;
 
-        // Ensure that epochs are sequential
+        // Ensure that epochs are sequential or this is the inital attempt at an
+        // LRTQ upgrade.
+        //
+        // In the latter case the proposed epoch will be 2, as LRTQ has an epoch
+        // of 1 that is encoded as a ZFS dataset property.
         let latest_epoch = latest_config.as_ref().map(|c| c.epoch);
         bail_unless!(
-            latest_epoch == proposed.epoch.previous(),
+            latest_epoch == proposed.epoch.previous()
+                || (latest_epoch.is_none()
+                    && proposed.is_lrtq_upgrade == IsLrtqUpgrade::Yes
+                    && proposed.epoch == Epoch(2)),
             "Epochs for trust quorum configurations must be sequential. \
             Current epoch = {:?}, Proposed Epoch = {:?}",
             latest_epoch,
@@ -1537,7 +1545,7 @@ mod tests {
         .await
         .unwrap();
 
-        // Last committed epoch is incoreect (should be 1)
+        // Last committed epoch is incorrect (should be 1)
         let bad_config = ProposedTrustQuorumConfig {
             rack_id,
             epoch: Epoch(2),
@@ -1595,6 +1603,51 @@ mod tests {
         logctx.cleanup_successful();
     }
 
+    #[tokio::test]
+    async fn test_tq_insert_initial_lrtq_upgrade() {
+        let logctx = test_setup_log("test_tq_update_prepare_and_commit");
+        let db = TestDatabase::new_with_datastore(&logctx.log).await;
+        let (opctx, datastore) = (db.opctx(), db.datastore());
+
+        let hw_ids = insert_hw_baseboard_ids(&db).await;
+        let rack_id = RackUuid::new_v4();
+        let members: BTreeSet<_> =
+            hw_ids.iter().cloned().map(BaseboardId::from).collect();
+
+        // Propse a an LRTQ upgrade and successfully insert it
+        let config = ProposedTrustQuorumConfig {
+            rack_id,
+            epoch: Epoch(2),
+            is_lrtq_upgrade: IsLrtqUpgrade::Yes,
+            members: members.clone(),
+        };
+
+        // Insert should succeed
+        datastore.tq_insert_latest_config(opctx, config.clone()).await.unwrap();
+
+        // Read the config back and check that it's preparing for LRTQ upgrade
+        // with no acks.
+        let read_config = datastore
+            .tq_get_latest_config(opctx, rack_id)
+            .await
+            .expect("no error")
+            .expect("returned config");
+
+        // The read config should be preparing
+        assert_eq!(read_config.epoch, config.epoch);
+        assert_eq!(
+            read_config.state,
+            TrustQuorumConfigState::PreparingLrtqUpgrade
+        );
+        assert!(read_config.encrypted_rack_secrets.is_none());
+        assert!(read_config.members.iter().all(|(_, info)| {
+            info.state == TrustQuorumMemberState::Unacked
+        }));
+
+        db.terminate().await;
+        logctx.cleanup_successful();
+    }
+
     #[tokio::test]
     async fn test_tq_update_prepare_and_commit() {
         let logctx = test_setup_log("test_tq_update_prepare_and_commit");

nexus/db-queries/src/db/datastore/trust_quorum.rs

@@ -16,4 +16,5 @@ omicron-uuid-kinds.workspace = true
 omicron-workspace-hack.workspace = true
 schemars.workspace = true
 serde.workspace = true
+trust-quorum-types.workspace = true
 uuid.workspace = true

nexus/lockstep-api/Cargo.toml

@@ -52,6 +52,7 @@ use omicron_uuid_kinds::*;
 use schemars::JsonSchema;
 use serde::Deserialize;
 use serde::Serialize;
+use trust_quorum_types::types::Epoch;
 use uuid::Uuid;
 
 const RACK_INITIALIZATION_REQUEST_MAX_BYTES: usize = 10 * 1024 * 1024;
@@ -564,15 +565,29 @@ pub trait NexusLockstepApi {
         rqctx: RequestContext<Self::Context>,
     ) -> Result<HttpResponseOk<QuiesceStatus>, HttpError>;
 
-    /// Retrieve the latest ongoing rack cluster membership change
+    /// Retrieve the trust quorum configuration for the given epoch, or latest
+    // if no epoch is given
     #[endpoint {
         method = GET,
-        path = "/trust-quorum/{rack_id}/config/latest",
+        path = "/trust-quorum/config/{rack_id}",
     }]
-    async fn trust_quorum_get_latest_config(
+    async fn trust_quorum_get_config(
         rqctx: RequestContext<Self::Context>,
-        path_params: Path<RackPathParam>,
-    ) -> Result<HttpResponseOk<Option<TrustQuorumConfig>>, HttpError>;
+        path_params: Path<params::RackMembershipConfigPathParams>,
+        query_params: Query<TrustQuorumEpochQueryParam>,
+    ) -> Result<HttpResponseOk<TrustQuorumConfig>, HttpError>;
+
+    /// Initiate an LRTQ upgrade
+    ///
+    /// Return the epoch of the proposed configuration, so it can be polled
+    /// asynchronously.
+    #[endpoint {
+        method = POST,
+        path = "/trust-quorum/lrtq-upgrade"
+    }]
+    async fn trust_quorum_lrtq_upgrade(
+        rqctx: RequestContext<Self::Context>,
+    ) -> Result<HttpResponseOk<Epoch>, HttpError>;
 }
 
 /// Path parameters for Rack requests.
@@ -621,3 +636,8 @@ pub struct SledId {
 pub struct VersionPathParam {
     pub version: u32,
 }
+
+#[derive(Clone, Debug, Deserialize, Serialize, JsonSchema)]
+pub struct TrustQuorumEpochQueryParam {
+    pub epoch: Option<Epoch>,
+}