TQ: Support adding sleds via trust quorum by andrewjstone · Pull Request #9650 · oxidecomputer/omicron

andrewjstone · 2026-01-14T21:47:26Z

This PR introduces two new external APIs to allow adding multiple sleds to a rack at once and to query status about the ongoing operation. It also adds an omdb command for more detailed status. Much more omdb to come in the near future.

This PR also introduces a background task for driving the trust quorum reconfiguration to completion. Reconfiguration is driven by two steps. Synchronously updating the DB in the new external endpoint handler and then asynchronously trying to commit the operation via the background task.

I tested this on a4x2 and it works as expected. See the trace from the original external API test below:

➜  oxide.rs git:(main) ✗ echo '{"rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28", "sled_ids": [{"part": "PPP-PPPPPPP","serial": "00000000002"}]}' | target/debug/oxide --profile recovery api /v1/trust-quorum/new-members --method POST --input -
➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery  api /v1/trust-quorum/config/latest/0dbef452-a6dd-4831-bbdc-769ea3353f28
{
  "abort_reason": null,
  "commit_crash_tolerance": 1,
  "coordinator": {
    "part_number": "PPP-PPPPPPP",
    "serial_number": "00000000003"
  },
  "encrypted_rack_secrets": null,
  "epoch": 2,
  "last_committed_epoch": 1,
  "members": {
    "PPP-PPPPPPP:00000000000": {
      "share_digest": null,
      "state": "unacked",
      "time_committed": null,
      "time_prepared": null
    },
    "PPP-PPPPPPP:00000000001": {
      "share_digest": null,
      "state": "unacked",
      "time_committed": null,
      "time_prepared": null
    },
    "PPP-PPPPPPP:00000000002": {
      "share_digest": null,
      "state": "unacked",
      "time_committed": null,
      "time_prepared": null
    },
    "PPP-PPPPPPP:00000000003": {
      "share_digest": null,
      "state": "unacked",
      "time_committed": null,
      "time_prepared": null
    }
  },
  "rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28",
  "state": "preparing",
  "threshold": 3,
  "time_aborted": null,
  "time_committed": null,
  "time_committing": null,
  "time_created": "2026-01-14T21:32:18.780136Z"
}
➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery  api /v1/trust-quorum/config/latest/0dbef452-a6dd-4831-bbdc-769ea3353f28
{
  "abort_reason": null,
  "commit_crash_tolerance": 1,
  "coordinator": {
    "part_number": "PPP-PPPPPPP",
    "serial_number": "00000000003"
  },
  "encrypted_rack_secrets": null,
  "epoch": 2,
  "last_committed_epoch": 1,
  "members": {
    "PPP-PPPPPPP:00000000000": {
      "share_digest": "fcfb09128c84d82cc81b200c6c682510f63160a4417856f4041b1886445e8b14",
      "state": "prepared",
      "time_committed": null,
      "time_prepared": "2026-01-14T21:32:55.826622Z"
    },
    "PPP-PPPPPPP:00000000001": {
      "share_digest": "d8cad02bd3bccd08109a79e3bf6d8dab0d460a0ba879bf42887dc0fc8d855786",
      "state": "prepared",
      "time_committed": null,
      "time_prepared": "2026-01-14T21:32:55.848235Z"
    },
    "PPP-PPPPPPP:00000000002": {
      "share_digest": "dd57ad8e271734fabfe97d6180d6da3e5c3805e17dacf58e0f2a6d5ed7f1242b",
      "state": "prepared",
      "time_committed": null,
      "time_prepared": "2026-01-14T21:32:55.806644Z"
    },
    "PPP-PPPPPPP:00000000003": {
      "share_digest": "6b27327ca49976ccca83972e6578ef195c99489e62811e8d0a0cb061fca9c0c4",
      "state": "prepared",
      "time_committed": null,
      "time_prepared": "2026-01-14T21:32:55.837154Z"
    }
  },
  "rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28",
  "state": "preparing",
  "threshold": 3,
  "time_aborted": null,
  "time_committed": null,
  "time_committing": null,
  "time_created": "2026-01-14T21:32:18.780136Z"
}
➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery  api /v1/trust-quorum/config/latest/0dbef452-a6dd-4831-bbdc-769ea3353f28
{
  "abort_reason": null,
  "commit_crash_tolerance": 1,
  "coordinator": {
    "part_number": "PPP-PPPPPPP",
    "serial_number": "00000000003"
  },
  "encrypted_rack_secrets": {
    "data": "53de7731deec3f298a7f5067e256a63bb2869a91c9710d9b23dbf3d261d1b730039d9cb11b543c14906ff77cd409d32953959e9ff8933858",
    "salt": "ec609ed5ff7aee94e2e88ad94af56e0cbb8a66a683294005c7888f60a627956a"
  },
  "epoch": 2,
  "last_committed_epoch": 1,
  "members": {
    "PPP-PPPPPPP:00000000000": {
      "share_digest": "fcfb09128c84d82cc81b200c6c682510f63160a4417856f4041b1886445e8b14",
      "state": "committed",
      "time_committed": "2026-01-14T21:33:03.864617Z",
      "time_prepared": "2026-01-14T21:32:55.826622Z"
    },
    "PPP-PPPPPPP:00000000001": {
      "share_digest": "d8cad02bd3bccd08109a79e3bf6d8dab0d460a0ba879bf42887dc0fc8d855786",
      "state": "committed",
      "time_committed": "2026-01-14T21:33:03.864617Z",
      "time_prepared": "2026-01-14T21:32:55.848235Z"
    },
    "PPP-PPPPPPP:00000000002": {
      "share_digest": "dd57ad8e271734fabfe97d6180d6da3e5c3805e17dacf58e0f2a6d5ed7f1242b",
      "state": "committed",
      "time_committed": "2026-01-14T21:33:03.864617Z",
      "time_prepared": "2026-01-14T21:32:55.806644Z"
    },
    "PPP-PPPPPPP:00000000003": {
      "share_digest": "6b27327ca49976ccca83972e6578ef195c99489e62811e8d0a0cb061fca9c0c4",
      "state": "committed",
      "time_committed": "2026-01-14T21:33:03.864617Z",
      "time_prepared": "2026-01-14T21:32:55.837154Z"
    }
  },
  "rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28",
  "state": "committed",
  "threshold": 3,
  "time_aborted": null,
  "time_committed": "2026-01-14T21:33:04.652543Z",
  "time_committing": "2026-01-14T21:32:55.861158Z",
  "time_created": "2026-01-14T21:32:18.780136Z"
}
➜  oxide.rs git:(main) ✗

sled-agent/src/bootstrap/trust_quorum_setup.rs

jgallagher

Looks good! Just a bunch of nits and small suggestions

dev-tools/omdb/src/bin/omdb/nexus.rs

nexus/db-queries/src/db/datastore/sled.rs

nexus/external-api/src/lib.rs

nexus/types/src/external_api/views.rs

nexus/src/app/background/tasks/trust_quorum.rs

andrewjstone · 2026-01-15T23:19:04Z

Thanks for the comprehensive review @jgallagher. I think I fixed up everything. Let me know if you see anything else!

jgallagher

Changes LGTM, just a couple minor style suggestions I didn't notice the first time around.

Happy to approve with a couple caveats:

Your comment about resetting TRUST_QUORUM_INTEGRATION_ENABLED still needs to be applied
Would strongly prefer an external-API-focused set of eyes to review those bits (maybe @ahl?)

jgallagher · 2026-01-16T15:06:21Z

nexus/external-api/src/lib.rs

+        rqctx: RequestContext<Self::Context>,
+        path_params: Path<params::RackPath>,
+        req: TypedBody<params::AddSledsRequest>,
+    ) -> Result<HttpResponseOk<Epoch>, HttpError>;


Do the type names leak out into the API doc? If so, maybe RackMembershipEpoch would be more clear? If not, ignore this.

No leaking as far as I can tell.

"properties": { "epoch": { "description": "The generation / version of the configuration", "type": "integer", "format": "uint64", "minimum": 0 },

Yeah--in this case the type is #[serde(transparent)] so it just looks like a number. However, @andrewjstone, I think you're looking at the wrong place in the OpenAPI output:

"responses": { "200": { "description": "successful operation", "content": { "application/json": { "schema": { "title": "uint64", "type": "integer", "format": "uint64", "minimum": 0 } } }

It is at least a little weird (for us) to be returning a literal integer rather than, say, an object with a field that is an integer.

nexus/types/src/external_api/views.rs

nexus/src/app/background/tasks/trust_quorum.rs

andrewjstone · 2026-01-17T03:49:09Z

Tested in a4x2 with all the latest changes.

➜  oxide.rs git:(main) ✗ echo '{"sled_ids": [{"part": "PPP-PPPPPPP","serial": "00000000002"}]}' |  target/debug/oxide --profile recovery api /v1/system/hardware/racks/3df0b7cd-50ed-498e-afa4-41f178d393f9/sleds --method POST --input -
2
➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api /v1/system/hardware/racks/3df0b7cd-50ed-498e-afa4-41f178d393f9/sleds/2
{
  "epoch": 2,
  "members": [
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000000"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000001"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000002"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000003"
    }
  ],
  "rack_id": "3df0b7cd-50ed-498e-afa4-41f178d393f9",
  "state": "committed",
  "time_aborted": null,
  "time_committed": "2026-01-17T03:45:33.946797Z",
  "time_created": "2026-01-17T03:45:02.262102Z",
  "unacknowledged_members": []
}

I even was able to catch the bg task doing something in OMDB:

root@oxz_switch:~# omdb nexus --nexus-internal-url='http://[fd00:17:1:d01::6]:12232' background-tasks show trust_quorum_manager
note: using Nexus URL http://[fd00:17:1:d01::6]:12232
task: "trust_quorum_manager"
  configured period: every 1m
  currently executing: no
  last completed activation: iter 19, triggered by a periodic timer firing
    started at 2026-01-17T03:45:26.931Z (34s ago) and ran for 7643ms
Trust Quorum status for rack_id: 3df0b7cd-50ed-498e-afa4-41f178d393f9, epoch: 2:
    Commit Status from client PPP-PPPPPPP:00000000003:
      acked: PPP-PPPPPPP:00000000000
      pending:
      errors:
    Commit Status from client PPP-PPPPPPP:00000000001:
      acked: PPP-PPPPPPP:00000000001
      pending:
      errors:
    Commit Status from client PPP-PPPPPPP:00000000000:
      acked: PPP-PPPPPPP:00000000002, PPP-PPPPPPP:00000000003
      pending:
      errors:

All that remains is for someone to take a look over the external API.

ahl

I think I've added a bunch of quibbling comments that reflect a lack of my own understanding about how this is intended to be exposed to and used by customers. If there's something I should read or if you have some time to talk, I'd love to understand this better.

ahl · 2026-01-18T18:59:40Z

nexus/external-api/src/lib.rs

+        rqctx: RequestContext<Self::Context>,
+        path_params: Path<params::RackPath>,
+        req: TypedBody<params::AddSledsRequest>,
+    ) -> Result<HttpResponseOk<Epoch>, HttpError>;


Yeah--in this case the type is #[serde(transparent)] so it just looks like a number. However, @andrewjstone, I think you're looking at the wrong place in the OpenAPI output:

"responses": { "200": { "description": "successful operation", "content": { "application/json": { "schema": { "title": "uint64", "type": "integer", "format": "uint64", "minimum": 0 } } }

It is at least a little weird (for us) to be returning a literal integer rather than, say, an object with a field that is an integer.

nexus/external-api/src/lib.rs

nexus/types/src/external_api/params.rs

nexus/types/src/external_api/views.rs

nexus/external-api/src/lib.rs

nexus/src/external_api/http_entrypoints.rs

This PR introduces two new external APIs to allow adding multiple sleds to a rack at once and to query status about the ongoing operation. Both are currently experimental and live under `/v1/trust-quorum`. They need to be moved under `/system/hardware` like the original `sled-add` command. They also need to be reworked to not report trust quorum specific details if it can be avoided. Most of that should be in omdb for debugging. I may add some of that support to this PR. This PR also introduces a background task for driving the trust quorum reconfiguration to completion. Reconfiguration is driven by two steps. Synchronously updating the DB in the new external endpoint handler and then asynchronously trying to commit the operation via the background task. I tested this on a4x2 and it works as expected. See the trace from the original external API test below: ``` ➜ oxide.rs git:(main) ✗ echo '{"rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28", "sled_ids": [{"part": "PPP-PPPPPPP","serial": "00000000002"}]}' | target/debug/oxide --profile recovery api /v1/trust-quorum/new-members --method POST --input - ➜ oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api /v1/trust-quorum/config/latest/0dbef452-a6dd-4831-bbdc-769ea3353f28 { "abort_reason": null, "commit_crash_tolerance": 1, "coordinator": { "part_number": "PPP-PPPPPPP", "serial_number": "00000000003" }, "encrypted_rack_secrets": null, "epoch": 2, "last_committed_epoch": 1, "members": { "PPP-PPPPPPP:00000000000": { "share_digest": null, "state": "unacked", "time_committed": null, "time_prepared": null }, "PPP-PPPPPPP:00000000001": { "share_digest": null, "state": "unacked", "time_committed": null, "time_prepared": null }, "PPP-PPPPPPP:00000000002": { "share_digest": null, "state": "unacked", "time_committed": null, "time_prepared": null }, "PPP-PPPPPPP:00000000003": { "share_digest": null, "state": "unacked", "time_committed": null, "time_prepared": null } }, "rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28", "state": "preparing", "threshold": 3, "time_aborted": null, "time_committed": null, "time_committing": null, "time_created": "2026-01-14T21:32:18.780136Z" } ➜ oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api /v1/trust-quorum/config/latest/0dbef452-a6dd-4831-bbdc-769ea3353f28 { "abort_reason": null, "commit_crash_tolerance": 1, "coordinator": { "part_number": "PPP-PPPPPPP", "serial_number": "00000000003" }, "encrypted_rack_secrets": null, "epoch": 2, "last_committed_epoch": 1, "members": { "PPP-PPPPPPP:00000000000": { "share_digest": "fcfb09128c84d82cc81b200c6c682510f63160a4417856f4041b1886445e8b14", "state": "prepared", "time_committed": null, "time_prepared": "2026-01-14T21:32:55.826622Z" }, "PPP-PPPPPPP:00000000001": { "share_digest": "d8cad02bd3bccd08109a79e3bf6d8dab0d460a0ba879bf42887dc0fc8d855786", "state": "prepared", "time_committed": null, "time_prepared": "2026-01-14T21:32:55.848235Z" }, "PPP-PPPPPPP:00000000002": { "share_digest": "dd57ad8e271734fabfe97d6180d6da3e5c3805e17dacf58e0f2a6d5ed7f1242b", "state": "prepared", "time_committed": null, "time_prepared": "2026-01-14T21:32:55.806644Z" }, "PPP-PPPPPPP:00000000003": { "share_digest": "6b27327ca49976ccca83972e6578ef195c99489e62811e8d0a0cb061fca9c0c4", "state": "prepared", "time_committed": null, "time_prepared": "2026-01-14T21:32:55.837154Z" } }, "rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28", "state": "preparing", "threshold": 3, "time_aborted": null, "time_committed": null, "time_committing": null, "time_created": "2026-01-14T21:32:18.780136Z" } ➜ oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api /v1/trust-quorum/config/latest/0dbef452-a6dd-4831-bbdc-769ea3353f28 { "abort_reason": null, "commit_crash_tolerance": 1, "coordinator": { "part_number": "PPP-PPPPPPP", "serial_number": "00000000003" }, "encrypted_rack_secrets": { "data": "53de7731deec3f298a7f5067e256a63bb2869a91c9710d9b23dbf3d261d1b730039d9cb11b543c14906ff77cd409d32953959e9ff8933858", "salt": "ec609ed5ff7aee94e2e88ad94af56e0cbb8a66a683294005c7888f60a627956a" }, "epoch": 2, "last_committed_epoch": 1, "members": { "PPP-PPPPPPP:00000000000": { "share_digest": "fcfb09128c84d82cc81b200c6c682510f63160a4417856f4041b1886445e8b14", "state": "committed", "time_committed": "2026-01-14T21:33:03.864617Z", "time_prepared": "2026-01-14T21:32:55.826622Z" }, "PPP-PPPPPPP:00000000001": { "share_digest": "d8cad02bd3bccd08109a79e3bf6d8dab0d460a0ba879bf42887dc0fc8d855786", "state": "committed", "time_committed": "2026-01-14T21:33:03.864617Z", "time_prepared": "2026-01-14T21:32:55.848235Z" }, "PPP-PPPPPPP:00000000002": { "share_digest": "dd57ad8e271734fabfe97d6180d6da3e5c3805e17dacf58e0f2a6d5ed7f1242b", "state": "committed", "time_committed": "2026-01-14T21:33:03.864617Z", "time_prepared": "2026-01-14T21:32:55.806644Z" }, "PPP-PPPPPPP:00000000003": { "share_digest": "6b27327ca49976ccca83972e6578ef195c99489e62811e8d0a0cb061fca9c0c4", "state": "committed", "time_committed": "2026-01-14T21:33:03.864617Z", "time_prepared": "2026-01-14T21:32:55.837154Z" } }, "rack_id": "0dbef452-a6dd-4831-bbdc-769ea3353f28", "state": "committed", "threshold": 3, "time_aborted": null, "time_committed": "2026-01-14T21:33:04.652543Z", "time_committing": "2026-01-14T21:32:55.861158Z", "time_created": "2026-01-14T21:32:18.780136Z" } ➜ oxide.rs git:(main) ✗ ```

ahl

love it! some small doc suggestions that you should feel free to ignore.

nexus/external-api/src/lib.rs

ahl · 2026-01-21T18:18:28Z

nexus/types/src/external_api/views.rs

+    pub rack_id: Uuid,
+    pub version: RackMembershipVersion,
+    pub state: RackMembershipChangeState,
+    /// All members of the rack for this epoch


Suggested change

/// All members of the rack for this epoch

/// All members of the rack for this version

nexus/types/src/external_api/views.rs

ahl · 2026-01-21T18:21:22Z

nexus/types/src/external_api/params.rs


+#[derive(Clone, Debug, Deserialize, Serialize, JsonSchema)]
+pub struct RackMembershipVersionParam {
+    pub version: Option<shared::RackMembershipVersion>,


I'm not sure this is right

Suggested change

pub version: Option<shared::RackMembershipVersion>,

/// Membership version as returned from other membership interfaces

pub version: Option<shared::RackMembershipVersion>,

Skipped this one.

ahl · 2026-01-21T18:22:12Z

nexus/types/src/external_api/views.rs

+#[derive(Clone, Debug, Deserialize, Serialize, JsonSchema)]
+pub struct RackMembershipStatus {
+    pub rack_id: Uuid,
+    pub version: RackMembershipVersion,


Suggesting a comment, but I don't love what I've written:

Suggested change

pub version: RackMembershipVersion,

/// Membership version that can be used to check status.

pub version: RackMembershipVersion,

I used a slightly different wording

ahl · 2026-01-21T18:22:50Z

nexus/types/src/external_api/views.rs

+/// Status of the rack membership uniquely identified by the (rack_id, version)
+/// pair
+#[derive(Clone, Debug, Deserialize, Serialize, JsonSchema)]
+pub struct RackMembershipStatus {


Comments on these fields become part of the public docs so it might be worthwhile to put something for each of them (though not critical).

Mostly skipped.

nexus/types/src/external_api/params.rs

andrewjstone · 2026-01-21T22:15:27Z

Tested via latest API changes:

➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api /v1/system/hardware/racks/853c97d9-b5a1-4462-94d7-358fb3185ca7/membership{
  "members": [
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000000"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000001"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000003"
    }
  ],
  "rack_id": "853c97d9-b5a1-4462-94d7-358fb3185ca7",
  "state": "committed",
  "time_aborted": null,
  "time_committed": "2026-01-21T21:48:56.048712Z",
  "time_created": "2026-01-21T21:48:56.048712Z",
  "unacknowledged_members": [],
  "version": 1
}

➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api '/v1/system/hardware/racks/853c97d9-b5a1-4462-94d7-358fb3185ca7/membership?version=1'

{
  "members": [
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000000"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000001"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000003"
    }
  ],
  "rack_id": "853c97d9-b5a1-4462-94d7-358fb3185ca7",
  "state": "committed",
  "time_aborted": null,
  "time_committed": "2026-01-21T21:48:56.048712Z",
  "time_created": "2026-01-21T21:48:56.048712Z",
  "unacknowledged_members": [],
  "version": 1
}

➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api '/v1/system/hardware/racks/853c97d9-b5a1-4462-94d7-358fb3185ca7/membership?version=2'
error; status code: 404 Not Found
{
  "error_code": "Not Found",
  "message": "could not find rack membership",
  "request_id": "1aca782e-af21-4463-98cb-1691ca5f6dda"
}
error

➜  oxide.rs git:(main) ✗ echo '{"sled_ids": [{"part_number": "PPP-PPPPPPP","serial_number": "00000000002"}]}' |  target/debug/oxide --profile recovery api /v1/system/hardware/racks/853c97d9-b5a1-4462-94d7-358fb3185ca7/membership/add --method POST --input -
{
  "members": [
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000000"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000001"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000002"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000003"
    }
  ],
  "rack_id": "853c97d9-b5a1-4462-94d7-358fb3185ca7",
  "state": "in_progress",
  "time_aborted": null,
  "time_committed": null,
  "time_created": "2026-01-21T22:10:10.767645Z",
  "unacknowledged_members": [
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000000"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000001"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000002"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000003"
    }
  ],
  "version": 2
}
➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api '/v1/system/hardware/racks/853c97d9-b5a1-4462-94d7-358fb3185ca7/membership?version=2'
{
  "members": [
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000000"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000001"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000002"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000003"
    }
  ],
  "rack_id": "853c97d9-b5a1-4462-94d7-358fb3185ca7",
  "state": "committed",
  "time_aborted": null,
  "time_committed": "2026-01-21T22:10:35.319419Z",
  "time_created": "2026-01-21T22:10:10.767645Z",
  "unacknowledged_members": [],
  "version": 2
}

➜  oxide.rs git:(main) ✗ target/debug/oxide --profile recovery api '/v1/system/hardware/racks/853c97d9-b5a1-4462-94d7-358fb3185ca7/membership'
{
  "members": [
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000000"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000001"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000002"
    },
    {
      "part_number": "PPP-PPPPPPP",
      "serial_number": "00000000003"
    }
  ],
  "rack_id": "853c97d9-b5a1-4462-94d7-358fb3185ca7",
  "state": "committed",
  "time_aborted": null,
  "time_committed": "2026-01-21T22:10:35.319419Z",
  "time_created": "2026-01-21T22:10:10.767645Z",
  "unacknowledged_members": [],
  "version": 2
}

andrewjstone force-pushed the tq-nexus-api-2 branch from 04173b6 to 2d8fc67 Compare January 14, 2026 23:41

andrewjstone commented Jan 15, 2026

View reviewed changes

sled-agent/src/bootstrap/trust_quorum_setup.rs Outdated Show resolved Hide resolved

andrewjstone changed the title ~~WIP: TQ: Support adding sleds via trust quorum~~ TQ: Support adding sleds via trust quorum Jan 15, 2026

andrewjstone marked this pull request as ready for review January 15, 2026 16:43

andrewjstone requested review from davepacheco and jgallagher January 15, 2026 16:43

jgallagher reviewed Jan 15, 2026

View reviewed changes

andrewjstone commented Jan 15, 2026

View reviewed changes

nexus/src/app/background/tasks/trust_quorum.rs Outdated Show resolved Hide resolved

jgallagher approved these changes Jan 16, 2026

View reviewed changes

ahl reviewed Jan 18, 2026

View reviewed changes

ahl approved these changes Jan 21, 2026

View reviewed changes

andrewjstone added 14 commits January 21, 2026 18:47

Migrate get api to lockstep and rename some paths

f9188d1

Add omdb status

b9276a5

Make membership status less trust quorum specific

36f384c

remove unrelated comment

003345b

remove stale comment

f32d4b1

clippy

76b6db0

Actually commit bg task config changes

0403373

Better external API

634fc70

Fix return value and errors for bgtask/omdb

faceaf5

more review fixes

c7bcbd8

fill in missing nexus config

f02edfd

review updates and fix tests

ec840a2

review feedback on external API

7742597

openapi

2f873e2

andrewjstone force-pushed the tq-nexus-api-2 branch from 416eaaf to 2f873e2 Compare January 21, 2026 18:54

doc comments, test fixes, etc...

704327f

andrewjstone enabled auto-merge (squash) January 21, 2026 22:42

andrewjstone merged commit 6cef874 into main Jan 22, 2026
18 checks passed

andrewjstone deleted the tq-nexus-api-2 branch January 22, 2026 01:35

	/// All members of the rack for this epoch
	/// All members of the rack for this version

	pub version: Option<shared::RackMembershipVersion>,
	/// Membership version as returned from other membership interfaces
	pub version: Option<shared::RackMembershipVersion>,

	pub version: RackMembershipVersion,
	/// Membership version that can be used to check status.
	pub version: RackMembershipVersion,

Conversation

andrewjstone commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jgallagher left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewjstone commented Jan 15, 2026

Uh oh!

jgallagher left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewjstone Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewjstone commented Jan 17, 2026

Uh oh!

ahl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ahl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andrewjstone commented Jan 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

andrewjstone commented Jan 14, 2026 •

edited

Loading

andrewjstone Jan 16, 2026 •

edited

Loading