meta-nvidia: gate GPU ready state on local TEE attestation (nvattest)

[kvinwang]

Summary

Previously, nvidia-persistenced.service set the confidential-compute GPU ready state (nvidia-smi conf-compute -srs 1) unconditionally at boot — the GPU accepted work whether or not its TEE state had been verified.

This PR gates the ready state on local GPU TEE attestation, with the gate implemented in dstack-util setup and driven by an app-compose requirement (Dstack-TEE/dstack#765):

• New nvattest recipe building NVIDIA's attestation-sdk CLI (the official successor to the Python local_gpu_verifier), pinned to a fixed SRCREV, Apache-2.0. The recipe ships only the CLI — no attestation service, wrapper script, or unit-gating drop-ins.
• dstack-util setup (dstack-prepare.service) enforces the new requirements.verify_gpu field (default true): when an NVIDIA GPU is on the PCI bus it runs nvattest attest --device gpu --verifier local --nonce <32 random bytes> before key provisioning, and only sets the ready state after attestation succeeds. Any failure — non-CC GPU, CC mode left off by the host, missing nvattest — aborts boot fail-closed (boot.error to the VMM, dstack-prepare reboots). verify_gpu: false opts out and sets the ready state without verification.
• The only systemd change on the nvattest side is an ordering drop-in: dstack-prepare.service runs after nvidia-persistenced/nvidia-fabricmanager (driver loaded, fabric up). GPU-less instances skip everything (PCI detection in dstack-util; nvidia units skip via nvidia-gpu-detect).
• Verifier output is kept in /run/nvidia-gpu-attestation/attestation.out (0600) for later exposure via the guest agent.

Depends on Dstack-TEE/dstack#765; the dstack submodule will be bumped once that merges.

Build / reproducibility notes

• The SDK embeds the regorus Rego engine (Rust) via Corrosion → uses the prebuilt meta-rust-bin toolchain (cargo-bin-native), not oe-core's source-bootstrapped rust-native; host/target linker split handled like cargo_bin.bbclass (host triple == target triple).
• USE_SYSTEM_DEPS=ON: links sysroot openssl/curl/libxml2/xmlsec1 instead of ExternalProject-downloaded static copies.
• RUSTFLAGS --remap-path-prefix keeps build paths out of libnvat (passes [buildpaths] QA).
• Only nvattest + libnvat are shipped; cargo proc-macro host dylibs are excluded.
• Remaining TODO (comment in recipe): vendor CMake FetchContent + cargo deps via SRC_URI for a fully offline fetch (all upstream refs are already pinned).

Tested

The same nvattest invocation (local verifier + random nonce) was verified end-to-end on GCP a3-highgpu-1g (TDX + 1×H100 80GB, SPOT) in a dstack CVM built from the earlier systemd-service iteration of this branch:

    Hardware Model: GH100 A01 GSP BROM
    VBIOS Version: 96.00.CF.00.01
    Driver Version: 595.58.03
    Measurement Result: success
    Attestation Report Cert Chain:  Status: valid, OCSP: good
    Driver RIM Cert Chain:          Status: valid, OCSP: good
    VBIOS RIM Cert Chain:           Status: valid, OCSP: good
GPU attestation was successful

Workload container (after the gate released app-compose):

NVIDIA H100 80GB HBM3 | Driver 595.58.03 | CUDA 13.2
CC status: ON
Confidential Compute GPUs Ready state: ready

Boot-time overhead of the attestation step (evidence + RIM fetch + OCSP): ~2s. An end-to-end run of the dstack-util-based gate will follow once the submodule is bumped and the image rebuilt.

Follow-ups (not in this PR)

• Bump the dstack submodule to include guest: gate boot on GPU TEE attestation via requirements.verify_gpu dstack#765, then re-verify end-to-end (incl. the verify_gpu: false opt-out path).
• Bind the attestation nonce to the TDX quote (report_data derivation) and expose GPU evidence through the guest agent's attestation channel.
• Vendor FetchContent/cargo dependencies for fully offline builds.

[Copilot]

Pull request overview

This PR changes the NVIDIA-enabled image boot flow so the confidential-compute GPU “ready” state is only set after local GPU TEE attestation succeeds, preventing the GPU from accepting work before verification.

Changes:

• Remove unconditional nvidia-smi conf-compute -srs 1 from nvidia-persistenced.service; make attestation the sole place that sets the ready state.
• Add a new nvattest BitBake recipe that builds NVIDIA’s attestation CLI and installs a oneshot nvidia-gpu-attestation.service plus gating drop-ins for dependent services.
• Add a boot-time attestation runner (nvidia-gpu-attest) and include the nvattest package in the NVIDIA rootfs image group.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
meta-nvidia/recipes-graphics/nvidia/files/nvidia-persistenced.service	Removes ready-state post hook and documents attestation-owned readiness.
meta-nvidia/recipes-graphics/nvattest/nvattest_2026.06.09.bb	New recipe building nvattest/libnvat and installing systemd units + gating drop-ins.
meta-nvidia/recipes-graphics/nvattest/files/nvidia-gpu-attestation.service	New oneshot service that runs the attestation gate (and currently reboots on failure).
meta-nvidia/recipes-graphics/nvattest/files/nvidia-gpu-attestation.default	Default configuration for attestation args/output and chrony behavior.
meta-nvidia/recipes-graphics/nvattest/files/nvidia-gpu-attestation-gate.conf	Drop-in to gate key services on successful attestation.
meta-nvidia/recipes-graphics/nvattest/files/nvidia-gpu-attest	Boot-time script that runs nvattest, stores results in /run, then sets CC ready state.
meta-dstack/recipes-core/images/dstack-rootfs-nvidia.inc	Adds nvattest to the NVIDIA image package group.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[kvinwang]

Update: offline RIM verification (commit 4de6c33)

RIM reference measurements are now pre-provisioned in the measured rootfs instead of being fetched from the NVIDIA RIM service at boot:

• Bundles all 155 RIMs currently published by the RIM service: 151 VBIOS RIMs (every CC-capable board/firmware: H100 SXM/PCIe/NVL, H200, GH200, Blackwell) + the 4 driver RIMs for the image's 595.58.03 driver (GH100/GB100/GB110/GB202). ~3.4 MB raw, ~0.7 MB compressed in the squashfs (~0.15% rootfs growth); disk.raw size unchanged.
• Each file was sha256-verified against the RIM service metadata at vendoring time; nvattest additionally validates RIM signatures against the NVIDIA CA at every use, so bundling adds no new trust assumptions — and pins the golden values inside the measured image (a hijacked RIM service can no longer feed poisoned reference values to deployed fleets).
• The wrapper attests with --rim-store dir by default and falls back to the remote RIM service only if the local store misses (e.g. firmware released after the image was built). Tunables: NV_GPU_ATTESTATION_OFFLINE_RIM, NV_GPU_ATTESTATION_RIM_DIR in /etc/default/nvidia-gpu-attestation.
• OCSP remains online (the SDK's LocalGpuVerifier requires an OCSP client; follow-up options: LAN caching proxy, or an upstream file-based OCSP client).

Tested

Fresh GCP a3-highgpu-1g SPOT instance — which happened to have a different VBIOS (96.00.D9.00.01) than the board used in the first test (96.00.CF.00.01), exercising the fleet-wide coverage:

nvidia-gpu-attest[591]: Attesting with local RIM store: /usr/share/nvidia/rims
nvidia-gpu-attest[602]:     VBIOS Version: 96.00.D9.00.01
nvidia-gpu-attest[602]:     Measurement Result: success
nvidia-gpu-attest[602]: GPU attestation was successful
nvidia-gpu-attest[591]: GPU TEE attestation succeeded; GPU ready state set.

Workload container: CC status: ON / Confidential Compute GPUs Ready state: ready.

[kvinwang]

Re-verified end-to-end with the dstack-util-based gate (image built from this branch + Dstack-TEE/dstack#765 via SRC_DIR:pn-dstack-guest override), on GCP a3-highgpu-1g (TDX + H100, SPOT): default path attests in ~1.8s inside dstack-prepare before key provisioning (Measurement Result: success, RIM/OCSP good) and only then sets the ready state; requirements.verify_gpu: false skips nvattest and sets the ready state directly. Workload containers see CC status: ON / Ready state: ready in both cases. Details: Dstack-TEE/dstack#765 (comment)