test(speculative): add EAGLE-3 fp8 draft convergence smoke for SM89+

[khazic]

What

Adds a turnkey harness to validate fp8 draft-training convergence on fp8-capable hardware (sm_89+), the last open fp8 item in the speculative-decoding tracking issue #2958.

The fp8 draft feature (#2963) shipped with unit tests and an emulate: true numerics path, but was never run through a real fp8 GEMM on real silicon, so issue #2958 still carries this under "Validation debt":

• Validate fp8 draft convergence on SM89+ hardware.

Our fleet is A100/A800 (sm_80, pre-Hopper), which has no fp8 GEMM, so I cannot discharge this here. This PR prepares everything so someone with H100/Ada can run it in one command.

How it validates

run_eagle3_fp8_convergence_smoke.sh trains the same tiny EAGLE-3 draft twice from an identical seed and data order:

• fp8 variant (fp8.emulate: false, real hardware GEMM, compile: on)
• bf16 baseline (fp8.enabled: false)

then asserts:

1. the fp8 run reaches steps and logs finite loss (no NaN/inf from the fp8 GEMM);
2. the fp8 loss trends down over the run (the draft actually learns in fp8);
3. the fp8 loss tail tracks bf16 within a relative tolerance (fp8 quantization did not break convergence vs the bf16 reference).

(2) + (3) together are the convergence check the issue asks for.

It gates on compute capability >= 8.9 and refuses to certify on pre-Hopper GPUs (a pipeline-only escape hatch, ALLOW_NON_SM89=1, prints loudly that it does not discharge the debt).

Files

• examples/speculative/eagle3/qwen3_eagle3_fp8_smoke.yaml: shrunk EAGLE-3 fp8 config (Qwen3-8B target, seq 1024, few dozen samples, a handful of steps) so one H100 reaches a step in a couple of minutes.
• tests/functional_tests/speculative/run_eagle3_fp8_convergence_smoke.sh: the harness (hardware gate, unit tests, data gen, two runs, comparison assertions).

No production code changes: this is validation tooling only. The existing fp8 unit tests (test_spec_draft_fp8_lora.py) are re-run as step 1 of the harness.

Run it (H100/Ada node)

TARGET=/path/to/Qwen3-8B bash tests/functional_tests/speculative/run_eagle3_fp8_convergence_smoke.sh

Ask

@HuiyingLi could you run this on an sm_89+ box (H100/H200/L40S/Ada) and paste the ### 5/5 comparison block back here (and on #2958)? That records the real fp8 hardware validation and lets us tick the last fp8 box in the tracking issue. Knobs if useful: N_ROWS (sample count), TOL (fp8-vs-bf16 tolerance, default 0.15). Thanks!

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[HuiyingLi]

Ran the fp8 convergence smoke on local H100 hardware for PR #3081.

Context:

• PR head: dda3ae3c3bf92f040126da2675ba1f94e0647e14
• GPU: NVIDIA H100 80GB HBM3, compute capability sm_90
• Command: TARGET=Qwen/Qwen3-8B WORK=/lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_nemofw/users/huiyingl/sd/test-runs/pr3081-fp8 N_ROWS=64 TOL=0.15 bash tests/functional_tests/speculative/run_eagle3_fp8_convergence_smoke.sh
• Unit test step: 36 passed

### 5/5  assert: fp8 finite + decreasing, and tracks bf16 within TOL=0.15
fp8 : steps=  3  head=11.0379  tail=5.9552  min=5.9552
bf16: steps=  3  tail=6.4105  min=6.4105
PASS: fp8 loss decreased 11.0379 -> 5.9552.
PASS: fp8 tail tracks bf16 within tol (rel diff 0.071 <= 0.15).

FP8 CONVERGENCE SMOKE: OK

Logs:

• fp8: /lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_nemofw/users/huiyingl/sd/test-runs/pr3081-fp8/fp8.log
• bf16: /lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_nemofw/users/huiyingl/sd/test-runs/pr3081-fp8/bf16.log

[khazic]

The five gb200_L2_* failures on the previous run were an infra flake, not a code issue: the L2 suites themselves passed (coverage.xml was written) and the job died on the post-test coverage step with uuidgen: command not found (exit 127), i.e. that binary is missing on the gb200 runner image. This PR only adds a YAML config and a shell smoke script, no source the L2 jobs import or run, and the same gb200_L2_Datasets job is green on other recent PRs. Pushed a no-op commit to re-trigger CI on a fresh runner.