Skip to content

benchmarking.md

Latest commit

 

History

History
93 lines (68 loc) · 3.54 KB

benchmarking.md

File metadata and controls

93 lines (68 loc) · 3.54 KB
Raw

Benchmark Protocol

Purpose

The benchmark layer exists to prevent controller demonstrations from being mistaken for comparative research. Every comparison must use identical plant parameters, initial conditions, disturbances, seeds, constraints, sample times, references, and success criteria.

Built-in scenarios

Scenario Task Declared stressors
hover Recover to (0, 0, 1) Position and attitude offset
step Three-axis position step Reference discontinuity and actuator demand
figure_eight Analytic trajectory tracking Curvature and continuous feedforward
wind_mass Figure-eight tracking 15% mass mismatch, constant wind force, measurement noise
obstacle_avoidance Step tracking around a protected sphere CBF intervention and safe-set monitoring

Scenario values are defined in scenarios.py, not inferred from plots.

Recorded metrics

For position error e_p,k over N samples:

position RMSE = sqrt((1/N) sum ||e_p,k||^2)

ATLAS also records velocity RMSE, maximum and final position error, rotor-thrust RMS, saturation fraction, constraint-violation count, minimum barrier value, and solver latency percentiles p50, p95, and p99.

Success is a declared Boolean criterion combining:

  • a complete finite simulation;
  • scenario-specific final and maximum position-error thresholds;
  • zero recorded safety-constraint violations.

Thresholds are engineering acceptance criteria for this repository, not universal standards.

Determinism and uncertainty

Each run stores its integer seed. NumPy's Generator supplies simulated measurement noise. Deterministic scenarios should produce bitwise-stable configuration and numerically equivalent metrics on the same dependency solution. Monte Carlo reporting must include seed count, success rate, mean, and spread; one favorable seed is not evidence.

The default campaign requests 20 seeds. This is a development gate, not a statistical power claim. Any publication should justify sample count against the tested hypothesis.

Artifact schema

atlas-bench writes JSON with:

  • schema_version;
  • full scenario configuration;
  • controller identifier;
  • seed count;
  • summary statistics;
  • per-seed metrics.

Generated artifacts are ignored by Git by default. Publication artifacts should be placed in a versioned release or archival repository with machine and dependency metadata.

For auditable studies, the atlas platform adds a content-addressed experiment spec, one durable attempt directory per rollout execution, full numeric traces, evaluator diagnostics, runtime provenance, checksums, a SQLite status index, resumability, regression comparison, and exact-input replay. See Experiment Platform.

The estimator has a separate consistency campaign:

uv run atlas-estimator-bench --seeds 20 --output artifacts/eskf.json

It reports NEES against 15-state chi-square bounds and position/orientation NIS against three-dimensional chi-square bounds for an analytic moving-platform simulation.

Fair-comparison rules

  1. Do not tune a controller on the evaluation seeds.
  2. Do not change actuator or state constraints between controllers.
  3. Report failures and infeasible solves in the denominator.
  4. Report timing on the same machine and execution mode.
  5. Separate warm-up/JIT compilation time from steady-state solve time.
  6. State whether the controller receives truth, noisy measurements, or estimates.
  7. Do not call simulated outcomes measured results.