ATLAS treats an experiment as a declared, content-addressed protocol. Submitting that protocol expands it into independent controller/scenario/seed rollouts, executes those rollouts in local worker processes, and writes enough evidence to audit, re-evaluate, compare, resume, or replay each attempt.
The implemented path is:
experiment JSON
│ validate + canonicalize
▼
content-derived experiment ID
│ expand controller × scenario × seed
▼
content-derived rollout IDs
│ local process workers
├───────────────┬───────────────┐
▼ ▼ ▼
attempt attempt attempt
│ │ │
└──── traces + metrics + provenance ────┐
▼
filesystem artifacts + SQLite index
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
re-evaluate CI compare reproduce
This is a small simulation platform, not a remote cluster service. Distribution means multiple isolated local Python processes. The rollout contract is intentionally independent of the executor so a batch scheduler or HIL worker can be added without changing experiment identity.
The CI campaign is a compact working example:
uv run atlas submit configs/benchmarks/ci-platform.json \
--store artifacts/atlas \
--workers 4An experiment document declares:
| Field | Meaning |
|---|---|
schema_version |
Version of the experiment contract |
name |
Human-readable campaign name |
controllers |
Controller implementations to compare |
scenarios |
Shared simulated worlds |
seeds or seed_count |
Explicit replayable randomness |
safety_scenarios |
Scenarios that enable the CBF-QP safety filter |
required_metrics |
Quantities eligible for regression enforcement |
metadata |
JSON-serializable intent or study annotations |
Field order and insignificant JSON whitespace do not affect identity. The canonical
document is hashed into an exp_... identifier. Each expanded rollout is hashed into a
run_... identifier. Changing a controller, scenario, seed, safety setting, metric
contract, or experiment metadata creates a different identity.
A rollout can be attempted more than once. Every execution receives a unique
att_... identifier and one terminal status:
| Status | Interpretation |
|---|---|
SUCCEEDED |
Simulation and artifact publication completed |
SIMULATION_FAILED |
Dynamics, controller, or other execution code failed |
SOLVER_FAILED |
An optimization solver reported a hard failure |
TIMED_OUT |
The declared wall-time limit expired |
WORKER_LOST |
A worker terminated without returning structured evidence |
ARTIFACT_PUBLICATION_FAILED |
Execution finished but durable publication failed |
CANCELLED |
The submitter was interrupted before the rollout completed |
SUCCEEDED describes execution, not scientific acceptance. A successful attempt can
still have metrics.success = false; the index records that outcome separately. Failed
and timed-out attempts remain in the denominator and retain structured error evidence.
Submitting the same experiment again skips rollouts that already have a successful
attempt. --force creates new attempts. This makes interrupted campaigns resumable
without overwriting prior evidence.
The filesystem is the evidence plane. SQLite is the query and attempt-state index.
artifacts/atlas/
├── index.sqlite3
└── experiments/
└── exp_<content-hash>/
├── spec.json
└── rollouts/
└── run_<content-hash>/
├── rollout.json
└── attempts/
└── att_<time>_<nonce>/
├── manifest.json
├── result.json
├── metrics.json
├── trace.npz
└── error.json # failure attempts only
Successful manifests record the SHA-256 digest, media type, and byte count of every published artifact. Text and NumPy trace artifacts are written through temporary files, flushed, and atomically replaced. SQLite updates use explicit transactions and foreign keys. The store uses write-ahead logging for reliable local concurrent readers.
Verify an attempt before trusting or moving it:
uv run atlas verify att_<id> --store artifacts/atlasThe command recomputes every declared digest and fails when an artifact is missing, modified, or redirected outside its attempt directory.
uv run atlas status exp_<id> --store artifacts/atlas
uv run atlas inspect run_<id> --store artifacts/atlas
uv run atlas evaluate att_<id> --store artifacts/atlasstatus reports rollout states and scientific failures. inspect resolves either a
specific attempt or the latest attempt for a rollout. evaluate loads the immutable
trace and recomputes metrics with the versioned evaluator without rerunning dynamics.
It fails if artifact integrity is invalid or the recomputed metrics differ from the
stored result.
The trace contains time, plant state, actuator command, reference, and the diagnostics needed by the evaluator. It is stored as compressed numeric arrays with JSON metadata; Python object pickling is disabled when reading it.
The checked-in CI protocol and baseline are:
Create a baseline only from a complete experiment:
uv run atlas baseline exp_<id> \
--store artifacts/atlas \
--output configs/baselines/my-campaign.jsonCompare a completed replay with that baseline:
uv run atlas compare exp_<id> \
--store artifacts/atlas \
--baseline configs/baselines/my-campaign.json \
--output artifacts/atlas/regression.jsonTracking, control-effort, saturation, and violation metrics may improve without causing a failure. Regressions beyond the declared absolute or relative tolerance fail. Minimum barrier value is direction-aware: a sufficiently large decrease fails. Scientific success must remain true. Host-dependent solver latency is recorded as evidence but is not a hard numerical gate.
The committed baseline is a repository regression fixture. It is not a performance claim, a statistical confidence interval, or permission to refresh expected values after unexplained behavior changes.
uv run atlas reproduce att_<id> --store artifacts/atlasReproduction first checks the source commit, tracked dirty-state digest, lockfile, Python implementation and version, operating-system family and architecture, and core package versions. It then creates a new attempt from the exact rollout contract and compares time, state, control, reference, and all non-timing metrics with declared absolute and relative tolerances.
For a failed attempt, ATLAS reuses the recorded execution timeout and compares the new terminal status, exception type, and exception message with the original. A solver failure or timeout can therefore be replayed as a first-class outcome. A lost worker or storage failure is reported as reproduced only if the same failure class actually recurs.
This is exact-input, provenance-checked numerical reproduction. It does not promise
bitwise equality across processors or numerical libraries. A caller can use
--allow-provenance-mismatch to investigate a changed environment, but the report
retains the mismatch and should not be represented as an exact replay.
The next executor can consume the same RolloutSpec and return the same structured
execution result. A remote implementation still needs real work: a queue, leases and
heartbeats, duplicate-attempt handling, object storage, authentication, and recovery
from partial uploads. SQLite is deliberately a single-host index and should not be put
on a shared network filesystem as a substitute for that system.
A HIL executor belongs after the simulation protocol is stable. It must add hardware, firmware, calibration, time-synchronization, communication, safety-supervisor, and operator records. Hardware evidence must remain distinguishable from simulation at the schema level.