Practice failures. Verify recovery.
OpsRehearse is a self-hosted platform for rehearsing IT failures, validating recovery procedures, and improving operational readiness.
The product follows one operating loop:
Prepare -> Rehearse -> Observe -> Diagnose -> Recover -> Verify -> Improve
Version 0.1 proves that loop with one built-in Help Desk target and one reproducible database connection exhaustion scenario. It is a local pilot and learning release. It is not an enterprise-ready release.
An IT support, operations, or DevOps practitioner who prepares and runs an exercise. In v0.1, this person starts the scenario, records observations, follows the Runbook, initiates recovery, reviews verification results, and records improvement items.
The person responsible for the target service and its recovery procedure. In v0.1, the operator may also act as the system owner. The role exists in the product model so each Runbook and improvement item has a clear owner.
A team lead, mentor, or second practitioner who reviews the after-action report and checks whether the evidence supports the result. Review is a workflow responsibility in v0.1, not a separate permission role.
An actor inside the built-in Help Desk target who submits a support request. The demo employee is not an OpsRehearse user and has no access to the control plane.
- Run a repeatable technical exercise against the built-in Help Desk target.
- Confirm that a Runbook contains enough information to diagnose and recover the failure.
- Collect bounded health-check and log evidence during the exercise.
- Verify recovery with system checks rather than an operator assertion.
- Reset the target to a known baseline and repeat the same exercise.
- Produce an after-action report with evidence, timing, results, and improvement items.
- The operator opens the database connection exhaustion scenario.
- OpsRehearse checks that the target is the managed demo environment, no other exercise is active, reset state is clean, and all baseline checks pass.
- A demo employee submits a support request.
- The Help Desk API stores the request in its PostgreSQL database.
- The Help Desk worker completes the initial notification job, proving the normal path works.
- The Runner starts the connection exhaustion action under a fixed lease.
- After the fault is active, the Runner creates a bounded batch of notification jobs through the demo target's reserved management channel.
- The API becomes slow or returns an error, and the worker cannot drain the notification backlog.
- OpsRehearse records action results, health checks, log excerpts, operator notes, and timestamps as evidence.
- The operator follows the versioned Runbook and initiates the declared recovery action.
- OpsRehearse runs mandatory verification checks and records every result.
- OpsRehearse resets the demo target to its baseline fixture and confirms that reset succeeded.
- OpsRehearse generates a draft after-action report for both successful and unsuccessful exercises.
- The operator records improvement items for the Runbook, target, monitoring, or exercise definition and finalizes the report.
OpsRehearse owns:
- scenario definitions for managed exercise targets;
- safe dispatch of registered Runner actions;
- exercise state and timeline;
- bounded evidence collection;
- Runbook content used during an exercise;
- automatic recovery verification;
- reset status;
- after-action reports and improvement items.
The built-in Help Desk target owns:
- support requests;
- notification jobs;
- its application logs and health endpoints;
- its dedicated PostgreSQL database;
- its deterministic reset interface.
The Runner owns:
- validation of action envelopes;
- enforcement of action-specific limits;
- execution leases and idempotency;
- registered fault, recovery, probe, evidence, and reset handlers;
- reporting action results to the control plane.
Version 0.1 does not include:
- Kubernetes;
- Kafka;
- microservices;
- cloud adapters;
- a plugin SDK;
- an AI Copilot;
- arbitrary shell execution;
- production write operations;
- complete ITSM functionality;
- SSO;
- multi-tenancy;
- enterprise-ready claims.
It also does not replace ServiceNow, Jira Service Management, monitoring platforms, incident response platforms, or general chaos engineering systems.
The v0.1 product succeeds when a new local installation can complete the approved vertical slice without manual database edits, shell-based fault injection, or undocumented recovery steps.
The following outcomes are required:
- The normal Help Desk request and notification path works before injection.
- The scenario produces observable API degradation and a worker backlog within its activation window.
- The Runner releases all held fault resources when recovery starts or the lease expires.
- Mandatory verification checks distinguish recovered from unrecovered state.
- Reset restores the same baseline state and blocks new runs if it cannot do so.
- The same scenario completes three consecutive end-to-end runs from a clean installation.
- Every final result is supported by stored evidence.
- A failure is useful only when its effect is observable and reversible.
- Recovery is a measured system state, not a button click or operator opinion.
- The Runner enforces safety even when the control plane sends an invalid request.
- Evidence is append-only and bounded. Corrections create new records.
- A failed or incomplete exercise still produces a report.
- v0.1 optimizes for one reproducible workflow rather than feature breadth.
The first release may be described as a local, self-hosted technical preview for the built-in demo target. Documentation must not call it production-ready, enterprise-ready, secure for untrusted networks, or compatible with arbitrary company environments.