Skip to content

Commit 3dbac8b

Browse files
committed
feat: crash recovery and warmup enforcement (v1.8.0)
- Add recovery.rs with RecoveryState (Normal/Shock/Coma) and RecoveryAnalyzer - Detect unclean shutdown (shock) and prolonged inactivity (coma) on startup - Write graceful shutdown marker before exit - Add WarmupTracker to CommandHandler with Ping/Stats exemption - Return WarmingUp error during resurrection for non-exempt ops - Add shutdown marker read/write to AkashicStore - Add stability benchmark (10 benchmarks, all sub-microsecond) - 89 tests passing
1 parent 015c4cc commit 3dbac8b

File tree

9 files changed

+428
-16
lines changed

9 files changed

+428
-16
lines changed

CONTRIBUTING.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -84,8 +84,8 @@ Look for issues labeled `good first issue` - these are suitable for newcomers.
8484

8585
## Questions?
8686

87-
- Open a [Discussion](https://github.com/laphilosophia/mindfry/discussions) for questions
88-
- Check existing [Issues](https://github.com/laphilosophia/mindfry/issues) before opening a new one
87+
- Open a [Discussion](https://github.com/cluster-127/mindfry/discussions) for questions
88+
- Check existing [Issues](https://github.com/cluster-127/mindfry/issues) before opening a new one
8989

9090
## License
9191

Cargo.toml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ edition = "2021"
55
authors = ["Erdem Arslan <erdemarslan@ymail.com>"]
66
description = "Memory with a Conscience"
77
license = "Apache-2.0"
8-
repository = "https://github.com/laphilosophia/mindfry"
8+
repository = "https://github.com/cluster-127/mindfry"
99
keywords = ["database", "graph", "cognitive", "ephemeral", "decay"]
1010
categories = ["database", "data-structures"]
1111

@@ -73,6 +73,10 @@ harness = false
7373
name = "graph"
7474
harness = false
7575

76+
[[bench]]
77+
name = "stability"
78+
harness = false
79+
7680
# ═══════════════════════════════════════════════════════════════
7781
# BINARIES
7882
# ═══════════════════════════════════════════════════════════════

README.md

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,9 @@ MindFry is **subjective** — it processes data through a simulated cognitive la
2525
- **Strengthen** frequently accessed data (Hebbian learning)
2626
- **Propagate** stimulation through neural bonds (synaptic chains)
2727

28-
## 🧬 Core Principles
28+
## Core Principles
2929

30-
### 🧠 Tri-Cortex Architecture
30+
### Tri-Cortex Architecture
3131

3232
Decisions are made using **Balanced Ternary Logic** (Setun):
3333

@@ -37,7 +37,7 @@ Decisions are made using **Balanced Ternary Logic** (Setun):
3737

3838
The database has a **Personality Octet** (8 dimensions) and a **Mood** that affects data prioritization.
3939

40-
### ❤️ Mood & Personality
40+
### Mood & Personality
4141

4242
> **Note:** Mood affects which data surfaces first, not whether your requests succeed. All data remains accessible — mood just influences the "attention" priority.
4343
@@ -48,7 +48,7 @@ Mood modulates the consciousness threshold:
4848

4949
**Override anytime:** Use `BYPASS_FILTERS` flag for guaranteed access regardless of mood.
5050

51-
### 🕸️ Synaptic Propagation
51+
### Synaptic Propagation
5252

5353
When you `stimulate("A")`:
5454

@@ -58,7 +58,7 @@ A (+1.0) → B (+0.5) → C (+0.25) → ... (damped)
5858

5959
Touch one memory, its neighbors tremble.
6060

61-
### 💾 Resurrection
61+
### Resurrection
6262

6363
Shutdown and restart. The database remembers:
6464

@@ -68,17 +68,17 @@ Shutdown and restart. The database remembers:
6868

6969
## Quick Start
7070

71-
### 🐳 Docker (Recommended)
71+
### Docker (Recommended)
7272

7373
```bash
74-
docker run -d -p 9527:9527 ghcr.io/laphilosophia/mindfry:latest
74+
docker run -d -p 9527:9527 ghcr.io/cluster-127/mindfry:latest
7575
```
7676

7777
### From Source
7878

7979
```bash
8080
# Clone
81-
git clone https://github.com/laphilosophia/mindfry.git
81+
git clone https://github.com/cluster-127/mindfry.git
8282
cd mindfry
8383

8484
# Run server
@@ -125,7 +125,7 @@ console.log(associated.energy) // Increased by propagation
125125

126126
## License
127127

128-
[Apache-2.0](LICENSE) © [Erdem Arslan](https://github.com/laphilosophia)
128+
[Apache-2.0](LICENSE) © [Erdem Arslan](https://github.com/cluster-127)
129129

130130
---
131131

benches/stability.rs

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
//! Stability Layer Benchmarks
2+
//!
3+
//! Benchmarks for crash recovery, warmup tracking, and exhaustion monitoring.
4+
5+
use criterion::{black_box, criterion_group, criterion_main, Criterion};
6+
use mindfry::stability::{
7+
ExhaustionLevel, ExhaustionMonitor, RecoveryAnalyzer, RecoveryState, ShutdownMarker,
8+
WarmupTracker,
9+
};
10+
11+
// ═══════════════════════════════════════════════════════════════
12+
// RECOVERY BENCHMARKS
13+
// ═══════════════════════════════════════════════════════════════
14+
15+
fn bench_recovery_analyzer_new(c: &mut Criterion) {
16+
let marker = Some(ShutdownMarker::graceful());
17+
18+
c.bench_function("recovery_analyzer_new", |b| {
19+
b.iter(|| black_box(RecoveryAnalyzer::new(marker.clone())))
20+
});
21+
}
22+
23+
fn bench_recovery_analyzer_analyze(c: &mut Criterion) {
24+
let marker = Some(ShutdownMarker::graceful());
25+
let analyzer = RecoveryAnalyzer::new(marker);
26+
27+
c.bench_function("recovery_analyzer_analyze", |b| {
28+
b.iter(|| black_box(analyzer.analyze()))
29+
});
30+
}
31+
32+
fn bench_recovery_state_intensity(c: &mut Criterion) {
33+
c.bench_function("recovery_state_intensity", |b| {
34+
b.iter(|| {
35+
black_box(RecoveryState::Normal.intensity());
36+
black_box(RecoveryState::Shock.intensity());
37+
black_box(RecoveryState::Coma.intensity());
38+
})
39+
});
40+
}
41+
42+
// ═══════════════════════════════════════════════════════════════
43+
// WARMUP BENCHMARKS
44+
// ═══════════════════════════════════════════════════════════════
45+
46+
fn bench_warmup_tracker_new(c: &mut Criterion) {
47+
c.bench_function("warmup_tracker_new", |b| {
48+
b.iter(|| black_box(WarmupTracker::new()))
49+
});
50+
}
51+
52+
fn bench_warmup_tracker_is_ready(c: &mut Criterion) {
53+
let tracker = WarmupTracker::new();
54+
55+
c.bench_function("warmup_tracker_is_ready", |b| {
56+
b.iter(|| black_box(tracker.is_ready()))
57+
});
58+
}
59+
60+
fn bench_warmup_tracker_state(c: &mut Criterion) {
61+
let tracker = WarmupTracker::new();
62+
63+
c.bench_function("warmup_tracker_state", |b| {
64+
b.iter(|| black_box(tracker.state()))
65+
});
66+
}
67+
68+
fn bench_warmup_tracker_clone(c: &mut Criterion) {
69+
let tracker = WarmupTracker::new();
70+
71+
c.bench_function("warmup_tracker_clone", |b| {
72+
b.iter(|| black_box(tracker.clone()))
73+
});
74+
}
75+
76+
// ═══════════════════════════════════════════════════════════════
77+
// EXHAUSTION BENCHMARKS
78+
// ═══════════════════════════════════════════════════════════════
79+
80+
fn bench_exhaustion_level_from_energy(c: &mut Criterion) {
81+
c.bench_function("exhaustion_level_from_energy", |b| {
82+
b.iter(|| {
83+
black_box(ExhaustionLevel::from_energy(0.9));
84+
black_box(ExhaustionLevel::from_energy(0.5));
85+
black_box(ExhaustionLevel::from_energy(0.1));
86+
})
87+
});
88+
}
89+
90+
fn bench_exhaustion_allows_writes(c: &mut Criterion) {
91+
c.bench_function("exhaustion_allows_writes", |b| {
92+
b.iter(|| {
93+
black_box(ExhaustionLevel::Normal.allows_writes());
94+
black_box(ExhaustionLevel::Elevated.allows_writes());
95+
black_box(ExhaustionLevel::Exhausted.allows_writes());
96+
black_box(ExhaustionLevel::Emergency.allows_writes());
97+
})
98+
});
99+
}
100+
101+
fn bench_exhaustion_monitor_default(c: &mut Criterion) {
102+
c.bench_function("exhaustion_monitor_new", |b| {
103+
b.iter(|| black_box(ExhaustionMonitor::default()))
104+
});
105+
}
106+
107+
criterion_group!(
108+
benches,
109+
// Recovery
110+
bench_recovery_analyzer_new,
111+
bench_recovery_analyzer_analyze,
112+
bench_recovery_state_intensity,
113+
// Warmup
114+
bench_warmup_tracker_new,
115+
bench_warmup_tracker_is_ready,
116+
bench_warmup_tracker_state,
117+
bench_warmup_tracker_clone,
118+
// Exhaustion
119+
bench_exhaustion_level_from_energy,
120+
bench_exhaustion_allows_writes,
121+
bench_exhaustion_monitor_default,
122+
);
123+
124+
criterion_main!(benches);

src/bin/server.rs

Lines changed: 35 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,27 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
8484
}
8585
};
8686

87+
// Step 1.5: Crash Recovery Detection
88+
print!(" │ 🔍 Checking recovery state...");
89+
std::io::Write::flush(&mut std::io::stdout())?;
90+
let last_marker = store.read_shutdown_marker().ok().flatten();
91+
let recovery_analyzer = mindfry::stability::RecoveryAnalyzer::new(last_marker);
92+
let recovery_state = recovery_analyzer.analyze();
93+
match recovery_state {
94+
mindfry::stability::RecoveryState::Normal => println!(" ✓ (clean)"),
95+
mindfry::stability::RecoveryState::Shock => {
96+
println!(" ⚠ (SHOCK detected)");
97+
warn!("🩹 Unclean shutdown detected - building resistance");
98+
}
99+
mindfry::stability::RecoveryState::Coma => {
100+
println!(" ⚠ (COMA detected)");
101+
warn!(
102+
"😴 Prolonged downtime: {}s",
103+
recovery_analyzer.downtime_secs()
104+
);
105+
}
106+
}
107+
87108
// Step 2: Initialize Psyche Arena (empty)
88109
print!(" │ 🧠 Initializing Psyche Arena...");
89110
std::io::Write::flush(&mut std::io::stdout())?;
@@ -175,7 +196,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
175196
// ═══════════════════════════════════════════════════════════════
176197

177198
let shutdown_result = tokio::select! {
178-
result = accept_loop(listener, Arc::clone(&db)) => {
199+
result = accept_loop(listener, Arc::clone(&db), warmup.clone()) => {
179200
// Accept loop returned (error or explicit stop)
180201
result
181202
}
@@ -208,6 +229,13 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
208229
Ok(meta) => info!("💾 Pre-shutdown snapshot saved: {}", meta.id),
209230
Err(e) => warn!("⚠️ Failed to save shutdown snapshot: {}", e),
210231
}
232+
233+
// Write graceful shutdown marker for crash recovery
234+
let marker = mindfry::stability::ShutdownMarker::graceful();
235+
match store.write_shutdown_marker(&marker) {
236+
Ok(_) => info!("✅ Graceful shutdown marker written"),
237+
Err(e) => warn!("⚠️ Failed to write shutdown marker: {}", e),
238+
}
211239
}
212240
}
213241

@@ -225,18 +253,20 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
225253
async fn accept_loop(
226254
listener: TcpListener,
227255
db: Arc<RwLock<MindFry>>,
256+
warmup: mindfry::stability::WarmupTracker,
228257
) -> Result<mindfry::stability::ShutdownReason, Box<dyn std::error::Error + Send + Sync>> {
229258
loop {
230259
match listener.accept().await {
231260
Ok((socket, peer)) => {
232261
info!("📥 New connection from {}", peer);
233262

234-
// Clone Arc for the handler
263+
// Clone for the handler
235264
let db_clone = Arc::clone(&db);
265+
let warmup_clone = warmup.clone();
236266

237267
// Spawn connection handler
238268
tokio::spawn(async move {
239-
if let Err(e) = handle_connection(socket, db_clone).await {
269+
if let Err(e) = handle_connection(socket, db_clone, warmup_clone).await {
240270
error!("Connection error: {}", e);
241271
}
242272
info!("📤 Connection closed: {}", peer);
@@ -254,8 +284,9 @@ async fn accept_loop(
254284
async fn handle_connection(
255285
mut socket: TcpStream,
256286
db: Arc<RwLock<MindFry>>,
287+
warmup: mindfry::stability::WarmupTracker,
257288
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
258-
let mut handler = CommandHandler::new(db);
289+
let mut handler = CommandHandler::with_warmup(db, warmup);
259290
let mut buffer = vec![0u8; 4096];
260291
let mut read_buf = Vec::new();
261292

src/persistence/akashic.rs

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -293,6 +293,34 @@ impl AkashicStore {
293293
&self.indexer
294294
}
295295

296+
// ═══════════════════════════════════════════════════════════════
297+
// SHUTDOWN MARKER (for crash recovery)
298+
// ═══════════════════════════════════════════════════════════════
299+
300+
/// Write a graceful shutdown marker
301+
pub fn write_shutdown_marker(&self, marker: &crate::stability::ShutdownMarker) -> Result<()> {
302+
let meta_tree = self.db.open_tree("meta")?;
303+
let data = bincode::serialize(marker)?;
304+
meta_tree.insert("shutdown_marker", data)?;
305+
self.db.flush()?;
306+
Ok(())
307+
}
308+
309+
/// Read and clear the shutdown marker
310+
/// Returns None if no marker exists (first run or clean startup)
311+
pub fn read_shutdown_marker(&self) -> Result<Option<crate::stability::ShutdownMarker>> {
312+
let meta_tree = self.db.open_tree("meta")?;
313+
match meta_tree.get("shutdown_marker")? {
314+
Some(data) => {
315+
let marker: crate::stability::ShutdownMarker = bincode::deserialize(&data)?;
316+
// Clear the marker so next unclean shutdown is detected
317+
meta_tree.remove("shutdown_marker")?;
318+
Ok(Some(marker))
319+
}
320+
None => Ok(None),
321+
}
322+
}
323+
296324
// ═══════════════════════════════════════════════════════════════
297325
// SERIALIZATION HELPERS
298326
// ═══════════════════════════════════════════════════════════════

src/protocol/handler.rs

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ use std::time::Instant;
99

1010
use crate::arena::Lineage;
1111
use crate::graph::Bond;
12+
use crate::stability::WarmupTracker;
1213
use crate::MindFry;
1314

1415
use super::message::*;
@@ -25,6 +26,8 @@ pub struct CommandHandler {
2526
/// Exhaustion monitor for backpressure
2627
#[allow(dead_code)] // Reserved for operation cost tracking
2728
exhaustion: crate::stability::ExhaustionMonitor,
29+
/// Warmup tracker for progressive availability
30+
warmup: WarmupTracker,
2831
}
2932

3033
impl CommandHandler {
@@ -35,11 +38,36 @@ impl CommandHandler {
3538
start_time: Instant::now(),
3639
is_frozen: false,
3740
exhaustion: crate::stability::ExhaustionMonitor::default(),
41+
warmup: WarmupTracker::new(),
42+
}
43+
}
44+
45+
/// Create a new command handler with warmup tracker
46+
pub fn with_warmup(db: Arc<RwLock<MindFry>>, warmup: WarmupTracker) -> Self {
47+
Self {
48+
db,
49+
start_time: Instant::now(),
50+
is_frozen: false,
51+
exhaustion: crate::stability::ExhaustionMonitor::default(),
52+
warmup,
3853
}
3954
}
4055

4156
/// Handle a request and return a response
4257
pub fn handle(&mut self, request: Request) -> Response {
58+
// ═══════════════════════════════════════════════════════════════
59+
// WARMUP CHECK (Progressive Availability)
60+
// ═══════════════════════════════════════════════════════════════
61+
// Allow Ping and Stats during warmup (always accessible)
62+
let is_warmup_exempt = matches!(request, Request::Ping | Request::Stats);
63+
64+
if !is_warmup_exempt && !self.warmup.is_ready() {
65+
return Response::Error {
66+
code: ErrorCode::WarmingUp,
67+
message: "Server warming up - cognitively unavailable".into(),
68+
};
69+
}
70+
4371
// ═══════════════════════════════════════════════════════════════
4472
// EXHAUSTION CHECK (Backpressure)
4573
// ═══════════════════════════════════════════════════════════════

0 commit comments

Comments
 (0)