feat(skills): add ARISE trace evolution, STEM pattern-to-skill, and ERL heuristics (#2398, #2462, #2463) (#2482)

Implements three complementary skill evolution mechanisms in zeph-skills:

- ARISE: summarizes successful multi-tool solution traces into SKILL.md
  description updates via background LLM call; uses historical success/failure
  rates to bias embedding candidate re-ranking (policy-driven selection)
- STEM: tracks recurring tool-use patterns in skill_usage_log (migration 057);
  auto-generates quarantined SKILL.md candidates when a pattern repeats >= N
  times with sufficient success rate
- ERL: post-task reflection extracts transferable heuristics into
  skill_heuristics table (migration 058); injects top heuristics into skill
  matching context as a ## Learned Heuristics section

All three features are disabled by default, fire-and-forget via tokio::spawn,
expose *_provider config fields for multi-model routing, and emit TUI status
spinners. LLM-generated skill content is validated and starts at quarantined
trust level. Config migration and --init wizard updated for the 14 new fields.

Author: bug-ops

CHANGELOG.md

@@ -10,6 +10,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 - feat(tools): `[tools.shell] max_snapshot_bytes` config option to limit transaction snapshot size — returns `SnapshotFailed` when cumulative copied bytes exceed the limit; `0` means unlimited (default)
 - feat(tools): transactional `ShellExecutor` — opt-in snapshot+rollback for shell commands; file-level snapshot is captured before write commands (detected via `WRITE_INDICATORS` heuristic + redirection target extraction); rollback restores originals on configurable exit codes; new `ShellConfig` fields: `transactional`, `transaction_scope` (glob-filtered paths), `auto_rollback`, `auto_rollback_exit_codes`, `snapshot_required`; new `ToolError::SnapshotFailed`, `AuditResult::Rollback`, `ToolEvent::Rollback` variants; backed by `tempfile::TempDir` for automatic cleanup on success (closes #2414)
+- feat(skills): ARISE trace-based skill improvement — after a successful multi-tool turn, `spawn_arise_trace_improvement()` fires a background LLM call to summarize the tool sequence into an improved SKILL.md body; new version is saved with `source = 'arise_trace'` and starts at `quarantined` trust level (never inherits parent trust); controlled by `[skills.learning] arise_enabled = false`, `arise_min_tool_calls = 2`, `arise_trace_provider = ""` (closes #2398)
+- feat(skills): STEM pattern-to-skill conversion — `spawn_stem_detection()` logs every tool sequence to `skill_usage_log` after each turn; `find_recurring_patterns()` detects sequences meeting `stem_min_occurrences` and `stem_min_success_rate`; qualifying patterns trigger a background LLM call to generate a SKILL.md candidate written to the managed skills directory at `quarantined` trust level; retention pruning via `stem_retention_days`; controlled by `[skills.learning] stem_enabled = false` (closes #2462)
+- feat(skills): ERL experiential reflective learning — `spawn_erl_reflection()` fires a background LLM call after each successful skill+tool turn to extract transferable heuristics; heuristics are stored in `skill_heuristics` table with Jaccard deduplication; at skill matching time `build_erl_heuristics_prompt()` prepends a `## Learned Heuristics` section to the skill context; controlled by `[skills.learning] erl_enabled = false`, `erl_max_heuristics_per_skill = 3`, `erl_min_confidence = 0.5` (closes #2463)
+- feat(db): migrations 057 (`skill_usage_log`) and 058 (`skill_heuristics`) for STEM and ERL storage; both SQLite and Postgres variants
+- feat(config): `LearningConfig` extended with 14 new fields for ARISE/STEM/ERL (all disabled by default); new fields registered in `config/default.toml` as commented-out entries
 - feat(core): `/new` slash command — resets conversation context (messages, compaction state, tool caches, focus/sidequest, pending plans) while preserving memory, MCP connections, providers, and skills; creates a new `ConversationId` in SQLite for audit trail; generates a session digest for the outgoing conversation fire-and-forget unless `--no-digest` is passed; active sub-agents and background compression tasks are cancelled; `--keep-plan` preserves a pending plan graph; available in all channels (CLI, TUI, Telegram) via the unified `handle_builtin_command` path (closes #2451)
 - feat(memory): Kumiho AGM-inspired belief revision for graph edges — new `BeliefRevisionConfig` with `similarity_threshold`; `find_superseded_edges()` uses contradiction heuristic (same relation domain + high cosine similarity = supersession); `superseded_by` column added to `graph_edges` for audit trail; `invalidate_edge_with_supersession()` in `GraphStore`; `resolve_edge_typed` accepts optional `BeliefRevisionConfig`; controlled by `[memory.graph.belief_revision] enabled = false` (migration 056, closes #2441)
 - feat(memory): D-MEM RPE-based tiered graph extraction routing — `RpeRouter` computes heuristic surprise score from context similarity and entity novelty; low-RPE turns skip the MAGMA LLM extraction pipeline; `consecutive_skips` safety valve forces extraction after `max_skip_turns` consecutive skips; `extract_candidate_entities()` helper for cheap regex+keyword entity detection; controlled by `[memory.graph.rpe] enabled = false, threshold = 0.3, max_skip_turns = 5` (closes #2442)

config/default.toml

@@ -188,6 +188,23 @@ cooldown_minutes = 60
 # max_auto_sections = 3
 # Domain evaluation gate before promoting auto-generated skills
 # domain_success_gate = false
+# ARISE: trace-based skill improvement from successful multi-tool turns
+# arise_enabled = false
+# arise_min_tool_calls = 2
+# arise_trace_provider = ""
+# STEM: automatic tool pattern detection and skill candidate generation
+# stem_enabled = false
+# stem_min_occurrences = 3
+# stem_min_success_rate = 0.8
+# stem_provider = ""
+# stem_retention_days = 90
+# stem_pattern_window_days = 30
+# ERL: post-task heuristic extraction and injection at skill match time
+# erl_enabled = false
+# erl_extract_provider = ""
+# erl_max_heuristics_per_skill = 3
+# erl_dedup_threshold = 0.9
+# erl_min_confidence = 0.5
 
 [skills.trust]
 # Default trust level for newly discovered skills: trusted, verified, quarantined, blocked

crates/zeph-config/src/learning.rs

@@ -79,6 +79,38 @@ fn default_max_auto_sections() -> u32 {
     3
 }
 
+fn default_arise_min_tool_calls() -> u32 {
+    2
+}
+
+fn default_stem_min_occurrences() -> u32 {
+    3
+}
+
+fn default_stem_min_success_rate() -> f64 {
+    0.8
+}
+
+fn default_stem_retention_days() -> u32 {
+    90
+}
+
+fn default_stem_pattern_window_days() -> u32 {
+    30
+}
+
+fn default_erl_max_heuristics_per_skill() -> u32 {
+    3
+}
+
+fn default_erl_dedup_threshold() -> f32 {
+    0.9
+}
+
+fn default_erl_min_confidence() -> f64 {
+    0.5
+}
+
 /// Strategy for detecting implicit user corrections.
 #[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Deserialize, Serialize)]
 #[serde(rename_all = "lowercase")]
@@ -184,6 +216,58 @@ pub struct LearningConfig {
     /// activation is skipped (the version is still saved for manual review).
     #[serde(default)]
     pub domain_success_gate: bool,
+
+    // --- ARISE: trace-based skill improvement ---
+    /// Enable ARISE trace-based skill improvement (disabled by default).
+    #[serde(default)]
+    pub arise_enabled: bool,
+    /// Minimum tool calls in a turn to trigger ARISE trace improvement.
+    #[serde(default = "default_arise_min_tool_calls")]
+    pub arise_min_tool_calls: u32,
+    /// Provider name from `[[llm.providers]]` for ARISE trace summarization.
+    /// Empty = fall back to primary provider.
+    #[serde(default)]
+    pub arise_trace_provider: String,
+
+    // --- STEM: pattern-to-skill conversion ---
+    /// Enable STEM automatic tool pattern detection and skill generation (disabled by default).
+    #[serde(default)]
+    pub stem_enabled: bool,
+    /// Minimum occurrences of a tool sequence before generating a skill candidate.
+    #[serde(default = "default_stem_min_occurrences")]
+    pub stem_min_occurrences: u32,
+    /// Minimum success rate of the pattern before generating a skill candidate.
+    #[serde(default = "default_stem_min_success_rate")]
+    pub stem_min_success_rate: f64,
+    /// Provider name from `[[llm.providers]]` for STEM skill generation.
+    /// Empty = fall back to primary provider.
+    #[serde(default)]
+    pub stem_provider: String,
+    /// Days to retain rows in `skill_usage_log` before pruning.
+    #[serde(default = "default_stem_retention_days")]
+    pub stem_retention_days: u32,
+    /// Window in days for pattern detection queries (limits scan cost on large tables).
+    #[serde(default = "default_stem_pattern_window_days")]
+    pub stem_pattern_window_days: u32,
+
+    // --- ERL: experiential reflective learning ---
+    /// Enable ERL post-task heuristic extraction (disabled by default).
+    #[serde(default)]
+    pub erl_enabled: bool,
+    /// Provider name from `[[llm.providers]]` for ERL heuristic extraction.
+    /// Empty = fall back to primary provider.
+    #[serde(default)]
+    pub erl_extract_provider: String,
+    /// Maximum heuristics prepended per skill at match time.
+    #[serde(default = "default_erl_max_heuristics_per_skill")]
+    pub erl_max_heuristics_per_skill: u32,
+    /// Text similarity threshold (Jaccard) for heuristic deduplication.
+    /// When exact text match exceeds this, increment `use_count` instead of inserting.
+    #[serde(default = "default_erl_dedup_threshold")]
+    pub erl_dedup_threshold: f32,
+    /// Minimum confidence to include a heuristic at match time.
+    #[serde(default = "default_erl_min_confidence")]
+    pub erl_min_confidence: f64,
 }
 
 impl Default for LearningConfig {
@@ -215,6 +299,20 @@ impl Default for LearningConfig {
             min_sessions_before_demote: default_min_sessions_before_demote(),
             max_auto_sections: default_max_auto_sections(),
             domain_success_gate: false,
+            arise_enabled: false,
+            arise_min_tool_calls: default_arise_min_tool_calls(),
+            arise_trace_provider: String::new(),
+            stem_enabled: false,
+            stem_min_occurrences: default_stem_min_occurrences(),
+            stem_min_success_rate: default_stem_min_success_rate(),
+            stem_provider: String::new(),
+            stem_retention_days: default_stem_retention_days(),
+            stem_pattern_window_days: default_stem_pattern_window_days(),
+            erl_enabled: false,
+            erl_extract_provider: String::new(),
+            erl_max_heuristics_per_skill: default_erl_max_heuristics_per_skill(),
+            erl_dedup_threshold: default_erl_dedup_threshold(),
+            erl_min_confidence: default_erl_min_confidence(),
         }
     }
 }
@@ -298,6 +396,68 @@ feedback_provider = "fast""#;
         assert_eq!(cfg.min_sessions_before_demote, 1);
     }
 
+    #[test]
+    fn arise_stem_erl_defaults() {
+        let cfg = LearningConfig::default();
+        assert!(!cfg.arise_enabled);
+        assert_eq!(cfg.arise_min_tool_calls, 2);
+        assert!(cfg.arise_trace_provider.is_empty());
+        assert!(!cfg.stem_enabled);
+        assert_eq!(cfg.stem_min_occurrences, 3);
+        assert!((cfg.stem_min_success_rate - 0.8).abs() < f64::EPSILON);
+        assert!(cfg.stem_provider.is_empty());
+        assert_eq!(cfg.stem_retention_days, 90);
+        assert_eq!(cfg.stem_pattern_window_days, 30);
+        assert!(!cfg.erl_enabled);
+        assert!(cfg.erl_extract_provider.is_empty());
+        assert_eq!(cfg.erl_max_heuristics_per_skill, 3);
+        assert!((cfg.erl_dedup_threshold - 0.9).abs() < f32::EPSILON);
+        assert!((cfg.erl_min_confidence - 0.5).abs() < f64::EPSILON);
+    }
+
+    #[test]
+    fn arise_stem_erl_serde_roundtrip() {
+        let toml = r#"
+arise_enabled = true
+arise_min_tool_calls = 3
+arise_trace_provider = "fast"
+stem_enabled = true
+stem_min_occurrences = 5
+stem_min_success_rate = 0.9
+stem_provider = "mid"
+stem_retention_days = 60
+stem_pattern_window_days = 14
+erl_enabled = true
+erl_extract_provider = "fast"
+erl_max_heuristics_per_skill = 5
+erl_dedup_threshold = 0.85
+erl_min_confidence = 0.6
+"#;
+        let cfg: LearningConfig = toml::from_str(toml).unwrap();
+        assert!(cfg.arise_enabled);
+        assert_eq!(cfg.arise_min_tool_calls, 3);
+        assert_eq!(cfg.arise_trace_provider, "fast");
+        assert!(cfg.stem_enabled);
+        assert_eq!(cfg.stem_min_occurrences, 5);
+        assert!((cfg.stem_min_success_rate - 0.9).abs() < f64::EPSILON);
+        assert_eq!(cfg.stem_provider, "mid");
+        assert_eq!(cfg.stem_retention_days, 60);
+        assert_eq!(cfg.stem_pattern_window_days, 14);
+        assert!(cfg.erl_enabled);
+        assert_eq!(cfg.erl_extract_provider, "fast");
+        assert_eq!(cfg.erl_max_heuristics_per_skill, 5);
+        assert!((cfg.erl_dedup_threshold - 0.85_f32).abs() < f32::EPSILON);
+        assert!((cfg.erl_min_confidence - 0.6).abs() < f64::EPSILON);
+    }
+
+    #[test]
+    fn arise_stem_erl_empty_section_uses_defaults() {
+        let cfg: LearningConfig = toml::from_str("").unwrap();
+        assert!(!cfg.arise_enabled);
+        assert!(!cfg.stem_enabled);
+        assert!(!cfg.erl_enabled);
+    }
+
     #[test]
     fn learning_config_new_fields_serde_roundtrip() {
         let toml = r"

crates/zeph-core/src/agent/context/assembly.rs

@@ -1484,11 +1484,16 @@ impl<C: Channel> Agent<C> {
             other => other,
         };
 
-        let skills_prompt = if effective_mode == crate::config::SkillPromptMode::Compact {
+        let mut skills_prompt = if effective_mode == crate::config::SkillPromptMode::Compact {
             format_skills_prompt_compact(&active_skills)
         } else {
             format_skills_prompt(&active_skills, &trust_map, &health_map)
         };
+        // ERL: append learned heuristics for active skills (no-op when erl_enabled = false).
+        let erl_suffix = self.build_erl_heuristics_prompt().await;
+        if !erl_suffix.is_empty() {
+            skills_prompt.push_str(&erl_suffix);
+        }
         let catalog_prompt = format_skills_catalog(&remaining_skills);
         self.skill_state
             .last_skills_prompt