|
| 1 | +# Stage Agents & Planner Overview |
| 2 | + |
| 3 | +This page summarizes: |
| 4 | + |
| 5 | +1. How stage agents map to the existing Zyra stages |
| 6 | +2. The new `zyra plan` capabilities (reasoning trace, suggestions, clarifications) |
| 7 | +3. Value Engine suggestions and how they influence the manifest |
| 8 | +4. Running the generated plan with `zyra swarm` |
| 9 | + |
| 10 | +## Stage Agents at a Glance |
| 11 | + |
| 12 | +Each Zyra workflow stage now corresponds to a **stage agent** with a normalized spec: |
| 13 | + |
| 14 | +| Stage | Purpose | Implementation Notes | |
| 15 | +|------------|-------------------------------------------------|----------------------------------------------------| |
| 16 | +| acquire | Fetch/source data (HTTP, FTP, S3, etc.) | CLI agent (`zyra acquire …`) | |
| 17 | +| process | Transform/clean data (scan-frames, pad-missing) | CLI agent | |
| 18 | +| simulate | Skeleton mock (emits placeholder ensemble) | Mock agent (ready for future simulation logic) | |
| 19 | +| decide | Skeleton mock (scores/optimizes results) | Mock agent (ensures DAG coverage) | |
| 20 | +| visualize | Render plots/animations | CLI agent (e.g., `compose-video`) | |
| 21 | +| narrate | Narration/summary | Proposal → CLI (LLM-backed) | |
| 22 | +| verify | Quality/completeness checks | CLI agent (`verify evaluate`) | |
| 23 | +| export | Save or disseminate artifacts | CLI agent (`decimate local`, `export s3`, etc.) | |
| 24 | + |
| 25 | +When you run `zyra swarm`, the orchestrator resolves each spec into an agent, honors `depends_on`, and executes the DAG with optional parallelism. |
| 26 | + |
| 27 | +## Hybrid Design Workflow: Structured Tool Confirmation (The General Contractor Model) |
| 28 | + |
| 29 | +Zyra includes the integration of the **ValueEngine** for proactive value suggestion, consistent with the architectural principles established in the sources. |
| 30 | + |
| 31 | +**Analogy:** If the current implementation is like a factory floor manager (Orchestrator) specifying exactly which bolt (Tool) the worker (Stage Agent) must use, the hybrid design is like the manager telling the worker "Use the best high-strength fastener for this application" (Natural Language), and the worker replies, "I propose using a grade 8 titanium bolt with these specifications" (Structured Proposal). The manager then verifies that titanium is on the approved materials list (Validation) and logs the decision (Provenance) before allowing the work to proceed (Execution). This process is safer, more adaptable, and fully traceable. |
| 32 | + |
| 33 | +The workflow is divided into three distinct phases, managing the transition from high-level user intent to validated, executable code. |
| 34 | + |
| 35 | +### Phase 1: Planning, Value Augmentation, and Initial Validation (Central Control) |
| 36 | + |
| 37 | +This phase establishes the complete, safe plan (DAG) and is primarily managed by the Orchestrator's LLM Reasoning, Planner, Validator, and ValueEngine components. |
| 38 | + |
| 39 | +| Step | Component | Action / Description | Source Alignment | |
| 40 | +| :--- | :--- | :--- | :--- | |
| 41 | +| **1. User Intent and Interpretation** | LLM System Prompt / Reasoning Layer | Interprets the **natural-language user intent**, identifying verbs and objects, and maps them to high-level Zyra stages. Output is restricted to **Structured reasoning via JSON output**. | | |
| 42 | +| **2. Base DAG Construction** | Planner (Python) | Verifies requested actions exist in the **capabilities registry**. Builds the initial Dependency Graph (DAG) based on canonical stages (e.g., `import → process → visualize → export`). | | |
| 43 | +| **3. Proactive Value Suggestion** | **ValueEngine (LLM)** | Evaluates the initial DAG and provenance for potential low-cost, high-value augmentations (e.g., suggesting a `verify` or `narrate` stage). Returns structured suggestions (e.g., JSON containing stage, description, confidence). | | |
| 44 | +| **4. Augmentation Validation & Acceptance** | Validator / User Interaction | Orchestrator presents suggestions to the user for **acceptance or dismissal**. The Validator ensures accepted additions are **safe and compatible**; accepted suggestions are recorded (stage, confidence, rationale) in Provenance and inserted via the template registry (`insert_node`). | | |
| 45 | +| **5. Deterministic Stage Check** | Planner (Python) | Determines if the first executable stage is **deterministic** (e.g., `acquire ftp`, `process scan-frames`). Deterministic stages skip Phase 2 and proceed directly to execution (Step 9). **Ambiguous** stages (e.g., `visualize compose`, `narrate swarm`) enter Phase 2 for structured tool proposals. | | |
| 46 | + |
| 47 | +--- |
| 48 | + |
| 49 | +### Phase 2: Specialized Reasoning and Proposal (Hybrid Loop – For Ambiguous Tasks Only) |
| 50 | + |
| 51 | +This phase delegates specialized tool selection to the Stage Agent but retains central control via a structured contract and mandatory validation. |
| 52 | + |
| 53 | +| Step | Component | Action / Description | Source Alignment | |
| 54 | +| :--- | :--- | :--- | :--- | |
| 55 | +| **6. Dispatch Natural Language Subtask** | Executor (Python) | Orchestrator dispatches the Stage Agent, providing a clear, high-level **natural language description** of the subtask. | | |
| 56 | +| **7. Local Reasoning and Structured Proposal** | Stage Agent (LLM Reasoning + Tools) | The specialized Stage Agent uses its **domain-specific tools or skills** to select the optimal low-level method and returns a **structured JSON proposal** (the execution contract) to the Orchestrator, e.g.: `{"stage_id":"visualize_1","proposed_tool":"visualize heatmap","args":{"input":"...","output":"..."}}`. | | |
| 57 | +| **Required Schema Contract** | JSON Payload | Must include: `stage_id`, **`proposed_tool`** (alias for capabilities lookup), mandatory **`args`**, optional `justification`, and structured outputs to match the capabilities registry. | | |
| 58 | +| **8. Proposal Validation and Hook** | Validator (Guardrails / Pydantic) | Validator performs **schema enforcement and safety checks**. It uses the `proposed_tool` alias to verify the command against the **capabilities registry** and ensure the `args` confirm **proper inputs/outputs**. | | |
| 59 | +| **8a. Validation Rejection Escalation** | Planner / User Interaction | If the proposal is rejected (e.g., safety failure, invalid schema): **1. Retry** with the same stage agent (keeping context) and request safer parameters. **2. Enter the Clarification Loop** if required inputs are missing. **3. Bubble an error** to the user if recovery is impossible (e.g., unsupported tool). | | |
| 60 | + |
| 61 | +--- |
| 62 | + |
| 63 | +### Phase 3: Central Execution and Provenance |
| 64 | + |
| 65 | +This phase maintains the core principle that the Orchestrator, running Python code, is the sole executor, ensuring the workflow is **deterministic and auditable**. |
| 66 | + |
| 67 | +| Step | Component | Action / Description | Source Alignment | |
| 68 | +| :--- | :--- | :--- | :--- | |
| 69 | +| **9. Centralized Execution** | Executor (Python) | If validated (or if the stage was deterministic), the Orchestrator’s Python Executor invokes the necessary internal runtime function (e.g., `_run_cli` or `_run_function`) using the approved command and parameters from the structured proposal. **The Stage Agent does not execute code remotely**. | | |
| 70 | +| **10. Provenance Logging** | Provenance & Memory | The Executor ensures the stage **logs its artifacts, metadata, hash signatures, accepted/rejected suggestions, and tool proposals/validation results**. This achieves **full transparency and reproducibility**. | | |
| 71 | +| **11. Workflow Advancement** | Planner / Executor | The Planner updates the DAG state and dispatches the next dependent Stage Agent or signals workflow completion. | | |
| 72 | + |
| 73 | +## Planner Enhancements |
| 74 | + |
| 75 | +`poetry run zyra plan --intent "<request>" --output plan.json --verbose` now: |
| 76 | + |
| 77 | +- Emits a reasoning trace explaining why each stage was selected |
| 78 | +- Prompts for missing arguments (e.g., FTP path, pad-missing fill mode) |
| 79 | +- Invokes the **Value Engine** to suggest optional stages (narrate summaries, verification checks, retry logic). Suggestions can be accepted interactively. |
| 80 | +- Automatically materializes verify stages when `scan-frames` detects missing or duplicate timestamps, wiring `verify evaluate --metric completeness --input <frames_meta>` into the manifest. |
| 81 | + |
| 82 | +### Example Plan Intent |
| 83 | + |
| 84 | +```bash |
| 85 | +poetry run zyra plan \ |
| 86 | + --intent "Download the last year of Weekly Drought Risk PNG frames from FTP, fill missing frames, compose an MP4 animation, and save to disk." \ |
| 87 | + --output plan.json --verbose |
| 88 | +``` |
| 89 | + |
| 90 | +**What to expect:** |
| 91 | + |
| 92 | +- Verbose output logs the planner/LLM reasoning, clarification prompts, and suggestions. |
| 93 | +- `plan.json` contains agents for `fetch_frames`, `scan_frames`, `pad_missing`, `compose_animation`, `save_local`, `verify_animation`, and `narrate_animation`. |
| 94 | +- The Value Engine typically suggests adding narration and verification stages; accepting the narrate suggestion inserts a second proposal agent (e.g., `narrate_1`) with a specific intent like “Describe spatial patterns.” |
| 95 | +- `plan_summary` now lists human-readable justifications for each agent (helpful downstream and for provenance). |
| 96 | + |
| 97 | +## Value Engine Suggestions |
| 98 | + |
| 99 | +The Value Engine inspects the manifest + semantic intent tags and offers low-cost augmentations (narration, verification, logging, etc.). Key upgrades in this branch: |
| 100 | + |
| 101 | +- Suggestions include `intent_text` so accepted items append context back into the manifest intent. |
| 102 | +- Verify suggestions can carry concrete `agent_template` payloads (e.g., `verify evaluate --metric completeness --input data/frames_meta.json`) so the planner inserts runnable agents rather than proposals. |
| 103 | +- Accepted suggestions are recorded under `accepted_suggestions`; unaccepted ones remain in `suggestions` for reference. |
| 104 | + |
| 105 | +## Running the Plan with `zyra swarm` |
| 106 | + |
| 107 | +``` |
| 108 | +poetry run zyra swarm \ |
| 109 | + --plan plan.json \ |
| 110 | + --memory run.db \ |
| 111 | + --log-events \ |
| 112 | + --parallel \ |
| 113 | + --provider ollama --model gemma --base-url http://host.docker.internal:11434 |
| 114 | +``` |
| 115 | + |
| 116 | +**Highlights:** |
| 117 | + |
| 118 | +- `--memory run.db` records provenance (agent stats, verify/narrate results). Use `--dump-memory run.db` to inspect past runs. |
| 119 | +- `--log-events` prints a stream like `[event agent_verify_result] agent=verify_animation stage=verify verdict=passed …`. |
| 120 | +- `--parallel/--no-parallel` toggles concurrency (defaults to auto). `--agents simulate,decide,narrate` runs a subset if you need to focus on specific stages. |
| 121 | +- `--provider/--model/--base-url` let you override the LLM client without editing `.env`; the planner and Value Engine honor the same settings. |
| 122 | + |
| 123 | +### Sample Output (truncated) |
| 124 | + |
| 125 | +``` |
| 126 | +[event run_started] {"agent_count": 8, ...} |
| 127 | +[event agent_started] {"agent": "fetch_frames", "stage": "acquire"} |
| 128 | +... |
| 129 | +[event agent_verify_result] agent=verify_animation stage=verify verdict=passed metric=completeness message=verify evaluate: completeness PASSED - all frames present |
| 130 | +[event agent_completed] {"agent": "narrate_animation", "role": "narrate", "duration_ms": 15957} |
| 131 | +[event run_completed] agent_count=8 errors=0 failed=0 proposals=narrate_1:swarm, narrate_animation:swarm |
| 132 | +``` |
| 133 | + |
| 134 | +The narrate stages receive enriched inputs (plan summary, frames metadata, verify results) so `narrative_summary.yaml` references missing weeks, sample frames, and completeness verdicts instead of just codec info. |
| 135 | + |
| 136 | +## Sample Manifests |
| 137 | + |
| 138 | +- `samples/swarm/mock_basic.yaml`: Minimal simulate → narrate DAG (mock outputs). |
| 139 | +- `samples/swarm/drought_animation.yaml`: Full drought pipeline (FTP download, pad-missing, compose-video, narrate, verify, save). |
| 140 | +- `samples/swarm/simulate_decide.yaml` *(new)*: Chains simulate → decide → narrate to exercise the skeleton stages without real data. |
| 141 | + |
| 142 | +Use `poetry run zyra swarm --plan samples/swarm/simulate_decide.yaml --dry-run` to verify the mock pipeline works end-to-end. |
| 143 | + |
| 144 | + |
0 commit comments