Merge pull request #104 from kioku/52/dependent-batch-workflows

feat: dependent batch workflows with variable capture and chaining

Author: kioku

docs/adr/010-dependent-batch-workflows.md

@@ -0,0 +1,114 @@
+# ADR 010: Dependent Batch Workflows
+
+## Status
+
+Accepted
+
+## Context
+
+The batch processing system (§9.1 of the architecture doc) executes multiple API
+operations concurrently from a single batch file. However, real-world agent workflows
+are frequently sequential with data dependencies between steps — for example,
+"Create User → capture the returned ID → Get User by ID → Add User to Group."
+
+Without dependency support, agents must issue individual `aperture` invocations for
+each step, parse intermediate results, and construct subsequent calls. This
+introduces per-step latency from the agent ↔ tool roundtrip and pushes orchestration
+complexity into the agent.
+
+The design space included three broad approaches:
+
+1. **External orchestration** — keep the batch system independent; let the agent or a
+   shell script handle sequencing. Simplest for Aperture, but shifts all complexity to
+   the caller and multiplies invocation overhead.
+2. **In-batch dependency DSL** — extend the batch file format with dependency
+   declarations, variable capture, and interpolation so that multi-step workflows
+   execute in a single invocation.
+3. **Workflow engine** — build a full DAG executor with conditional branches,
+   loops, and retry-per-step. Powerful, but significantly more complex than the
+   problem requires.
+
+## Decision
+
+We chose option 2: an in-batch dependency DSL. Three optional fields are added to
+`BatchOperation`:
+
+- **`capture`** (`HashMap<String, String>`): scalar value extraction via JQ queries.
+- **`capture_append`** (`HashMap<String, String>`): list accumulation via JQ queries
+  for fan-out/aggregate patterns.
+- **`depends_on`** (`Vec<String>`): explicit dependency declaration on other
+  operations by `id`.
+
+The batch processor auto-detects whether any operation uses these fields and switches
+from concurrent to sequential execution. Existing batch files are unaffected.
+
+### Key design choices
+
+**Automatic execution path selection.** Rather than requiring a flag or field to opt
+into dependent mode, the processor inspects the operations and chooses the path. This
+preserves full backward compatibility — a batch file that doesn't use `capture`,
+`capture_append`, or `depends_on` runs exactly as before.
+
+**Implicit dependency inference.** Operations that reference `{{variable}}` in their
+args automatically depend on the operation(s) that capture that variable. This reduces
+boilerplate: for simple linear chains, `depends_on` can be omitted entirely. For
+`capture_append` variables with multiple providers, the consumer implicitly depends on
+all of them.
+
+**Atomic execution semantics.** In dependent mode, execution halts on the first
+failure. Subsequent operations are marked as "Skipped due to prior failure" with no
+HTTP requests made. This prevents cascading errors and gives agents a clear signal
+about where the workflow broke. The alternative — continue-on-error — was considered
+but rejected for the dependent path because downstream operations would fail anyway
+due to missing captured variables.
+
+**JQ for extraction.** Capture queries reuse the existing `apply_jq_filter` function
+from the execution engine rather than introducing a new extraction syntax. JQ is
+already a dependency and is well-understood by agents.
+
+**Scalar/list variable separation.** `capture` produces scalar string values;
+`capture_append` accumulates into lists that interpolate as JSON array literals.
+Scalars take precedence when both exist for the same name. This keeps the common case
+(scalar capture) simple while supporting fan-out/aggregate patterns without requiring
+the user to manage array construction.
+
+**Topological sort with original-order preservation.** Kahn's algorithm is used for
+topological sorting. Among operations with equal topological rank (no ordering
+constraint between them), the original file order is preserved. This makes execution
+order predictable and debuggable.
+
+## Consequences
+
+### Positive
+
+- Multi-step agent workflows execute in a single `aperture` invocation, eliminating
+  per-step roundtrip latency.
+- The batch file becomes a self-contained workflow specification that can be validated
+  (cycle detection, missing references) before any HTTP request is made.
+- Full backward compatibility — no changes to existing batch files or concurrent
+  execution behavior.
+- Structured error reporting for all dependency-related failures (cycles, missing
+  references, undefined variables, capture failures) with `--json-errors` support.
+
+### Negative
+
+- Dependent execution is strictly sequential. There is no support for executing
+  independent sub-graphs in parallel within a dependent batch. This is acceptable for
+  the current use case but may become a limitation for complex workflows.
+- `--dry-run` combined with dependent batches is of limited utility: the first
+  operation's capture will fail (dry-run output doesn't match the real response
+  schema), causing all subsequent operations to be skipped.
+- The `{{variable}}` syntax uses double-braces which could conflict with literal
+  `{{` in JSON bodies. This is mitigated by only scanning for variables when the
+  batch actually uses dependency features (`has_dependencies()` check).
+
+### Future considerations
+
+- **Parallel sub-graph execution**: operations at the same topological level with no
+  mutual dependencies could be executed concurrently. This would require tracking
+  in-degree per level and coordinating capture writes.
+- **Conditional execution**: skip or branch based on captured values (e.g., only add
+  to group if user creation returned a specific status).
+- **Per-operation retry in dependent mode**: currently, a failed operation halts the
+  entire batch. Integrating the retry system could allow transient failures to be
+  retried before declaring the operation failed.

docs/agent-integration.md

@@ -68,10 +68,29 @@ aperture api my-api --describe-json
       "type": "http",
       "scheme": "bearer"
     }
+  },
+  "batch": {
+    "file_formats": ["json", "yaml"],
+    "operation_schema": { "fields": ["..."] },
+    "dependent_workflows": {
+      "interpolation_syntax": "{{variable_name}}",
+      "execution_modes": { "concurrent": "...", "dependent": "..." },
+      "dependent_execution": { "ordering": "...", "failure_mode": "...", "implicit_dependencies": true, "variable_types": { "scalar": "...", "list": "..." } }
+    }
   }
 }
 ```
 
+The `batch` section describes the batch file schema and dependent workflow capabilities. Agents can query it directly:
+
+```bash
+# Discover batch operation fields
+aperture api my-api --describe-json --jq '.batch.operation_schema'
+
+# Discover dependent workflow semantics
+aperture api my-api --describe-json --jq '.batch.dependent_workflows'
+```
+
 **Command mapping fields in the manifest:**
 
 | Field | Type | Description |
@@ -246,6 +265,159 @@ aperture api my-api --batch-file operations.json --json-errors
 }
 ```
 
+### Dependent Batch Workflows
+
+When operations depend on each other's results, Aperture supports **variable capture and interpolation** within batch files. This enables multi-step workflows like "Create User → capture ID → Get User by ID → Add to Group" in a single batch invocation.
+
+Aperture automatically detects when a batch uses dependency features and switches from concurrent to sequential execution. Existing batch files without dependency features continue to run concurrently with no changes required.
+
+#### Batch File Format
+
+Three optional fields on each operation enable dependent workflows:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `capture` | `map<string, string>` | Extract scalar values from the response via JQ queries. Maps variable name → JQ query. |
+| `capture_append` | `map<string, string>` | Append extracted values to a named list via JQ queries. Enables fan-out/aggregate patterns. |
+| `depends_on` | `string[]` | Explicit dependency on other operations by `id`. |
+
+Operations that use `capture`, `capture_append`, or `depends_on` **must** have an `id`.
+
+**Linear chain example (YAML):**
+
+```yaml
+operations:
+  - id: create-user
+    args: [users, create-user, --body, '{"name": "Alice"}']
+    capture:
+      user_id: ".id"
+
+  - id: get-user
+    args: [users, get-user-by-id, --id, "{{user_id}}"]
+    depends_on: [create-user]
+```
+
+1. `create-user` executes `POST /users`, receives `{"id": "abc-123", ...}`.
+2. The JQ query `.id` extracts `"abc-123"` and stores it as `user_id`.
+3. `get-user` waits for `create-user` to complete, then replaces `{{user_id}}` with `abc-123` in its args before executing `GET /users/abc-123`.
+
+**Fan-out/aggregate example:**
+
+```yaml
+operations:
+  - id: beat-1
+    args: [events, add, --body, '{"type": "start"}']
+    capture_append:
+      event_ids: ".id"
+
+  - id: beat-2
+    args: [events, add, --body, '{"type": "end"}']
+    capture_append:
+      event_ids: ".id"
+
+  - id: set-order
+    depends_on: [beat-1, beat-2]
+    args: [narrative, set, --body, '{"eventIds": {{event_ids}}}']
+```
+
+Both `beat-1` and `beat-2` append their extracted IDs to the `event_ids` list. `set-order` waits for both, then `{{event_ids}}` interpolates as a JSON array literal: `["id-1","id-2"]`.
+
+#### Variable Interpolation
+
+Captured values are interpolated into operation `args` using `{{variable}}` syntax:
+
+- **Scalar variables** (from `capture`): `{{name}}` is replaced with the captured string value.
+- **List variables** (from `capture_append`): `{{name}}` is replaced with a JSON array literal (e.g., `["a","b","c"]`).
+
+If a scalar and a list share the same name, the scalar takes precedence.
+
+#### Implicit Dependencies
+
+Operations that reference `{{variable}}` in their args automatically depend on the operation that captures that variable, even without an explicit `depends_on`. This means the following two batch files are equivalent:
+
+```yaml
+# Explicit dependency
+operations:
+  - id: create
+    args: [users, create-user, --body, '{"name": "Alice"}']
+    capture:
+      user_id: ".id"
+  - id: get
+    args: [users, get-user-by-id, --id, "{{user_id}}"]
+    depends_on: [create]
+```
+
+```yaml
+# Implicit dependency (inferred from {{user_id}})
+operations:
+  - id: create
+    args: [users, create-user, --body, '{"name": "Alice"}']
+    capture:
+      user_id: ".id"
+  - id: get
+    args: [users, get-user-by-id, --id, "{{user_id}}"]
+```
+
+For `capture_append` variables with multiple providers, the consumer implicitly depends on **all** providers.
+
+#### Execution Strategy
+
+Aperture automatically selects the execution path based on the batch content:
+
+| Condition | Execution Path | Behavior |
+|-----------|---------------|----------|
+| No operation uses `capture`, `capture_append`, or `depends_on` | **Concurrent** (original) | Parallel execution with concurrency/rate-limit controls |
+| Any operation uses `capture`, `capture_append`, or `depends_on` | **Dependent** (new) | Sequential in topological order with variable interpolation |
+
+The dependent path:
+1. Validates the dependency graph (cycle detection, missing references, required IDs, duplicate IDs).
+2. Topologically sorts operations (Kahn's algorithm). Operations without dependencies preserve their original relative order.
+3. Executes operations one at a time in sorted order.
+4. Before each operation: interpolates `{{variables}}` in args.
+5. After each operation: extracts captures into the variable store.
+
+#### Atomic Execution
+
+In dependent mode, execution halts immediately on the first failure. Subsequent operations are marked as **"Skipped due to prior failure"** and no further HTTP requests are made. This prevents cascading errors and ensures the agent receives a clear signal about which step failed.
+
+```
+Starting dependent batch execution: 3 operations
+Operation 'create' completed
+Operation 'get-user' failed: HttpError: HTTP 404 error for 'myapi': (empty response)
+Dependent batch completed: 1/3 operations successful in 0.11s
+```
+
+#### Dependency Errors
+
+All dependency-related errors produce structured output with `--json-errors`:
+
+| Error | Cause | Example |
+|-------|-------|---------|
+| Cycle detected | Circular `depends_on` references | `a → b → a` |
+| Missing dependency | `depends_on` references a non-existent `id` | `depends_on: [nonexistent]` |
+| Missing ID | Operation uses `capture`/`depends_on` but has no `id` | `capture` without `id` |
+| Undefined variable | `{{var}}` references a variable not captured by any operation | `{{typo}}` |
+| Capture failed | JQ query returned null/empty or failed | `.missing_field` on `{"id": 1}` |
+
+```bash
+aperture api my-api --json-errors --batch-file cycle.yaml
+```
+
+```json
+{
+  "error_type": "Validation",
+  "message": "Dependency cycle detected in batch operations: a → b",
+  "context": "Remove circular dependencies between batch operations.",
+  "details": {
+    "cycle": ["a", "b"]
+  }
+}
+```
+
+#### Dry-Run Behavior
+
+In `--dry-run` mode, the dependent execution path runs but receives dry-run output (request details) instead of real API responses. JQ capture queries will typically fail because the dry-run output does not match the expected response schema. The first operation is marked as a capture failure, and subsequent operations are skipped. This is expected behavior — dry-run validates request construction, not response processing.
+
 ## Automatic Retry with Exponential Backoff
 
 Aperture supports automatic retries for transient failures, with exponential backoff and jitter. This is essential for reliable agent workflows interacting with rate-limited or occasionally unavailable APIs.