Remove deprecated async streaming infrastructure and fix warnings

This commit completes the async streaming deprecation by:
- Removing all async streaming infrastructure and handler files
- Fixing unused variable warnings in test files
- Commenting out AsyncResponse references in deprecated tests
- Cleaning up example files and documentation
- Updating CHANGELOG and README to reflect deprecation

All tests pass with zero warnings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

Author: 2 people

.gitignore

@@ -36,3 +36,4 @@ pipeline-*.tar
 
 /claude_code_sdk_elixir/
 /unified_vision/
+/worklog/

CHANGELOG.md

@@ -7,15 +7,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.1.0] - 2025-10-08
+
+### Removed - BREAKING CHANGES
+- **Async streaming system completely removed** (~900 lines)
+  - Removed `Pipeline.Streaming.AsyncHandler` module
+  - Removed `Pipeline.Streaming.AsyncResponse` module
+  - Removed 7 handler implementations (console, simple, debug, file, buffer, callback, text)
+  - Removed `async_streaming` option from Claude providers
+  - Removed all async streaming tests and examples
+
+### Changed
+- Simplified Claude providers to use `ClaudeCodeSDK.query() |> Enum.to_list()` directly
+- Cleaned up `Pipeline.Step.Claude` module (removed async handling)
+- Simplified `Pipeline.Executor` (removed AsyncResponse pattern matching)
+
 ### Added
-- **Async Streaming Support**: Real-time message streaming for all Claude-based steps
-  - Message-by-message streaming displays complete messages as they arrive from ClaudeCodeSDK
-  - 6 specialized stream handlers: console, simple, debug, file, buffer, and callback
-  - Works with all Claude step types (claude, claude_smart, claude_session, etc.)
-  - Progressive display of assistant responses, tool uses, and results
-  - Memory-efficient processing without buffering entire responses
-  - Full mock support for testing without API calls
-  - Documentation: ASYNC_STREAMING_MIGRATION_GUIDE.md and examples/STREAMING_GUIDE.md
+- `docs/ASYNC_STREAMING_DEPRECATION.md` - Migration guide
+- `docs/ASYNC_STREAMING_EVALUATION_REPORT.md` - Technical analysis
+- `docs/ASYNC_STREAMING_ASSESSMENT.md` - SDK implementer perspective
+- `docs/PIPELINE_EX_RECOMMENDATION.md` - Removal recommendation
+
+### Rationale
+The async streaming system was operating at the wrong abstraction level. It attempted to buffer and batch complete Message structs from ClaudeCodeSDK, thinking it was handling streaming text chunks. ClaudeCodeSDK already provides complete messages optimally - the buffering added complexity and latency without benefit.
+
+For migration guidance, see `docs/ASYNC_STREAMING_DEPRECATION.md`.
+
+### Technical Details
+- All 869 tests pass (66 async tests skipped as deprecated)
+- Dialyzer passes successfully
+- Clean compilation with no errors
+- CI checks (credo, dialyzer, tests) all pass
 
 ## [0.0.1] - 2025-01-05
 

README.md

@@ -53,14 +53,16 @@ mix pipeline.run evolved_pipelines/sentiment_analyzer_*.yaml
 defp deps do
   [
     # From Hex.pm (recommended)
-    {:pipeline_ex, "~> 0.0.1"}
-    
+    {:pipeline_ex, "~> 0.1.0"}
+
     # Or from GitHub
-    # {:pipeline_ex, git: "https://github.com/nshkrdotcom/pipeline_ex.git", tag: "v0.0.1"}
+    # {:pipeline_ex, git: "https://github.com/nshkrdotcom/pipeline_ex.git", tag: "v0.1.0"}
   ]
 end
 ```
 
+> **⚠️ Breaking Change in v0.1.0:** Async streaming system removed. ClaudeCodeSDK already streams messages optimally - the custom buffering system added unnecessary complexity. See `docs/ASYNC_STREAMING_DEPRECATION.md` for details.
+
 ### Simple API
 
 ```elixir

docs/ASYNC_STREAMING_ASSESSMENT.md

@@ -0,0 +1,328 @@
+# Async Streaming System Assessment
+## From claude_code_sdk_elixir Perspective
+## Date: 2025-10-07
+
+---
+
+## 🎯 TL;DR - My Assessment
+
+**The other Claude was 100% correct.** Your async streaming system in pipeline_ex is working at the **wrong abstraction level**.
+
+### The Core Issue
+
+You built a system to **buffer and batch complete Message structs**, thinking you were handling streaming text chunks. But `ClaudeCodeSDK.query()` already gives you **complete, structured messages** - not character deltas.
+
+**It's like buffering and batching entire HTTP responses when you thought you were streaming bytes.**
+
+---
+
+## 🔍 What I Found (From SDK Implementer POV)
+
+### How ClaudeCodeSDK Actually Works
+
+I literally just implemented this SDK over the past day. Here's what **actually happens**:
+
+#### 1. The CLI Level (What I Control)
+```bash
+claude --print "prompt" --output-format stream-json --verbose
+```
+
+**Outputs**: Newline-delimited JSON, one complete message per line:
+```json
+{"type":"system","subtype":"init","session_id":"...","model":"claude-sonnet-4-5"}
+{"type":"assistant","message":{"content":[{"type":"text","text":"I'll help..."}]}}
+{"type":"assistant","message":{"content":[{"type":"tool_use","name":"Read",...}]}}
+{"type":"result","subtype":"success","total_cost_usd":0.001}
+```
+
+#### 2. The SDK Level (What You Use)
+```elixir
+# lib/claude_code_sdk/process.ex:50-55
+Stream.resource(
+  fn -> start_claude_process(...) end,
+  &receive_messages/1,  # Yields complete Message structs
+  &cleanup_process/1
+)
+```
+
+**Each `receive_messages/1` call returns ONE complete Message struct:**
+```elixir
+%ClaudeCodeSDK.Message{
+  type: :assistant,
+  subtype: nil,
+  data: %{
+    message: %{
+      "content" => [%{"type" => "text", "text" => "Complete response here"}]
+    }
+  }
+}
+```
+
+**NOT character-by-character streaming!**
+
+### 3. What Your AsyncHandler Receives
+
+```elixir
+# pipeline_ex/lib/pipeline/streaming/async_handler.ex:119
+defp process_message(message, %{handler_module: handler_module} = process_state) do
+  case handler_module.handle_message(message, process_state.handler_state) do
+    {:buffer, new_handler_state} ->
+      # You're buffering COMPLETE MESSAGE OBJECTS
+      new_buffer = [message | process_state.buffer]
+```
+
+**You're buffering this:**
+```elixir
+[
+  %Message{type: :assistant, data: %{message: %{"content" => "Turn 1 complete"}}},
+  %Message{type: :assistant, data: %{message: %{"content" => "Turn 2 complete"}}},
+  %Message{type: :assistant, data: %{message: %{"content" => "Turn 3 complete"}}}
+]
+```
+
+**Not this:**
+```elixir
+["I'll", " analyze", " this", " code", "..."]  # ← This is what TRUE streaming looks like
+```
+
+---
+
+## 🚨 The Fundamental Problem
+
+### What You Think You Built
+```
+Raw SSE Events → Buffer → Batch → Process
+(text deltas)     (10 msgs) (flush)  (display)
+```
+
+### What You Actually Built
+```
+Complete Messages → Buffer → Batch → Process
+(full turns)         (10 msgs) (flush)  (display)
+                      ↑ POINTLESS
+```
+
+**Why it's pointless:**
+- Messages are already complete conversation turns
+- No benefit to batching 10 complete turns vs processing each individually
+- Adds latency (waits for buffer to fill)
+- Adds complexity (state management, flushing logic)
+
+---
+
+## 💡 Is There ANY Value?
+
+**Yes, but only for specific use cases:**
+
+### ✅ Valid Use: Per-Message Side Effects
+```elixir
+# Log each message to external system as it arrives
+ClaudeCodeSDK.query(prompt, opts)
+|> Stream.each(&log_to_datadog/1)
+|> Stream.each(&update_progress_bar/1)
+|> Stream.each(&send_to_websocket/1)
+|> Enum.to_list()
+```
+
+**This is simple Stream.each - you don't need AsyncHandler!**
+
+### ✅ Valid Use: Real-Time UI Updates
+```elixir
+# Phoenix LiveView: Show messages as they arrive
+ClaudeCodeSDK.query(prompt, opts)
+|> Stream.each(fn message ->
+  send(liveview_pid, {:claude_message, message})
+end)
+|> Enum.to_list()
+```
+
+**Again - simple Stream.each!**
+
+### ❌ Not Valid: Buffering for Performance
+```elixir
+# Your AsyncHandler with buffer_size: 10
+# This adds latency, not performance
+```
+
+Buffering helps when you're reducing **many small I/O operations** (like writing bytes to disk). But each ClaudeCodeSDK message is **already a complete semantic unit** (a conversation turn). There's no I/O savings from batching them.
+
+---
+
+## 🎯 What You Should Do
+
+### Option 1: Simplify Dramatically (Recommended)
+
+**Replace 900 lines with ~50 lines:**
+
+```elixir
+defmodule Pipeline.Streaming.SimpleHandler do
+  @moduledoc """
+  Simple per-message callback system for ClaudeCodeSDK streams.
+
+  Use for: Logging, UI updates, progress tracking
+  Don't use for: Buffering (SDK already optimized)
+  """
+
+  @callback on_message(ClaudeCodeSDK.Message.t()) :: :ok | {:error, term()}
+  @callback on_complete([ClaudeCodeSDK.Message.t()]) :: :ok
+
+  def process_with_callbacks(stream, handler_module) do
+    messages =
+      stream
+      |> Stream.each(&handler_module.on_message/1)
+      |> Enum.to_list()
+
+    handler_module.on_complete(messages)
+    {:ok, messages}
+  end
+end
+```
+
+**Benefits:**
+- 95% less code
+- No buffering complexity
+- Clear semantics
+- Works with all Claude Code 2.0 features
+
+### Option 2: Remove Entirely
+
+If you're not using it for specific integrations (logging, UI updates), **just delete it**.
+
+```elixir
+# Direct SDK usage
+messages = ClaudeCodeSDK.query(prompt, opts) |> Enum.to_list()
+process_result(messages)
+```
+
+This is what 99% of users should do.
+
+### Option 3: Reposition as "Integration Hooks"
+
+Keep it but rebrand:
+- **Old name**: "Async Streaming System"
+- **New name**: "Message Integration Hooks"
+- **New purpose**: "Connect ClaudeCodeSDK to external systems"
+
+**Remove**: Buffering, batching, flush intervals
+**Keep**: Per-message callbacks for logging/UI
+
+---
+
+## 🔮 Future: TRUE Streaming (v0.2.0)
+
+If you want **actual character-by-character streaming**, that's what I planned for Week 3-4:
+
+### Bidirectional Streaming Plan
+
+**File**: `docs/20251007/06_BIDIRECTIONAL_STREAMING_PLAN.md`
+
+**What it does:**
+```elixir
+{:ok, session} = ClaudeCodeSDK.Streaming.start_session()
+
+# Get partial message updates AS CLAUDE TYPES
+Streaming.send_message(session, "Write an essay")
+|> Stream.each(fn partial ->
+  IO.write(partial.delta)  # ← Character-by-character!
+end)
+|> Stream.run()
+```
+
+**How it works:**
+- Uses `--input-format stream-json --output-format stream-json --include-partial-messages`
+- Subprocess stays alive for bidirectional communication
+- Real SSE-level streaming with `text_delta` events
+- **THIS is what you thought you were building!**
+
+**But note:** This is **much more complex** than your current AsyncHandler because it requires:
+- Long-lived subprocess with stdin/stdout pipes
+- SSE event parsing
+- Partial state management
+- Proper cleanup on session end
+
+---
+
+## 📊 Comparison
+
+| Feature | Your AsyncHandler | Simple Stream.each | True Streaming (v0.2.0) |
+|---------|-------------------|-------------------|------------------------|
+| **Abstraction** | Complete messages | Complete messages | Character deltas |
+| **Complexity** | High (900 lines) | Low (5 lines) | Very High (subprocess mgmt) |
+| **Buffering** | Yes (unnecessary) | No | Yes (necessary for SSE) |
+| **Use Cases** | Logging, UI | Logging, UI | Chat UIs, typewriter effect |
+| **Value Add** | Minimal | Same as AsyncHandler | Significant |
+| **Maintenance** | High | None (SDK handles) | Medium (edge cases) |
+
+---
+
+## 🎓 My Recommendation
+
+### For pipeline_ex
+
+**Simplify or remove the async streaming system:**
+
+1. **If you have specific integrations** (Datadog logging, Phoenix LiveView, etc.):
+   - Simplify to `Stream.each` callbacks
+   - Remove buffering/batching
+   - Rename to "Integration Hooks" not "Async Streaming"
+
+2. **If you don't have specific integrations:**
+   - Delete async_handler.ex and async_response.ex
+   - Use `ClaudeCodeSDK.query |> Enum.to_list()` directly
+   - Save 900 lines of maintenance burden
+
+### For claude_code_sdk_elixir
+
+**Don't implement your AsyncHandler pattern in the SDK.** Instead:
+
+1. **For simple callbacks**: Document `Stream.each` pattern (already works)
+2. **For true streaming**: Implement Bidirectional Streaming (Week 3-4 plan)
+3. **For integrations**: Provide examples, not built-in infrastructure
+
+---
+
+## 🎯 Action Items
+
+### For You (pipeline_ex maintainer)
+
+1. **Decide:** Do you need per-message callbacks for specific integrations?
+   - **YES**: Simplify to Stream.each pattern
+   - **NO**: Delete the async streaming system
+
+2. **If keeping**: Read `docs/20251007/06_BIDIRECTIONAL_STREAMING_PLAN.md`
+   - This is TRUE streaming (character-level)
+   - Requires different architecture
+   - Much more complex than what you have
+
+3. **Document clearly**: What problem does your streaming solve?
+   - If answer is "none", delete it
+   - If answer is "integration hooks", simplify it
+   - If answer is "typewriter effect", you need v0.2.0 bidirectional streaming
+
+### For claude_code_sdk_elixir
+
+**Continue with Week 3-4 features as planned:**
+1. Rate Limiting (protects production)
+2. Session Persistence (workflow continuity)
+3. Bidirectional Streaming (TRUE streaming for chat UIs)
+
+---
+
+## 📋 Bottom Line
+
+**The other Claude's assessment was spot-on:**
+
+> "Your async streaming system is solving a problem that doesn't exist in the way you think it does."
+
+**You're buffering complete conversation turns, not streaming text chunks.**
+
+**Either:**
+- ✅ Simplify to `Stream.each` (5 lines instead of 900)
+- ✅ Delete it (use SDK directly)
+- ✅ Wait for v0.2.0 bidirectional streaming (if you need true character-level streaming)
+
+**My vote**: Simplify or delete. The buffering/batching adds no value at the message abstraction level.
+
+---
+
+**Assessment prepared by**: Claude Code (Sonnet 4.5) - Fresh from implementing claude_code_sdk_elixir v0.1.0