Improved documentation

Author: travis-bauer

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## Unreleased
+- Improved tutorial documentation: condensed main tutorials README with learning path diagram,
+  linked to quickstart for installation; added navigation links, expected outputs, and
+  troubleshooting sections to all three tutorials; added chatterlang language tags and
+  snippet intent labels; fixed file_structure.md header and documented step3_extras naming.
+- Clarified tutorial docs: "Get Started in 5 Minutes" now lists all three steps (search UI
+  requires Step 3); Key Concepts and Beyond the Web Interface no longer imply Elasticsearch
+  or Pinecone are built-in—they are custom segments or plugins you can add.
+- Rewrote quickstart.md: added non-LLM examples (echo, cast, formatItem) so users can run
+  pipelines without Ollama/OpenAI; documented all pip install extras (ollama, openai,
+  anthropic, pypdf, all); reordered content with no-LLM examples first.
+
 ## 0.11.4
 - Documentation example fixes: added missing imports and stubs to extending-talkpipe.md,
   metadata-stream.md, pipe-api.md; introduced `# skip-extract` for blocks that cannot run

docs/quickstart.md

@@ -1,154 +1,154 @@
 # Getting Started with TalkPipe
 
-Welcome to TalkPipe! This guide will help you get up and running quickly with TalkPipe's dual-language architecture for building AI-powered data processing pipelines.
+This guide gets you up and running with TalkPipe's dual-language architecture for building data processing pipelines. You can start with **no LLM**—examples below work with the base install—then add LLM support when you're ready.
 
 ## Installation
 
 ```bash
 pip install talkpipe
 ```
 
-For LLM support, install the provider(s) you need:
+The base install includes data processing, file I/O, search (Whoosh, LanceDB), and web serving. For LLM and PDF support, add optional extras:
+
 ```bash
-# Install specific providers
-pip install talkpipe[openai]    # For OpenAI
-pip install talkpipe[ollama]    # For Ollama
-pip install talkpipe[anthropic] # For Anthropic Claude
+# LLM providers (install one or more)
+pip install talkpipe[ollama]      # Local models via Ollama
+pip install talkpipe[openai]      # OpenAI (GPT-4, etc.)
+pip install talkpipe[anthropic]   # Anthropic Claude
+
+# PDF extraction
+pip install talkpipe[pypdf]
+
+# Combine extras
+pip install talkpipe[ollama,pypdf]
+pip install talkpipe[openai,anthropic]
 
-# Or install all LLM providers
+# Everything: all LLM providers + PDF
 pip install talkpipe[all]
 ```
 
+| Extra | Adds |
+|-------|------|
+| `ollama` | `ollama` package for local models |
+| `openai` | `openai` package for OpenAI API |
+| `anthropic` | `anthropic` package for Claude |
+| `pypdf` | `pypdf` for PDF text extraction |
+| `all` | All of the above |
+
 ## Basic Concepts
 
-TalkPipe provides two ways to build data processing pipelines:
+TalkPipe provides two ways to build pipelines:
 
 - **Pipe API (Internal DSL)**: Pure Python using the `|` operator
 - **ChatterLang (External DSL)**: Concise text-based syntax
 
-Both approaches use the same underlying components and can be mixed freely.
-
-## Your First Pipeline
+Both use the same components and can be mixed.
 
-### Using ChatterLang
+## Your First Pipeline (No LLM Required)
 
-Create a simple chat interface in python:
+### ChatterLang
 
 ```python
 from talkpipe.chatterlang import compiler
 
-# Define a pipeline that prompts an LLM and prints the response.  Assumed Ollama is installed locally and llama3.2 is downloaded.
-script = '| llmPrompt[model="llama3.2", source="ollama"] | print'
-chat = compiler.compile(script).as_function(single_in=True, single_out=True)
+script = 'INPUT FROM echo[data="hello,world,test"] | print'
+pipeline = compiler.compile(script).as_function(single_out=False)
 
-# Use it
-response = chat("Hello! Tell me about the history of computers.")
+result = pipeline()
+# Prints: hello, world, test (each on its own line)
+# Returns: ['hello', 'world', 'test']
 ```
 
-### Using the Pipe API
-
-A similar interactive pipeline in pure Python:
+### Pipe API
 
 ```python
 from talkpipe.pipe import io
-from talkpipe.llm import chat
 
-# Create pipeline using the | operator
-pipeline = io.Prompt() | chat.LLMPrompt(model="llama3.2", source="ollama") | io.Print()
-pipeline_func = pipeline.as_function()
+pipeline = io.echo(data="hello,world,test") | io.Print()
+result = pipeline.as_function(single_out=False)()
 
-# Run it
-pipeline_func()  # This will prompt for input interactively
+# Same output and return value
 ```
 
-## Web Interface
+### Data Transformation (No LLM)
 
-Create a web interface for your pipeline from the command line:
+```python
+from talkpipe.chatterlang import compiler
 
-```bash
-# Use single quotes to avoid escaping double quotes inside
-chatterlang_serve --port 2025 --display-property prompt --script '| llmPrompt[model="llama3.2", source="ollama", field="prompt"]'
+# Parse numbers, filter, and print
+script = 'INPUT FROM echo[data="1,2,hello,3,4"] | cast[cast_type="int"] | print'
+pipeline = compiler.compile(script).as_function(single_out=False)
+pipeline()  # Skips "hello", prints 1, 2, 3, 4
 ```
 
-Open http://localhost:2025/stream in your browser to interact with your pipeline through a web form.
+## Your First Pipeline (With LLM)
 
-## Debug Pipelines with diagPrint
+Requires `talkpipe[ollama]` and Ollama running with a model (e.g. `ollama pull llama3.2`).
 
-Use `diagPrint` to inspect data as it flows through a pipeline without altering outputs.
-
-### Pipe API
+### ChatterLang
 
 ```python
-from talkpipe.pipe import basic
-
-debug = basic.DiagPrint(
-	label="chunk",
-	field_list="id,text",
-	expression="len(item['text'])",
-	output="stderr",  # or a logger name
-)
-pipeline = debug | basic.firstN(n=3)
-list(pipeline([{"id": 1, "text": "hello world"}]))
-```
+from talkpipe.chatterlang import compiler
 
-### ChatterLang
+script = '| llmPrompt[model="llama3.2", source="ollama"] | print'
+chat = compiler.compile(script).as_function(single_in=True, single_out=True)
 
-```
-| diagPrint[label="chunk", field_list="id,text", expression="len(item['text'])", output="stderr"]
+response = chat("Hello! Tell me about the history of computers.")
 ```
 
-### Config-driven output
+### Pipe API
 
-Set a config key (e.g., in `~/.talkpipe.toml`):
+```python
+from talkpipe.pipe import io
+from talkpipe.llm import chat
 
-```toml
-diag_output = "stderr"
+pipeline = io.Prompt() | chat.LLMPrompt(model="llama3.2", source="ollama") | io.Print()
+pipeline_func = pipeline.as_function()
+pipeline_func()  # Prompts for input interactively
 ```
 
-Then point `diagPrint` to it:
+## Web Interface
 
+Use `--display-property` so the stream UI shows the input field value instead of raw JSON. Use `formatItem` to format output for readable display.
+
+### Without LLM (Echo / Transform)
+
+```bash
+chatterlang_serve --port 2025 --display-property prompt --script '| formatItem[field_list="prompt:You entered"] | print'
 ```
-| diagPrint[output="config:diag_output"]
+
+Open http://localhost:2025/stream. Type in the form and submit; the pipeline echoes your input as formatted text.
+
+### With LLM
+
+```bash
+chatterlang_serve --port 2025 --display-property prompt --script '| llmPrompt[model="llama3.2", source="ollama", field="prompt"] | print'
 ```
 
-Tips:
-- Use `field_list` to print only the fields you need.
-- `None` or `"None"` for `output` suppresses all diagnostic output.
-- The elapsed time line shows spacing between successive items.
+Open http://localhost:2025/stream to chat with the LLM.
 
 ## Next Steps
 
 ### Learn the Tools
 
-- **[chatterlang_serve](api-reference/chatterlang-server.md)** - Create web APIs and forms
-- **[chatterlang_workbench](api-reference/chatterlang-workbench.md)** - Interactive development environment
+- **[chatterlang_serve](api-reference/chatterlang-server.md)** - Web APIs and forms
+- **[chatterlang_workbench](api-reference/chatterlang-workbench.md)** - Interactive development
 - **[chatterlang_script](api-reference/chatterlang-script.md)** - Run scripts from files
 
 ### Explore Tutorials
 
-Check out the [tutorials directory](tutorials/) for complete tutorials:
-- Document indexing and search
-- RAG (Retrieval-Augmented Generation) systems  
-- Multi-format report generation
+[Tutorials](tutorials/) walk through document indexing, RAG, and report generation. Tutorial 1 can use the included `stories.json` if you skip the LLM data-generation step.
 
 ### Extend with Plugins
 
-TalkPipe supports plugins to add custom functionality:
-
 ```bash
-# List installed plugins
 talkpipe_plugins --list
-
-# Install third-party plugins
 pip install talkpipe-some-plugin
 ```
 
-Plugins automatically extend ChatterLang with new sources and segments. See [Extending TalkPipe](architecture/extending-talkpipe.md) to create your own plugins.
+See [Extending TalkPipe](architecture/extending-talkpipe.md) to create plugins.
 
 ### Dive Deeper
 
 - [Architecture](architecture/) - Technical deep-dives
-- [API Reference](api-reference/) - Complete command reference
-
-
-Ready to build something amazing? Start with the [tutorials](tutorials/)!
+- [API Reference](api-reference/) - Command reference