Refactor ingestion pipeline: modularize chunking, enhance error handling, and improve API interaction

- Removed deprecated `embed` function and `create_config` from `embed.rs`.
- Introduced `chunking_utils.rs` for shared chunking logic, ensuring fields are chunked to ≤250 characters.
- Enhanced `marvelai_ingest.rs` with configurable retry and delay for embedding API calls, added debug output, and improved error messages.
- Updated `single_character_ingest.rs` to streamline processing of the 'Vision' character, including chunking and embedding with detailed logging.
- Created `ingest_utils.rs` for JSONL validation and error handling.
- Added a workspace structure in `Cargo.toml` for better project organization.
- Included a sample `vision.jsonl` file for testing and validation.

Author: BryanChasko

Cargo.toml

@@ -0,0 +1,6 @@
+[workspace]
+members = [
+    "src",
+    "rust_ingest"
+]
+resolver = "2"

rust_ingest/Cargo.toml

@@ -7,10 +7,15 @@ edition = "2021"
 name = "rust_ingest"
 path = "src/main.rs"
 
+[[bin]]
+name = "marvelai_ingest"
+path = "../src/ingest/marvelai_ingest.rs"
+
 [lib]
 name = "rust_ingest"
 path = "src/lib.rs"
 
+
 [dependencies]
 reqwest = { version = "0.12", features = ["json", "stream"] }
 tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
@@ -22,6 +27,7 @@ walkdir = "2"
 clap = { version = "4", features = ["derive"] }
 anyhow = "1"
 bincode = "1.3"
+tempfile = "3.6.0"
 
 [features]
 integration-tests = []

src/ingest/INGEST.md

@@ -4,6 +4,35 @@ This directory contains the Rust implementation of the HARALD ingestion
 pipeline, migrated from the previous `rust_ingest` directory as part of our
 organizational improvements.
 
+## Important Notes & Best Practices
+
+- **Chunking Limits:** All text fields are chunked to ≤250 characters for
+  reliable embedding. This is enforced in the code and recommended for API
+  stability and resource management.
+- **Error Handling:** The pipeline includes robust error handling and
+  diagnostics. Common error messages (e.g., "No chunks were processed") are
+  documented below.
+- **Debug Output:** Enable debug output to see chunk sizes and counts. This
+  helps diagnose chunking and embedding issues.
+- **CLI Features:** Some binaries require specific features (e.g.,
+  `--features="cli"`). Always check the documentation for required flags.
+- **Input/Output Formats:** Ingestion expects JSONL input and outputs vector
+  data to the `data/` directory. Validate your input and output files for
+  correct structure.
+
+## Troubleshooting
+
+- **No chunks were processed:** This message means the input file was empty or
+  incorrectly formatted. Check your JSONL file and ensure it contains valid
+  character objects.
+- **Network/API errors:** If embedding fails, check Ollama API status and logs.
+  Retry logic is built-in for transient failures.
+- **Debugging chunking:** Use the debug output to verify chunk sizes and counts.
+  If chunking does not match expectations, review the chunking logic and input
+  data.
+
+## Key Components
+
 ## Key Components
 
 - `main.rs` - CLI entry point for ingest and query operations
@@ -20,13 +49,25 @@ From the `src` directory:
 # Build the project
 cargo build
 
-# Run ingestion
+# Run ingestion (with CLI features if required)
+cargo run --bin single_character_ingest --features="cli"
+
+# Run main ingest
 cargo run -- ingest
 
 # Run query
 cargo run -- query "your query here"
 ```
 
+## Debugging & Validation
+
+- To enable debug output for chunking, run the test harness or ingest binary and
+  review the printed chunk sizes/counts.
+- Always validate your JSONL input before running ingestion. Use `jq` or similar
+  tools to check structure.
+
+## Migration Notes
+
 ## Migration Notes
 
 This code was migrated from the original `rust_ingest` directory as part of the
@@ -38,3 +79,12 @@ updated module structure that better fits our overall architecture.
 - Outputs vector data to the `data/` directory
 - Reads from project directories based on configuration
 - Provides both a library API and command-line interface
+- Expects JSONL input for ingestion; output files are validated and summarized
+  in logs
+
+## Common Pitfalls
+
+- Forgetting to enable required CLI features (e.g., `--features="cli"`)
+- Incorrectly formatted JSONL input (empty or missing fields)
+- Not keeping chunk sizes ≤250 characters, leading to API errors
+- Overlooking debug output for diagnostics