Refactor migration documentation and cleanup plans

- Updated JSON-TOOLS.md to reference new migration documentation.
- Revised TESTS.md to link to DEVELOPMENT-PRINCIPLES.md for cleanup plans.
- Added comprehensive Copilot instructions for HeraldStack, detailing project overview, architecture, developer workflows, and best practices.
- Created INGEST-MIGRATION-MODULAR-PLAN.md to outline the refactor to a modular ingestion architecture.
- Established a Migration Archive with historical documents and guidelines for future migrations.
- Documented the Script Migration Cleanup Plan, detailing the process for cleaning up shell scripts post-migration.
- Compiled a Script Migration to Rust Plan, outlining core principles and successfully migrated scripts.
- Summarized Shell Script Prevention strategies to prevent the creation of new shell scripts and promote automated tools.

Author: BryanChasko

.github/copilot-instructions.md

@@ -0,0 +1,73 @@
+# Copilot Instructions for HeraldStack
+
+## Project Overview
+- **HeraldStack** is a Rust-based, context-aware AI framework for personal and professional productivity, integrating memory, emotion, and modular execution across a cohort of AI entities.
+- The system is migrating all application logic from shell scripts to Rust binaries. Shell scripts are only used for infrastructure orchestration (e.g., deployment, CI/CD, AWS CLI).
+
+## Architecture & Key Components
+- **src/**: Core Rust code for all application logic, including data processing, JSON/JSONL tools, embedding, and ingestion.
+- **ai-entities/**: Definitions and metadata for AI personalities (see `entity-registry.json`, individual `.md` files).
+- **config/**: Schemas, ethics, and model configuration.
+- **docs/**: System design, migration, and vector search documentation.
+- **scripts/**: Shell scripts for deployment and validation (do not add new app logic here).
+- **rust_ingest/**: Rust CLI tools for ingestion and embedding (see `marvelai_ingest.rs`, `ingest.rs`).
+- **data/**: Vector store registry and ingested data.
+
+## Developer Workflows
+- **Build Rust Binaries:**
+  ```bash
+  cd src && cargo build --release --features cli
+  # Binaries: format_json, validate_json_schema, ingest_chunked, embedding_tool, text_chunker
+  ```
+- **Run Ingestion Pipeline:**
+  - Use Rust binaries for all data ingestion and embedding.
+  - Chunking and embedding logic is in `src/ingest/` and `rust_ingest/`.
+  - Embeddings are saved as `.emb.json` files alongside chunked data.
+- **Deploy:**
+  ```bash
+  ./scripts/deploy/deploy.sh [--build-only|prod|staging|--no-tests]
+  ```
+- **Validation & Linting:**
+  ```bash
+  ./scripts/validation/check-json.sh
+  ./scripts/validation/check-rust.sh
+  ./src/target/release/format_md
+  ```
+
+## Project-Specific Conventions
+- **No new shell scripts for app logic.** All new features must be implemented in Rust.
+- **Chunking for Embedding:**
+  - All text for embedding must be chunked to ≤250 characters (see `chunking_utils.rs`).
+  - Embeddings are generated per sub-chunk and saved as `<chunkfile>.<entity>.<field>.<start>-<end>.emb.json`.
+- **JSONL as Standard:**
+  - All ingestion and embedding pipelines use JSONL for input/output.
+- **Error Handling:**
+  - Use Rust's `anyhow` and `Result` for error propagation.
+  - All ingestion failures must be logged with actionable diagnostics.
+- **Documentation:**
+  - Update `README.md` and relevant `.md` files for any new workflow or tool.
+  - See `docs/vector-search/ollama-embedding-limits.md` for API constraints.
+
+## Integration & External Dependencies
+- **Ollama API** for embedding (see `docs/vector-search/ollama-embedding-limits.md`).
+- **AWS Lambda, S3, DynamoDB, Pinecone** for compute, storage, state, and semantic memory.
+- **No direct shell logic in Rust binaries.**
+
+## Examples
+- To add a new ingestion feature, extend `src/ingest/marvelai_ingest.rs` and document the CLI usage.
+- To validate JSON, use `src/utils/json_tools/` Rust binaries, not shell scripts.
+- To add a new entity, update `ai-entities/entity-registry.json` and add a corresponding `.md` file.
+
+## References
+- [README.md](../README.md)
+- [docs/vector-search/character-based-chunking.md](../docs/vector-search/character-based-chunking.md)
+- [docs/vector-search/ollama-embedding-limits.md](../docs/vector-search/ollama-embedding-limits.md)
+- [docs/CONTRIBUTING.md](../docs/CONTRIBUTING.md)
+
+---
+
+## Copilot Chat Preferences
+
+If you are unsure about a workflow or convention, check the documentation including the Root "HARALD" directory README.md and the appropriate .md file in the working directory, as well as general documentation in `docs/` paying special attention to the `migration/` directory for migration guidelines and the `CONTRIBUTING.md` for contribution standards.
+
+- When suggesting code or pipeline changes, do not offer multiple alternatives or open‑ended questions. Clearly state the proposed fix and tell me you will apply it after I say “yes.” Only ask me to reply “yes” when confirmation is absolutely necessary before executing the change.

Cargo.toml

@@ -1,6 +1,6 @@
+
 [workspace]
 members = [
-    "src",
     "rust_ingest"
 ]
 resolver = "2"

README.md

@@ -140,7 +140,13 @@ handling.
   Rust
 - [Project Structure](docs/migration/RECOMMENDED-STRUCTURE.md) - Recommended
   organization
-- [Migration Documentation](docs/migration/) - Detailed migration information
+- [Migration Documentation](docs/migration/) - Shell-to-Rust migration details
+- **Ingestion/Embedding Architecture:** All ingestion and embedding logic must
+  follow the
+  [Modular Ingest Refactor Plan](docs/migration/INGEST-MIGRATION-MODULAR-PLAN.md).
+  This plan defines the canonical, reusable ingest library and the pattern for
+  domain-specific wrappers (e.g., marvelai_ingest.rs). All new pipelines and
+  refactors must use this architecture and update documentation accordingly.
 
 ## Operating Model
 
@@ -159,6 +165,9 @@ pragmatic execution, and narrative continuity.
 - Workflows
 - [JSONL Format for Vector Embedding](docs/vector-search/jsonl-ingestion.md)
 - [Migration Documentation](docs/migration/) - Shell-to-Rust migration details
+- [Modular Ingest Refactor Plan](docs/migration/INGEST-MIGRATION-MODULAR-PLAN.md)
+  – Step-by-step plan for refactoring to a reusable, component-based ingestion
+  architecture
 
 ## Ethics & Consent
 
@@ -175,9 +184,19 @@ guidelines including those defined in
 - **Test Data**: Test fixtures are available in `tests/fixtures/` (see
   [FIXTURES.md](tests/fixtures/FIXTURES.md) for details)
 
-## Further Information
+## Directory Structure Overview
+
+For a detailed, canonical description of the project’s directory structure, see:
+
+- [docs/DETAILED.md](docs/DETAILED.md) – **Directory Structure and Naming Best
+  Practices** (includes a `tree` overview and rationale)
+- [docs/naming-conventions.md](docs/naming-conventions.md) – **Directory and
+  file naming conventions**
 
-See docs/DETAILED.md for more information.
+Other structure-related documents in `docs/migration/` (such as
+`RECOMMENDED-STRUCTURE.md`, `DIRECTORY-REORGANIZATION.md`, and
+`IMPLEMENTATION-PLAN.md`) are project planning artifacts and will be moved to a
+`docs/project-planning/` subdirectory.
 
 ---
 

docs/CONTRIBUTING.md

@@ -1,5 +1,9 @@
 # Contributing to HARALD
 
+**Created**: July 2025  
+**Last Updated**: July 24, 2025  
+**Version**: 1.1
+
 This document provides practical guidance for contributing to the HARALD
 project. For high-level development philosophy and principles, see
 [DEVELOPMENT-PRINCIPLES.md](DEVELOPMENT-PRINCIPLES.md).
@@ -16,7 +20,15 @@ Before contributing, please familiarize yourself with our
 - **Automation over manual work** - Use and extend our automated tools
 
 For the complete migration strategy and decision framework, refer to the
-[Development Principles](DEVELOPMENT-PRINCIPLES.md#migration-strategy--guidelines) document.
+[Development Principles](DEVELOPMENT-PRINCIPLES.md#-rust-vs-shell-decision-framework)
+document.
+
+**Ingestion/Embedding Architecture:** All ingestion and embedding logic must
+follow the
+[Modular Ingest Refactor Plan](migration/INGEST-MIGRATION-MODULAR-PLAN.md). This
+plan defines the canonical, reusable ingest library and the pattern for
+domain-specific wrappers (e.g., marvelai_ingest.rs). All new pipelines and
+refactors must use this architecture and update documentation accordingly.
 
 ## 🔧 Automated Cleanup Tools First
 
@@ -141,13 +153,10 @@ standards when working with others:
   preferred over long comment threads.
 - **Listen past the first answer**—follow-up questions deepen understanding.
 
-> "Empathy is a muscle: left unused, it atrophies; put to work, it grows."
-> — Jamil Zaki, Stanford
-> "Minds are mirrors to one another."
-> — David Hume
-> "Seeing the world through the eyes of the other, not seeing your world
-> reflected in their eyes."
-> — Carl Rogers
+> "Empathy is a muscle: left unused, it atrophies; put to work, it grows." —
+> Jamil Zaki, Stanford "Minds are mirrors to one another." — David Hume "Seeing
+> the world through the eyes of the other, not seeing your world reflected in
+> their eyes." — Carl Rogers
 
 For more on our collaboration philosophy, see
 [DEVELOPMENT-PRINCIPLES.md](DEVELOPMENT-PRINCIPLES.md).

docs/DETAILED.md

@@ -18,7 +18,7 @@ _exclusively for Bryan Chasko_. It integrates memory, emotion, and modular
 execution across a trusted cohort of entities to restore momentum, anchor
 decisions, and evolve alongside Bryan's ongoing personal and professional arcs.
 
-### Core Components:
+### Core Components
 
 - **🦊 HARALD** – The default entity. Serves as an emotional mirror, decision
   anchor, and continuity manager—especially effective during moments of
@@ -175,7 +175,9 @@ relationships, as well as routines and behavioral patterns.
 With explicit consent, HeraldStack observes and logs sleep patterns,
 conversations (when dual-consent is recorded), random thoughts, and important
 insights. All thoughts are automatically tagged and categorized (e.g., #idea,
-#todo, #relationship, #coding), with full access to raw logs for auditing,
+
+# todo, #relationship, #coding), with full access to raw logs for auditing
+
 tuning, or retraining.
 
 ### Calendar Intelligence
@@ -294,22 +296,54 @@ each component.
   and [GitHub repository](https://github.com/jean-pierreBoth/hnswlib-rs) for
   details and updates.
 
-herald-stack/ # Project root (kebab-case) . ├── ai-entities │ ├── ellow.md │ ├──
-entity-registry.json │ ├── harald.md │ ├── kade-vox.md │ ├── liora.md │ ├──
-myrren.md │ ├── Orin.md │ ├── solan.md │ └── stratia.md ├── docs │ ├──
-architecture-decisions │ │ └── 001-entity-cohort-design.md │ ├── changelog.md │
-├── roadmap.md │ └── weekly-reviews ├── infrastructure │ ├── aws-stack.md │ ├──
-cost-monitoring.md │ ├── deployment-guide.md │ └── pinecone-schemas.md ├──
-integration-guides │ ├── agentic-orchestration.md │ ├──
+HARALD/ # Project root (kebab-case) ├── ai-entities/ # AI entity
+definitions and metadata │ ├── entity-registry.json # Entity registry (all
+entities) │ ├── harald.md # Entity: HARALD │ ├── stratia.md # Entity: Stratia │
+├── myrren.md # Entity: Myrren │ ├── liora.md # Entity: Liora │ ├──
+kade-vox.md # Entity: Kade Vox │ ├── solan.md # Entity: Solan │ ├── ellow.md #
+Entity: Ellow │ ├── orin.md # Entity: Orin │ └── prompts/ # Prompt templates for
+entities ├── config/ # Schemas, ethics, and model configs │ ├── CONFIG.md #
+Config documentation │ ├── ethics/ # Ethical guidelines (e.g.,
+LawsOfRobotics.json) │ │ └── LawsOfRobotics.json │ ├── models/ # Model
+configuration files │ └── schemas/ # Data schemas for validation ├── data/ #
+Vector store registry, ingested data │ ├── vector-stores-registry.json │ └──
+schemas/ # Data schemas (if present) ├── datasets/ # Source datasets for
+ingestion/embedding ├── docs/ # System, migration, and vector search docs │ ├──
+CONTRIBUTING.md # Contribution guidelines │ ├── DETAILED.md # This file
+(detailed docs) │ ├── DEVELOPMENT-PRINCIPLES.md │ ├── naming-conventions.md │
+├── migration/ # Shell-to-Rust migration plans │ │ ├── RECOMMENDED-STRUCTURE.md
+│ │ ├── DIRECTORY-REORGANIZATION.md │ │ └── IMPLEMENTATION-PLAN.md │ └──
+vector-search/ # Vector search and embedding docs │ ├──
+character-based-chunking.md │ ├── ollama-embedding-limits.md │ └──
+jsonl-ingestion.md ├── infrastructure/ # Cloud and deployment infrastructure
+docs │ ├── aws-stack.md │ ├── cost-monitoring.md │ ├── deployment-guide.md │ └──
+pinecone-schemas.md ├── integration-guides/ # Integration docs for external
+APIs/services │ ├── agentic-orchestration.md │ ├──
 amazon-voice-interoperability.md │ ├── anthropic.md │ ├── aws.md │ ├──
 bedrock.md │ ├── cohere.md │ ├── google.md │ ├── griptape.md │ ├── haystack.md │
 ├── hugging-face.md │ ├── microsoft.md │ ├── open-ai.md │ └── pinecone.md ├──
-LawsOfRobotics.json ├── memory-schemas │ ├── conversation-metadata.json │ ├──
+logs/ # Ingestion and embedding logs │ ├── embedding*size_test*_.log │ ├──
+ingest*log*_.log │ └── embedding_api/ # API-specific logs ├── memory-schemas/ #
+Schemas for memory and context │ ├── conversation-metadata.json │ ├──
 emotion-vectors.json │ ├── entity-context.json │ └── narrative-arc.json ├──
-personality-archetypes │ ├── mythological │ │ ├── celtic │ │ ├──
-human-inspired.md │ │ └── norse │ │ ├── Heralds.json │ │ └── heralds.md │ └──
-pop-culture │ ├── bojack-horseman │ │ └── Bojack.json │ ├── literary │ ├──
-marvel │ │ ├── MarvelAIs.json │ │ ├── pop-culture-ai-references.md │ │ └──
-VictorMancha.json │ └── marvel.md ├── README.md └── workflows ├──
-consent-logging.md ├── entity-routing.md ├── task-orchestration.md └──
-weekly-review.md
+personality-archetypes/ # Archetype definitions and docs │ ├── Heralds.json │
+├── heralds.md │ ├── mythological/ │ │ ├── celtic/ │ │ ├── norse/ │ │ └──
+human-inspired.md │ └── pop-culture/ │ ├── bojack-horseman/ │ │ └── Bojack.json
+│ ├── literary/ │ └── marvel/ │ ├── MarvelAIs.json │ ├──
+pop-culture-ai-references.md │ └── VictorMancha.json ├── rust_ingest/ # Rust CLI
+tools for ingestion/embedding │ ├── Cargo.toml │ ├── Cargo.lock │ ├──
+rustREADME.md │ ├── src/ │ └── target/ ├── scripts/ # Shell scripts for
+deployment/validation only │ ├── build_rust_tools.sh │ └── validation/ │ ├──
+check-json.sh │ └── check-rust.sh │ └── deploy/ │ └── deploy.sh ├── src/ # Core
+Rust code (all app logic) │ ├── ingest/ # Ingestion pipeline logic │ │ ├──
+marvelai_ingest.rs # Domain-specific ingest wrapper │ │ ├── ingest.rs # Core
+ingest logic │ │ ├── chunking_utils.rs # Character-based chunking │ │ ├──
+embedding.rs # Embedding API integration │ │ └── ... │ ├── utils/ │ │ ├──
+json_tools/ │ │ │ ├── format_json.rs │ │ │ ├── validate_json_schema.rs │ │ │ └──
+... │ │ └── ... │ └── target/ # Rust build output (release/debug) ├── target/ #
+Rust build output (workspace root) ├── tests/ # Test fixtures and test code │
+├── fixtures/ │ │ └── FIXTURES.md │ ├── ingest_tests.rs # Ingestion/embedding
+tests │ ├── utils_tests.rs # Utility function tests │ └── ... ├── workflows/ #
+CI/CD and automation configs │ ├── rust.yml # Rust build/test workflow │ ├──
+lint.yml # Linting/formatting workflow │ └── ... ├── README.md # Project
+overview, build, and dev standards └── Cargo.toml # Rust workspace config (root)