Skip to content

Commit 0cfbc0c

Browse files
BryanChaskoBryanChasko
committed
Implement system status check and validation utilities
- Added a new status check utility in `src/utils/system/status.rs` to monitor the Ollama service, including memory usage, API connectivity, and model availability. - Introduced logging functions for success, error, info, and warning messages. - Implemented checks for filesystem status and vector store registry. - Created a JSON validation tool in `src/utils/validation/check_json.rs` to validate and format JSON files, with options to build Rust tools and fix formatting issues. - Developed a Markdown formatting utility in `src/utils/validation/format_md.rs` using Prettier, with verbose and check-only options. - Added a naming validation utility in `src/utils/validation/validate_naming.rs` to enforce naming conventions for directories, Rust files, Markdown files, and JSON files, with an option to suggest and apply fixes.
1 parent c48424d commit 0cfbc0c

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

46 files changed

+6169
-1783
lines changed

‎README.md‎

Lines changed: 95 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,38 @@ and modular execution across a trusted cohort of AI entities to restore
1212
momentum, anchor decisions, and evolve alongside Bryan's ongoing personal and
1313
professional journey.
1414

15+
## 🚨 Critical Development Principles
16+
17+
### No New Shell Scripts for Application Logic
18+
19+
**We are migrating FROM shell scripts TO Rust.** Do not create new shell scripts
20+
for any application functionality. Instead:
21+
22+
- **Add features to existing Rust binaries**
23+
- **Update documentation** (README.md, .md files)
24+
- **Add --help flags** to existing tools
25+
26+
#### Exceptions: Scripts That Remain as Shell Scripts
27+
28+
The following should remain as shell scripts:
29+
30+
- **Deployment scripts** that orchestrate external tools (e.g., AWS CLI)
31+
- **check-rust.sh** which must work even when Rust code has issues
32+
- **CI/CD pipeline scripts** for infrastructure tasks
33+
34+
### Use Automated Cleanup Tools First
35+
36+
Before manually fixing linting/formatting issues, run our automated tools:
37+
38+
```bash
39+
./scripts/validation/check-json.sh # Fix JSON issues
40+
./scripts/validation/check-rust.sh # Fix Rust issues
41+
./src/target/release/format_md # Fix Markdown issues
42+
```
43+
44+
**See [CONTRIBUTING.md](docs/CONTRIBUTING.md) for complete development
45+
guidelines.**
46+
1547
## Key Components
1648

1749
- **🦊 HARALD** – Default entity for emotional mirroring, decision anchoring,
@@ -48,16 +80,67 @@ professional journey.
4880
| Storage | Amazon S3 |
4981
| State Tracking | Amazon DynamoDB |
5082
| Semantic Memory | Pinecone |
83+
| Core Logic | Rust |
84+
| Infrastructure | Shell Scripts |
85+
86+
## Build & Deploy
87+
88+
### Building Rust Components
89+
90+
HeraldStack uses Rust for core application logic (data processing, JSON tools,
91+
embedding utilities):
92+
93+
```bash
94+
# Build all Rust binaries
95+
cd src && cargo build --release --features cli
96+
97+
# Available binaries:
98+
# - format_json (JSON formatting and validation)
99+
# - validate_json_schema (Schema validation and generation)
100+
# - ingest_chunked (Character-based data ingestion)
101+
# - embedding_tool (Embedding generation and testing)
102+
# - text_chunker (Text processing utilities)
103+
```
104+
105+
### Deployment
106+
107+
Deployment uses shell scripts for infrastructure orchestration:
108+
109+
```bash
110+
# Quick build (useful for CI/CD)
111+
./scripts/deploy/deploy.sh --build-only
112+
113+
# Deploy to development (default)
114+
./scripts/deploy/deploy.sh
115+
116+
# Deploy to production
117+
./scripts/deploy/deploy.sh prod
118+
119+
# Skip tests for faster deployment
120+
./scripts/deploy/deploy.sh staging --no-tests
121+
```
122+
123+
**Why Shell for Deployment?** Infrastructure scripts remain as shell because
124+
they orchestrate external tools (Docker, AWS CLI) and need rapid iteration -
125+
perfect for shell's ecosystem integration.
126+
127+
**Why Rust for Application Logic?** Data processing, JSON validation, and
128+
embedding tools benefit from Rust's type safety, performance, and error
129+
handling.
51130

52131
## Development Standards
53132

54133
- [Naming Conventions](docs/naming-conventions.md) - Standards for files and
55134
directories
56-
- Automated validation scripts in `scripts/validation`
57-
(see [VALIDATION.md](scripts/validation/VALIDATION.md))
58-
- JSON tooling and vector store registry in `scripts/json-tools`
59-
(see [JSON-TOOLS.md](scripts/json-tools/JSON-TOOLS.md))
60-
- [Project Structure](docs/migration/RECOMMENDED-STRUCTURE.md) - Recommended organization
135+
- **Build & Deploy**: Use `./scripts/deploy/deploy.sh` for deployment (see
136+
[DEPLOY.md](scripts/deploy/DEPLOY.md) for usage)
137+
- **JSON Tools**: Rust-based JSON processing utilities in `src/utils/json_tools`
138+
(see [JSON-TOOLS.md](src/utils/json_tools/JSON-TOOLS.md))
139+
- **Shell vs Rust**: Infrastructure scripts use shell, application logic uses
140+
Rust
141+
- [Project Structure](docs/migration/RECOMMENDED-STRUCTURE.md) - Recommended
142+
organization
143+
- [Migration Documentation](docs/migration/) - Detailed migration information
61144

62145
## Operating Model
63146

@@ -75,15 +158,7 @@ pragmatic execution, and narrative continuity.
75158
- Personality Models
76159
- Workflows
77160
- [JSONL Format for Vector Embedding](docs/vector-search/jsonl-ingestion.md)
78-
- [Directory Structure](docs/migration/RECOMMENDED-STRUCTURE.md) - Organization
79-
standards
80-
- [Implementation Plan](docs/migration/IMPLEMENTATION-PLAN.md) - Migration
81-
strategy
82-
- [Ingest Migration](docs/migration/INGEST-MIGRATION.md) - Rust code migration
83-
notes
84-
- [Directory Reorganization](docs/migration/DIRECTORY-REORGANIZATION.md) - File
85-
reorganization details
86-
- [Migration Documentation](docs/migration/MIGRATION.md) - Migration overview
161+
- [Migration Documentation](docs/migration/) - Shell-to-Rust migration details
87162

88163
## Ethics & Consent
89164

@@ -93,12 +168,12 @@ guidelines including those defined in
93168

94169
## Development Tools
95170

96-
- **Code Quality & Validation**: Scripts for checking and formatting code are
97-
available in `scripts/validation/`. See
98-
[VALIDATION.md](scripts/validation/VALIDATION.md) for usage details.
99-
- **Models**: Model configurations can be found in `config/models/`.
100-
- **Test Data**: Test fixtures are available in `tests/fixtures/`. See
101-
[FIXTURES.md](tests/fixtures/FIXTURES.md) for details.
171+
- **Rust Binaries**: Core application tools built with
172+
`cargo build --release --features cli`
173+
- **Deployment**: Shell-based deployment script at `scripts/deploy/deploy.sh`
174+
- **Models**: Model configurations can be found in `config/models/`
175+
- **Test Data**: Test fixtures are available in `tests/fixtures/` (see
176+
[FIXTURES.md](tests/fixtures/FIXTURES.md) for details)
102177

103178
## Further Information
104179

‎docs/CONTRIBUTING.md‎

Lines changed: 114 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,87 @@
1-
# Contributing to rust_ingest
1+
# Contributing to HARALD
22

3-
This docs/CONTRIBUTING.md document outlines development standards and practices
4-
for the HeraldStack rust_ingest tool.
3+
This document provides practical guidance for contributing to the HARALD
4+
project. For high-level development philosophy and principles, see
5+
[DEVELOPMENT-PRINCIPLES.md](DEVELOPMENT-PRINCIPLES.md).
6+
7+
## Getting Started
8+
9+
### Core Development Rules
10+
11+
Before contributing, please familiarize yourself with our
12+
[Development Principles](DEVELOPMENT-PRINCIPLES.md), which include:
13+
14+
- **NO NEW SHELL SCRIPTS** - We're migrating to Rust for application logic
15+
- **Documentation over code** - Prefer updating docs to writing new scripts
16+
- **Automation over manual work** - Use and extend our automated tools
17+
18+
For the complete migration strategy and decision framework, refer to the
19+
[Development Principles](DEVELOPMENT-PRINCIPLES.md#migration-strategy--guidelines) document.
20+
21+
## 🔧 Automated Cleanup Tools First
22+
23+
Before manually fixing any formatting or linting issues, **ALWAYS** run our
24+
automated tools:
25+
26+
```bash
27+
# Fix JSON formatting and validation issues
28+
./target/release/check_json --fix
29+
30+
# Fix Rust formatting, run clippy, and tests
31+
./scripts/validation/check-rust.sh
32+
33+
# Fix Markdown formatting (line length, spacing, etc.)
34+
./src/target/release/format_md
35+
36+
# Check and optionally fix naming convention problems
37+
./src/target/release/validate_naming --fix --verbose
38+
```
39+
40+
**These tools will automatically resolve most linting and formatting issues.**
41+
Only manually edit files after running the appropriate automated tool.
42+
43+
### 🔎 Tool Paths and Usage
44+
45+
Our development tools are Rust programs that get compiled to executable files
46+
during the build process. When you run `cargo build --release` in a Rust
47+
project, it creates optimized binaries in the `target/release` directory.
48+
49+
In our project, binaries can be found in two different locations:
50+
51+
**Main tools directory**: `./src/target/release/`
52+
53+
- Most Rust tools are here (format_md, validate_naming, etc.)
54+
- Built from the src/Cargo.toml file
55+
- Example: `./src/target/release/format_md`
56+
57+
**Secondary location**: `./target/release/`
58+
59+
- Some newer tools are here (check_json, status, etc.)
60+
- Example: `./target/release/check_json`
61+
62+
You need to check both locations when looking for tools. Always refer to
63+
specific documentation for each tool to find its correct path.
64+
65+
Here are the correct commands to run our most common tools from the project root
66+
directory:
67+
68+
```bash
69+
# Format Markdown files with prettier
70+
./src/target/release/format_md path/to/your/file.md
71+
72+
# Check and fix JSON files (wrapper around format_json)
73+
./target/release/check_json --fix
74+
75+
# Validate naming conventions across the codebase
76+
./src/target/release/validate_naming --fix --verbose
77+
78+
# Check system status (Ollama services, models, etc)
79+
./target/release/status
80+
```
81+
82+
**IMPORTANT:** Always run these tools from the project root directory to ensure
83+
correct path resolution. Running them from other directories may cause file path
84+
errors.
585

686
## Development Environment
787

@@ -40,3 +120,34 @@ for the HeraldStack rust_ingest tool.
40120
}
41121
}
42122
```
123+
124+
## Collaboration Standards
125+
126+
We value empathy, transparency, and practical collaboration. Please follow these
127+
standards when working with others:
128+
129+
- **Begin meetings with an emotion check-in** (e.g., "What excites you? What
130+
worries you?") to build psychological safety and surface risks early.
131+
- **Translate technical details into user language** to ensure features are
132+
user-focused and accessible.
133+
- **Use small courtesies** (like "thank you") to foster equitable knowledge
134+
sharing.
135+
- **Communicate authentically**—speak like a human, not a robot.
136+
- **Practice empathy**: it grows with use and is essential for effective
137+
teamwork.
138+
- **Assume good intent and seek clear impact**—disagree respectfully, document
139+
decisions, and move forward together.
140+
- **Bias toward co-creation**: sketches, sticky notes, and quick prototypes are
141+
preferred over long comment threads.
142+
- **Listen past the first answer**—follow-up questions deepen understanding.
143+
144+
> "Empathy is a muscle: left unused, it atrophies; put to work, it grows."
145+
> — Jamil Zaki, Stanford
146+
> "Minds are mirrors to one another."
147+
> — David Hume
148+
> "Seeing the world through the eyes of the other, not seeing your world
149+
> reflected in their eyes."
150+
> — Carl Rogers
151+
152+
For more on our collaboration philosophy, see
153+
[DEVELOPMENT-PRINCIPLES.md](DEVELOPMENT-PRINCIPLES.md).

‎docs/DEVELOPMENT-PRINCIPLES.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# HARALD Development Principles
2+
3+
This document outlines the core development philosophy and architectural
4+
decisions for the HARALD project. For practical contribution guidance, see
5+
[CONTRIBUTING.md](CONTRIBUTING.md).
6+
7+
## 🚫 Absolutely no new shell scripts
8+
9+
### We give preferential treatment to writing code in Rust in this project
10+
11+
Creating new shell scripts for application logic where Rust can perform equally
12+
is counterproductive.
13+
technical strategy.
14+
15+
### ✅ Do Instead
16+
17+
1. **Update existing documentation** (README.md, relevant .md files)
18+
2. **Add --help flags** to existing Rust tools
19+
3. **Create/update markdown documentation** in appropriate directories
20+
4. **Build functionality into existing Rust binaries**
21+
22+
## � Focus on Documentation and Automation
23+
24+
We prioritize well-maintained documentation and automated tooling over ad-hoc
25+
scripts. When encountering an issue:
26+
27+
1. First, check if our **automated tools** can solve it
28+
([see tools in CONTRIBUTING.md](CONTRIBUTING.md#-automated-cleanup-tools-first))
29+
2. Next, consult or update **documentation** rather than creating a script
30+
3. If needed, extend our **Rust tooling** rather than writing a shell script
31+
32+
## Migration Strategy & Guidelines
33+
34+
We follow a structured approach to migrating functionality from shell scripts to
35+
Rust. This ensures maintainability, type safety, and better error handling.
36+
37+
### What TO Migrate to Rust ✅
38+
39+
- Data processing scripts (text chunking, JSON formatting, validation)
40+
- Application utilities (embedding tools, ingestion scripts)
41+
- Business logic scripts (character processing, data transformation)
42+
- Testing utilities that involve application logic
43+
44+
### What NOT to Migrate to Rust ❌
45+
46+
- Deployment scripts (orchestrating external tools like Docker, AWS CLI)
47+
- Infrastructure scripts (system administration, environment setup)
48+
- CI/CD pipeline scripts (git hooks, build orchestration)
49+
- Simple file operations (backup, cleanup, monitoring)
50+
- Scripts that primarily shell out to other tools
51+
52+
## Decision Framework
53+
54+
When evaluating whether to migrate a script or build new functionality:
55+
56+
**Migrate/Build in Rust if the script:**
57+
58+
- Contains complex application logic
59+
- Processes data or performs calculations
60+
- Benefits from type safety and error handling
61+
- Is part of the core application functionality
62+
63+
**Keep/Create as shell script if it:**
64+
65+
- Primarily orchestrates external tools
66+
- Handles system/infrastructure concerns
67+
- Needs rapid iteration and deployment
68+
- Is better served by shell's ecosystem integration
69+
70+
## Key Resources
71+
72+
- [Full Migration Guidelines](migration/SCRIPT-CLEANUP-PLAN.md)
73+
- [Contributing Guidelines](CONTRIBUTING.md)
74+
- [Script Migration Status](migration/SCRIPT-MIGRATION.md)
75+
- [Build & Deploy Guide](../scripts/deploy/DEPLOY.md)
76+
77+
## Ideas Summary
78+
79+
**Documentation over ad-hoc scripts** - Write clear documentation instead of
80+
creating new scripts
81+
**Rust over shell for application logic** - Use Rust's type safety and error
82+
handling for complex tasks
83+
**Automation over manual processes** - Invest in automated tooling that can
84+
be used by everyone
85+
**Reuse over recreation** - Extend existing tools rather than creating new
86+
ones
87+
**Testing over hoping** - Ensure all code has appropriate tests
88+
89+
Remember: **Documentation over shell scripts, automation over manual fixes.**

0 commit comments

Comments
 (0)