Implement test infrastructure for faster development

[BryanChasko]

Status:

• Started planning test strategy
• Identified key areas needing test coverage

Tasks:

1. Create unit tests for core functionality
2. Add mock tests that don't require actual file system or API calls
3. Implement test helpers to create small indices
4. Add a --sample flag to CLI for processing limited data sets
5. Create a separate test data directory with minimal content for quick testing

[BryanChasko]

Update: Progress made on test infrastructure:

• Basic unit tests for core functions added
• Integration test framework with conditional compilation implemented
• Test dependencies added to Cargo.toml

Still needed:

• Async testing improvements to replace placeholder tests
• Mock-based testing for external API calls
• Performance testing with criterion benchmarks
• CI integration for automated testing

Next steps detailed in comment

[BryanChasko]

HARALD git:(main) ✗ ./scripts/test_basic_embedding.sh

🧪 HARALD EMBEDDING TEST - SINGLE CHARACTER
🕒 Started at 2025-07-21 14:42:12

✅ Ollama service is running
📄 Testing with character: Vision
🔄 Simple test: Generating embedding for character name only
Request: {"model":"harald-phi4","prompt":"Vision"}
🔄 Sending request to Ollama API...
✅ Simple test passed - Embedding vectors received!
Vector dimensions: 3072
🔄 Full test: Generating embedding for character details
Request content size: 538 bytes
🔄 Sending full character data to Ollama API...
❌ Full test failed - No embedding vectors in response
{"error":"json: cannot unmarshal object into Go struct field EmbeddingRequest.prompt of type string"}%
➜ HARALD git:(main) ✗ ./scripts/test_basic_embedding.sh

🧪 HARALD EMBEDDING TEST - SINGLE CHARACTER
🕒 Started at 2025-07-21 14:45:44

✅ Ollama service is running
📄 Testing with character: Vision
🔄 Simple test: Generating embedding for character name only
Request: {"model":"harald-phi4","prompt":"Vision"}
🔄 Sending request to Ollama API...
✅ Simple test passed - Embedding vectors received!
Vector dimensions: 3072
🔄 Full test: Generating embedding for character details
Request content size: 539 bytes
Character data converted to text format
🔄 Sending full character data to Ollama API...
❌ Full test failed - No embedding vectors in response
{"error":"invalid character 'A' after object key:value pair"}%
➜ HARALD git:(main) ✗ ./scripts/test_basic_embedding.sh

🧪 HARALD EMBEDDING TEST - SINGLE CHARACTER
🕒 Started at 2025-07-21 14:46:15

✅ Ollama service is running
📄 Testing with character: Vision
🔄 Simple test: Generating embedding for character name only
Request: {"model":"harald-phi4","prompt":"Vision"}
🔄 Sending request to Ollama API...
✅ Simple test passed - Embedding vectors received!
Vector dimensions: 3072
🔄 Full test: Generating embedding for character details
Request content size: 376 bytes
Character data converted to text format
🔄 Sending full character data to Ollama API...
➜ HARALD git:(main) ✗ ./scripts/test_basic_embedding.sh

🧪 HARALD EMBEDDING TEST - SINGLE CHARACTER
🕒 Started at 2025-07-21 14:48:55

✅ Ollama service is running
📄 Testing with character: Vision
🔄 Simple test: Generating embedding for character name only
Request: {"model":"harald-phi4","prompt":"Vision"}
🔄 Sending request to Ollama API...
✅ Simple test passed - Embedding vectors received!
Vector dimensions: 3072
🔄 Full test: Generating embedding for character details
Request content size: 376 bytes
Character data converted to text format
🔄 Sending full character data to Ollama API...
⏱️ Using timeout: 90 seconds
➜ HARALD git:(main) ✗ chmod +x /Users/bryanchasko/Code/HARALD/sc
ripts/test_embedding_size.sh
➜ HARALD git:(main) ✗ ./scripts/test_embedding_size.sh

🔍 HARALD EMBEDDING SIZE TEST
🕒 Started at 2025-07-21 14:55:27

✅ Ollama service is running
📝 Logging to: ./logs/embedding_size_test_20250721_145527.log
🔄 Testing with increasing text sizes to find the limit...

🧪 Testing embedding with text size: 10 chars (timeout: 15s)
✅ Success - Embedding generated in 7s
Vector dimensions: 3072

🧪 Testing embedding with text size: 50 chars (timeout: 15s)
✅ Success - Embedding generated in 4s
Vector dimensions: 3072

🧪 Testing embedding with text size: 100 chars (timeout: 15s)
✅ Success - Embedding generated in 7s
Vector dimensions: 3072

🧪 Testing embedding with text size: 200 chars (timeout: 30s)
✅ Success - Embedding generated in 22s
Vector dimensions: 3072

🧪 Testing embedding with text size: 300 chars (timeout: 30s)
✅ Success - Embedding generated in 29s
Vector dimensions: 3072

🧪 Testing embedding with text size: 400 chars (timeout: 45s)
⚠️ Request timed out after 45s
Text size 400 is too large for the embedding API
🛑 Stopping tests at size 400

✅ HARALD EMBEDDING SIZE TEST COMPLETE
📊 Results:
Maximum successful text size: 300 characters
Recommendation: Keep chunks under 300 characters for reliable embedding
🔍 Logs saved to: ./logs/embedding_size_test_20250721_145527.log.*

🕒 Finished at 2025-07-21 14:57:32
➜ HARALD git:(main) ✗ chmod +x /Users/bryanchasko/Code/HARALD/sc
ripts/ingest_chunked.sh
➜ HARALD git:(main) ✗ ./scripts/ingest_chunked.sh

🚀 HARALD OPTIMIZED INGEST PROCESS
🔍 Processing MarvelAIs.json with size-optimized chunks
🕒 Started at 2025-07-21 14:58:47

✅ Ollama service is running
📝 Logging to: ./logs/ingest_log_20250721_145847.log
⏱️ 2025-07-21 14:58:47 | Starting to process: /Users/bryanchasko/Code/HARALD/test_single_character.json
📄 Processing single character test file
🧪 Testing API with minimal request...
🔄 Generating embedding for text (4 chars)
📦 Request size: 40 bytes
⏱️ Using timeout: 30s
✅ Success - Embedding generated in 6s
Vector dimensions: 3072
✅ API working correctly
📄 Processing character: Vision
⏱️ 2025-07-21 14:58:53 | Extracting character data into separate chunks
./scripts/ingest_chunked.sh: line 136: 7: command not found
🔄 Processing character data chunks...

🔄 Processing chunk 1/7: Name: Vision...
🔄 Generating embedding for text (12 chars)
📦 Request size: 48 bytes
⏱️ Using timeout: 30s
✅ Success - Embedding generated in 9s
Vector dimensions: 3072

🔄 Processing chunk 2/7: First Appearance: Avengers (19...
🔄 Generating embedding for text (37 chars)
📦 Request size: 73 bytes
⏱️ Using timeout: 30s
✅ Success - Embedding generated in 23s
Vector dimensions: 3072

🔄 Processing chunk 3/7: Affiliations: Avengers...
🔄 Generating embedding for text (22 chars)
📦 Request size: 58 bytes
⏱️ Using timeout: 30s
✅ Success - Embedding generated in 11s
Vector dimensions: 3072

🔄 Processing chunk 4/7: Attributes: Density control, I...
🔄 Generating embedding for text (83 chars)
📦 Request size: 119 bytes
⏱️ Using timeout: 30s
✅ Success - Embedding generated in 23s
Vector dimensions: 3072

🔄 Processing chunk 5/7: Themes: Redemption, Self-deter...
🔄 Generating embedding for text (65 chars)
📦 Request size: 101 bytes
⏱️ Using timeout: 30s
✅ Success - Embedding generated in 21s
Vector dimensions: 3072

🔄 Processing chunk 6/7: Traits: Self-aware, chooses go...
🔄 Generating embedding for text (74 chars)
📦 Request size: 110 bytes
⏱️ Using timeout: 30s
✅ Success - Embedding generated in 23s
Vector dimensions: 3072

🔄 Processing chunk 7/7: AI Alignment: Role Model...
🔄 Generating embedding for text (24 chars)
📦 Request size: 60 bytes
⏱️ Using timeout: 30s
✅ Success - Embedding generated in 11s
Vector dimensions: 3072

✅ HARALD CHUNKED INGEST COMPLETE
📊 SUMMARY:
Character: Vision
Successful chunks: 7
Failed chunks: 0
Total chunks: 7
Success rate: 100%
Total execution time: 2m 21s

📋 Recommendations:

1. Keep chunks under 250 characters
2. Process character data as separate attributes
3. Implement retry logic for failed embeddings
4. Consider increasing API timeouts based on text size
   ==================================================
   🕒 Finished at 2025-07-21 15:01:08