Quantum trajectory plot

[jeremymanning]

Let's take a book with long, engaging chapters (e.g., hyperion_djvu.txt). Divide it into chapters/sections (Prologue, Chapter 1, Chapter 2, ..., Chapter 6, Epilogue).

For each chapter (can be parallelized in different threads):

• Use TinyLlama-1.1b to embed each token. Create a number-of-tokens by number-of-embedding-dimensions matrix (for this chapter).
• For each of 100 "particles":
  • Project it forward by iteratively predicting next tokens (until we get to a stop token). If we run out of context, just slide the window forward to include only the last <length-of-context-window - 1> tokens.
  • Store the embeddings of the predicted token sequences (in a length number-of-particles list of number-of-tokens-for-that-particle by number-of-embedding-dimensions matrices)

Save everything as a pkl file (one per chapter).

Then, once all chapters' pkl files are saved out:

• Concatenate all of the embedded tokens into an enormous total-number-of-tokens by number-of-embedding-dimensions matrices (across all chapters and particles)
• Project into 2D using UMAP
• Split the concatenated matrix back out into separate chapters/particles
• Save out a pkl file with the 2D projections

Then make a plot like this (one panel per chapter):

The blue lines are chapter trajectories. The red lines (projecting forward from the end of each chapter) are particles' predictions. The blue dots are the starts/ends of each chapter.

[jeremymanning]

Implementation Plan for Quantum Trajectory Plot

Overview

This feature will process book chapters, generate particle trajectories using TinyLlama-1.1b embeddings, project them to 2D using UMAP, and create quantum trajectory visualizations showing both actual chapter paths (blue) and predicted particle paths (red).

New Components to Implement

1. BookProcessor (quantum_conversations/book_processor.py)

• Purpose: Handle text file parsing and chapter extraction
• Key Methods:
  • load_book(file_path): Load and validate text file (real file I/O)
  • extract_chapters(): Parse chapters based on patterns (Prologue, Chapter N, Epilogue)
  • chunk_text(max_tokens): Split text into context-window-sized chunks if needed
  • Real Integration: Direct file system access, no mocks

2. EmbeddingGenerator (quantum_conversations/embeddings.py)

• Purpose: Generate embeddings using TinyLlama model
• Key Methods:
  • get_token_embeddings(tokens): Extract embeddings from model's embedding layer
  • process_chapter_embeddings(chapter_text): Generate embeddings for all tokens
  • generate_particle_embeddings(particles): Extract embeddings for predicted sequences
  • Real Integration: Direct model access via HuggingFace transformers

3. ParticleTrajectoryGenerator (quantum_conversations/trajectory_generator.py)

• Purpose: Generate particle trajectories with sliding window support
• Key Methods:
  • generate_trajectories(chapter_text, n_particles=100): Main generation method
  • handle_context_overflow(tokens, context_length): Sliding window implementation
  • parallel_process_chapters(chapters, n_workers): Thread-based parallel processing
  • Real Integration: Uses existing ParticleFilter with real model inference

4. UMAPProjector (quantum_conversations/umap_projector.py)

• Purpose: Project high-dimensional embeddings to 2D
• Key Methods:
  • fit_transform(embeddings): UMAP projection with configurable params
  • batch_project(embedding_chunks): Handle large embedding matrices
  • save_projection_data(filepath): Persist projection results
  • Real Integration: Direct UMAP-learn library usage

5. QuantumTrajectoryVisualizer (quantum_conversations/quantum_visualizer.py)

• Purpose: Create the final visualization
• Key Methods:
  • plot_trajectories(chapter_data, particle_data): Main plotting function
  • add_chapter_path(ax, points, color='blue'): Plot actual chapter trajectories
  • add_particle_paths(ax, points, color='red'): Plot predicted particle paths
  • add_start_end_markers(ax, points): Add blue dots for chapter boundaries
  • Real Integration: Direct matplotlib rendering

Implementation Steps

1. Add UMAP dependency to requirements.txt:

   • umap-learn>=0.5.0

2. Create main pipeline script (code/quantum_trajectory_pipeline.py):

   • Load book file
   • Extract chapters
   • Process each chapter in parallel:
     • Generate token embeddings
     • Generate 100 particle trajectories
     • Handle context window sliding
   • Save intermediate pkl files per chapter
   • Concatenate all embeddings
   • Project to 2D with UMAP
   • Create visualization

Testing Strategy (ALL REAL, NO MOCKS)

Unit Tests (test_quantum_trajectory.py)

1. test_book_processor_real_file:

   • Download sample text from Project Gutenberg
   • Test chapter extraction on real text
   • Verify chunk sizes and boundaries

2. test_embedding_generation_real_model:

   • Use TinyLlama/TinyLlama-1.1B-Chat-v1.0
   • Generate embeddings for sample text
   • Verify embedding dimensions (2048 for TinyLlama)
   • Check embedding values are reasonable

3. test_particle_trajectory_real_generation:

   • Generate actual trajectories with model
   • Verify sliding window works correctly
   • Check particle divergence over time

4. test_umap_projection_real_data:

   • Generate real embeddings
   • Project with UMAP
   • Verify 2D output shape
   • Check projection preserves local structure

5. test_visualization_real_output:

   • Create actual plot with real data
   • Save as PNG
   • Manually verify visual output
   • Check all elements present (paths, dots, labels)

Integration Tests

1. test_full_pipeline_small_text:

   • Process a small sample text (3-4 chapters)
   • Generate trajectories with 10 particles (faster)
   • Create full visualization
   • Verify all intermediate files created

2. test_parallel_processing:

   • Process multiple chapters in parallel
   • Verify thread safety
   • Check memory usage stays reasonable

3. test_context_window_sliding:

   • Use text longer than model context
   • Verify sliding window maintains continuity
   • Check no tokens are lost

Edge Cases

1. test_empty_chapters: Handle books with empty sections
2. test_single_chapter: Process book with only one chapter
3. test_very_long_chapter: Handle chapters exceeding memory limits
4. test_special_characters: Unicode, formatting, etc.

Performance Considerations

• Parallel Processing: Use ThreadPoolExecutor for chapters
• Batch Processing: Process embeddings in batches to manage memory
• Incremental Saving: Save chapter pkl files as they complete
• Memory Management: Clear GPU cache between chapters

Dependencies & Resources

• TinyLlama model from HuggingFace (auto-downloads ~2.2GB)
• UMAP-learn library (pip installable)
• Sample text files from Project Gutenberg for testing
• ~8GB RAM for processing full book with 100 particles
• GPU recommended but not required (works on CPU)

Deliverables

1. All new module implementations
2. Comprehensive test suite (no mocks!)
3. Example pipeline script
4. Sample visualization output
5. Documentation updates

This plan ensures all functionality uses real integrations with actual models, file I/O, and computations - no simulations or mocks anywhere in the implementation or testing.

[jeremymanning]

Updated Implementation Plan - Adding Hyperion Exploration Notebook

Additional Deliverable: Sample Notebook

6. Hyperion Exploration Notebook (code/notebooks/hyperion_quantum_trajectories.ipynb)

Purpose: Demonstrate the complete quantum trajectory pipeline using Dan Simmons' Hyperion

Key Requirements:

• Use ONLY the new toolbox functions - minimize code written directly in notebook
• Include full bibliographic citation for the text
• Focus on clear explanations of each processing step
• NO conclusions or assumptions about what results will look like
• Must be manually tested to ensure visual quality

Notebook Structure:

1. Setup and Citation

   from quantum_conversations import (
       BookProcessor, 
       EmbeddingGenerator,
       ParticleTrajectoryGenerator,
       UMAPProjector,
       QuantumTrajectoryVisualizer
   )
   
   # Citation:
   # Simmons, Dan. Hyperion. New York: Doubleday, 1989.
   # Text sourced from Internet Archive: 
   # https://archive.org/stream/hyperion_Dan_Simmons_202109/hyperion_djvu.txt
   # Accessed: September 2025

2. Load and Process Book

   processor = BookProcessor()
   book = processor.load_book('data/raw/hyperion_djvu.txt')
   chapters = processor.extract_chapters()  # Extracts PROLOGUE, Tales, EPILOGUE

3. Generate Embeddings and Trajectories

   generator = ParticleTrajectoryGenerator(
       model_name="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
       n_particles=100
   )
   
   # Process chapters in parallel
   chapter_data = generator.parallel_process_chapters(chapters)

4. Project to 2D Space

   projector = UMAPProjector(
       n_neighbors=15,
       min_dist=0.1,
       random_state=42  # Will be optimized during testing
   )
   
   projected_data = projector.fit_transform(chapter_data)

5. Create Visualization

   visualizer = QuantumTrajectoryVisualizer()
   fig = visualizer.plot_trajectories(
       projected_data,
       smooth_factor=0.5,  # Will be tuned during testing
       figsize=(20, 12)
   )

Visual Quality Assurance Checklist:

• No excessive self-intersections in trajectories
• Smooth curves (apply smoothing if needed)
• Clean, uncluttered appearance
• No cut-off figure components
• No overlapping text labels
• Clear distinction between chapter paths (blue) and particle predictions (red)
• Visible start/end markers (blue dots)

Testing Protocol:

1. Run notebook with different UMAP parameters:
   • random_state: [42, 123, 456, 789]
   • n_neighbors: [10, 15, 30, 50]
   • min_dist: [0.05, 0.1, 0.2, 0.3]
2. Apply trajectory smoothing if needed:
   • Gaussian smoothing
   • Savitzky-Golay filter
   • Spline interpolation
3. Adjust visual parameters:
   • Alpha values for trajectory transparency
   • Line widths
   • Figure size and DPI
4. Save best parameter combinations in notebook

Data Preparation

The Hyperion text file (hyperion_djvu.txt) has been downloaded to data/raw/ directory from the Internet Archive. The file contains:

• PROLOGUE (line 1)
• Multiple character tales (THE PRIEST'S TALE, THE SOLDIER'S TALE, etc.)
• EPILOGUE (line 24415)
• Total size: ~1MB of text

Updated Timeline

1. Phase 1: Implement core modules (BookProcessor, EmbeddingGenerator, etc.)
2. Phase 2: Create and test pipeline script
3. Phase 3: Develop comprehensive test suite
4. Phase 4: Create Hyperion notebook
5. Phase 5: Manual quality assurance of visualizations
6. Phase 6: Optimize parameters based on visual inspection
7. Phase 7: Final documentation

Quality Metrics for Success

• Notebook runs end-to-end without errors
• All processing uses toolbox functions (minimal inline code)
• Visualizations pass all quality checks
• Processing completes in reasonable time (<30 min for full book)
• Memory usage stays under 16GB
• Results are reproducible with fixed seeds

This ensures the notebook serves as both a demonstration and validation of the entire quantum trajectory pipeline.