Skip to content

Commit 85c9bcf

Browse files
author
Bryan Howard
committed
Create Story 3.3: Native Object Passing System
Epic 3 - Single Process Execution Architecture: Next Story ## Story Summary Story 3.3: Native Object Passing System - Direct Python object references between nodes without serialization, supporting ML frameworks with zero-copy performance. ## Story Details - ID: 3.3 - Type: Feature - Priority: High - Status: Draft (ready for development) - Dependencies: Story 3.2 (Complete) - Estimated Effort: L ## Key Features - Direct object references with no copying/serialization - PyTorch tensors, NumPy arrays, Pandas DataFrames support - Memory-mapped sharing for RAM-resident objects - Reference counting system for automatic cleanup - Complete elimination of JSON fallbacks ## Implementation Scope - 6 major implementation tasks with 24 detailed subtasks - Object reference system enhancement - ML framework integration (PyTorch, TensorFlow, JAX) - Memory management and performance optimization - Comprehensive testing across all supported data types ## Technical Context Builds on Story 3.2's single-process architecture foundation. Includes extensive Dev Notes with architecture integration points, performance considerations, and technical constraints. Story validated with 9/10 clarity score and READY status. Prepared by Technical Scrum Master for Developer Agent implementation.
1 parent e204d51 commit 85c9bcf

File tree

1 file changed

+172
-0
lines changed

1 file changed

+172
-0
lines changed

‎docs/stories/3.3.story.md‎

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
# Story 3.3: Native Object Passing System
2+
3+
## Status
4+
Draft
5+
6+
## Story
7+
8+
**As a** user,
9+
**I want** to pass Python objects directly between nodes without any serialization,
10+
**so that** I can work with large tensors and DataFrames at maximum performance.
11+
12+
## Acceptance Criteria
13+
14+
1. Direct Python object references passed between nodes (no copying)
15+
2. Support for all Python types including PyTorch tensors, NumPy arrays, Pandas DataFrames
16+
3. Memory-mapped sharing for objects already in RAM
17+
4. Reference counting system for automatic cleanup
18+
5. No type restrictions or JSON fallbacks ever
19+
20+
## Tasks / Subtasks
21+
22+
- [ ] **Task 1**: Implement comprehensive object reference system (AC: 1)
23+
- [ ] Subtask 1.1: Enhance pin_values dictionary to handle all Python object types
24+
- [ ] Subtask 1.2: Remove any remaining JSON serialization fallbacks
25+
- [ ] Subtask 1.3: Implement direct object reference passing between nodes
26+
- [ ] Subtask 1.4: Add object type validation and error handling
27+
28+
- [ ] **Task 2**: Add advanced data science framework support (AC: 2)
29+
- [ ] Subtask 2.1: Add PyTorch tensor support with device management
30+
- [ ] Subtask 2.2: Add NumPy array support with dtype preservation
31+
- [ ] Subtask 2.3: Add Pandas DataFrame support with index/column preservation
32+
- [ ] Subtask 2.4: Add support for complex nested objects and custom classes
33+
34+
- [ ] **Task 3**: Implement memory-mapped sharing system (AC: 3)
35+
- [ ] Subtask 3.1: Add memory mapping detection for large objects
36+
- [ ] Subtask 3.2: Implement zero-copy sharing for compatible objects
37+
- [ ] Subtask 3.3: Add shared memory buffer management
38+
- [ ] Subtask 3.4: Optimize memory access patterns for large datasets
39+
40+
- [ ] **Task 4**: Create reference counting and cleanup system (AC: 4)
41+
- [ ] Subtask 4.1: Implement object reference tracking using weakref
42+
- [ ] Subtask 4.2: Add automatic garbage collection for unreferenced objects
43+
- [ ] Subtask 4.3: Create memory cleanup policies for long-running sessions
44+
- [ ] Subtask 4.4: Add GPU memory cleanup for ML framework objects
45+
46+
- [ ] **Task 5**: Eliminate all type restrictions and JSON fallbacks (AC: 5)
47+
- [ ] Subtask 5.1: Remove any remaining JSON conversion code paths
48+
- [ ] Subtask 5.2: Add universal object support without type checking
49+
- [ ] Subtask 5.3: Implement robust error handling for unsupported operations
50+
- [ ] Subtask 5.4: Add validation to prevent JSON fallback scenarios
51+
52+
- [ ] **Task 6**: Testing and validation (AC: 1-5)
53+
- [ ] Subtask 6.1: Create unit tests for direct object passing
54+
- [ ] Subtask 6.2: Create integration tests for ML framework objects
55+
- [ ] Subtask 6.3: Add memory leak detection tests
56+
- [ ] Subtask 6.4: Create performance benchmarks comparing copy vs reference passing
57+
58+
## Dev Notes
59+
60+
### Previous Story Insights
61+
Key learnings from Story 3.2 (Single Shared Python Interpreter):
62+
- SingleProcessExecutor successfully replaced subprocess isolation with direct execution
63+
- Pin_values dictionary now stores actual Python objects (foundation for 3.3)
64+
- Direct function calls are working in shared interpreter
65+
- Persistent namespace enables import and variable sharing between executions
66+
- Performance improvements of 100-1000x achieved by eliminating subprocess overhead
67+
- Security model changed from process isolation to direct execution with error handling
68+
- Memory management and reference counting infrastructure needs identified
69+
[Source: docs/stories/3.2.story.md#dev-agent-record]
70+
71+
### Current Object Passing Foundation
72+
The SingleProcessExecutor in Story 3.2 established the basic infrastructure:
73+
- **Object Store**: `self.object_store: Dict[Any, Any] = {}` for direct object storage
74+
- **Direct References**: Pin values store actual Python objects, not JSON strings
75+
- **Namespace Persistence**: All imports and variables remain loaded between executions
76+
- **No Serialization**: JSON serialization/deserialization completely removed
77+
- **Performance**: Direct object references eliminate copy overhead
78+
[Source: src/execution/single_process_executor.py lines 37-38, 167-178]
79+
80+
### Technical Implementation Details
81+
82+
#### Current Architecture Integration Points
83+
- **GraphExecutor**: `src/execution/graph_executor.py` - Main execution orchestrator using SingleProcessExecutor
84+
- **SingleProcessExecutor**: `src/execution/single_process_executor.py` - Direct function call execution with object storage
85+
- **Pin Values**: Dictionary mapping Pin objects to actual Python objects (no JSON)
86+
- **Object Store**: Direct object reference storage in SingleProcessExecutor
87+
- **Memory Management**: Basic cleanup with gc.collect() and GPU cache clearing
88+
[Source: src/execution/graph_executor.py lines 47-48, src/execution/single_process_executor.py lines 160-229]
89+
90+
#### File Locations & Structure
91+
- **Main Enhancement Target**: `src/execution/single_process_executor.py` - Enhance object reference and memory management
92+
- **GraphExecutor Updates**: `src/execution/graph_executor.py` - Update pin_values handling for enhanced object passing
93+
- **New Testing**: `tests/test_object_passing.py` (new) - Comprehensive object reference testing
94+
- **Integration Testing**: Extend `tests/test_execution_engine.py` for advanced object types
95+
[Source: docs/architecture/source-tree.md#execution-system]
96+
97+
#### Data Types and Framework Support
98+
- **Basic Types**: All Python built-in types (int, float, str, list, dict, set, tuple)
99+
- **NumPy Support**: Arrays with dtype, shape, and memory layout preservation
100+
- **PyTorch Support**: Tensors with device (CPU/GPU), dtype, and gradient tracking
101+
- **Pandas Support**: DataFrames, Series with indexes, dtypes, and metadata
102+
- **Custom Objects**: User-defined classes, nested structures, complex hierarchies
103+
- **Memory Objects**: Memory-mapped files, shared arrays, zero-copy buffers
104+
[Source: docs/prd.md#story-33-native-object-passing-system, src/execution/single_process_executor.py lines 66-74]
105+
106+
#### Memory Management Architecture
107+
- **Reference Counting**: WeakValueDictionary for automatic reference cleanup
108+
- **GPU Memory**: PyTorch CUDA cache clearing for GPU memory management
109+
- **Garbage Collection**: Explicit gc.collect() calls for Python object cleanup
110+
- **Memory Mapping**: Zero-copy sharing for large objects already in RAM
111+
- **Cleanup Policies**: Automatic cleanup for long-running sessions
112+
[Source: src/execution/single_process_executor.py lines 44, 180-203]
113+
114+
#### Performance Considerations
115+
- **Zero Copy**: Direct object references eliminate copy overhead completely
116+
- **Memory Sharing**: Objects shared by reference, not duplicated
117+
- **GPU Tensors**: Direct GPU memory sharing without CPU roundtrips
118+
- **Large DataFrames**: Memory-mapped sharing for datasets larger than RAM
119+
- **Cleanup Overhead**: Minimal reference counting with weak references
120+
[Source: docs/prd.md#epic-3-single-process-execution-architecture]
121+
122+
#### Security and Error Handling
123+
- **No Sandboxing**: All code executes in main process (security trade-off from 3.2)
124+
- **Error Isolation**: Comprehensive exception handling prevents main application crashes
125+
- **Memory Safety**: Reference counting prevents memory leaks from unreferenced objects
126+
- **Type Safety**: No type restrictions - support any Python object type
127+
- **Validation**: Robust error handling for edge cases and unsupported operations
128+
[Source: src/execution/single_process_executor.py lines 130-141, docs/stories/3.2.story.md#security-review]
129+
130+
### Testing
131+
132+
#### Testing Requirements
133+
- **Unit Tests**: `tests/test_object_passing.py` - Direct object reference testing
134+
- **Framework Tests**: Test PyTorch tensors, NumPy arrays, Pandas DataFrames
135+
- **Memory Tests**: Reference counting, garbage collection, memory leak detection
136+
- **Performance Tests**: Benchmark object passing performance vs copying
137+
- **Integration Tests**: End-to-end object passing through complex graphs
138+
[Source: docs/architecture/coding-standards.md#testing-standards]
139+
140+
#### Testing Framework and Patterns
141+
- **Framework**: Python unittest (established pattern in project)
142+
- **Test Runner**: Custom PySide6 GUI test runner for interactive testing
143+
- **Timeout**: All tests must complete within 10 seconds maximum
144+
- **No Mocking**: Use real objects for Qt components, avoid Mock with PySide6
145+
- **Test Naming**: Follow `test_{behavior}_when_{condition}` pattern
146+
[Source: docs/architecture/coding-standards.md#pyside6qt-testing-requirements, CLAUDE.md#testing]
147+
148+
#### Specific Testing Requirements for Story 3.3
149+
- Test direct object reference passing between nodes (no copying)
150+
- Test PyTorch tensor passing with device and dtype preservation
151+
- Test NumPy array passing with shape and memory layout preservation
152+
- Test Pandas DataFrame passing with indexes and metadata preservation
153+
- Test custom object and nested structure passing
154+
- Test memory leak detection with long-running object passing scenarios
155+
- Test reference counting cleanup when objects are no longer referenced
156+
- Test GPU memory management for PyTorch tensors
157+
- Test zero-copy sharing performance improvements
158+
- Test error handling for edge cases and type conflicts
159+
160+
### Technical Constraints
161+
- **Windows Platform**: Use Windows-compatible commands and paths, no Unicode characters
162+
- **PySide6 Framework**: Maintain compatibility with existing Qt-based architecture
163+
- **No JSON Fallbacks**: Eliminate all JSON serialization completely (AC: 5)
164+
- **Memory Safety**: Prevent memory leaks while maintaining performance
165+
- **Backward Compatibility**: Existing graphs must work without modification
166+
[Source: docs/architecture/coding-standards.md#prohibited-practices, CLAUDE.md]
167+
168+
## Change Log
169+
170+
| Date | Version | Description | Author |
171+
| ---------- | ------- | --------------------------- | --------- |
172+
| 2025-01-20 | 1.0 | Initial story creation based on PRD Epic 3 | Bob (SM) |

0 commit comments

Comments
 (0)