Parallel streaming: Support concurrent streams from Amp Server, partitioned by block_range (#10)

* streaming: Setup primitives for parallel streaming

* loader: Wire up parallel streaming

* docs: Parallel streaming implementation plan

* tests: Unit and integration tests for parallel streaming

- Integration tests require an Amp server

* docs, README: Document parallel streams usage

* postgresql_loader: Use Numeric type for Arrow UINT64 columns

* parallel streaming: various improvements

- Configurable reorg buffer
- Create table ahead of spinning up parallel workers to ensure it's ready for all of them and avoid complexity of thread locking
- SQL variables for string replacement
- Better docs, including limitations

Author: fordN

CLAUDE.md

@@ -72,8 +72,8 @@ When implementing new loaders:
 5. Follow existing patterns from PostgreSQL and Redis loaders
 
 ### Testing Strategy
-- **Unit tests**: Mock external dependencies, test business logic
-- **Integration tests**: Use testcontainers for real database testing
+- **Unit tests**: Test pure logic and data structures WITHOUT mocking. Unit tests should be simple, fast, and test isolated components (dataclasses, utility functions, partitioning logic, etc.). Do NOT add tests that require mocking to `tests/unit/`.
+- **Integration tests**: Use testcontainers for real database testing. Tests that require external dependencies (databases, Flight SQL server, etc.) belong in `tests/integration/`.
 - **Performance tests**: Benchmark data loading operations
 - Tests can be filtered using pytest markers (e.g., `-m unit` for unit tests only)
 

Makefile

@@ -44,6 +44,11 @@ test-lmdb:
 	@echo "⚡ Running LMDB tests..."
 	$(PYTHON) pytest tests/ -m "lmdb" -v --log-cli-level=ERROR
 
+# Parallel streaming integration tests
+test-parallel-streaming:
+	@echo "⚡ Running parallel streaming integration tests..."
+	$(PYTHON) pytest tests/integration/test_parallel_streaming.py -v -s --log-cli-level=INFO
+
 # Performance tests
 test-performance:
 	@echo "🏇 Running performance tests..."
@@ -109,16 +114,17 @@ clean:
 # Show available commands
 help:
 	@echo "Available commands:"
-	@echo "  make setup            - Setup development environment"
-	@echo "  make test-unit        - Run unit tests (fast)"
-	@echo "  make test-integration - Run integration tests"
-	@echo "  make test-all         - Run all tests with coverage"
-	@echo "  make test-postgresql  - Run PostgreSQL tests"
-	@echo "  make test-redis       - Run Redis tests"
-	@echo "  make test-snowflake   - Run Snowflake tests"
-	@echo "  make test-performance - Run performance tests"
-	@echo "  make lint             - Lint code with ruff"
-	@echo "  make format           - Format code with ruff"
-	@echo "  make test-setup       - Start test databases"
-	@echo "  make test-cleanup     - Stop test databases"
-	@echo "  make clean            - Clean test artifacts"
+	@echo "  make setup                    - Setup development environment"
+	@echo "  make test-unit                - Run unit tests (fast)"
+	@echo "  make test-integration         - Run integration tests"
+	@echo "  make test-parallel-streaming  - Run parallel streaming integration tests"
+	@echo "  make test-all                 - Run all tests with coverage"
+	@echo "  make test-postgresql          - Run PostgreSQL tests"
+	@echo "  make test-redis               - Run Redis tests"
+	@echo "  make test-snowflake           - Run Snowflake tests"
+	@echo "  make test-performance         - Run performance tests"
+	@echo "  make lint                     - Lint code with ruff"
+	@echo "  make format                   - Format code with ruff"
+	@echo "  make test-setup               - Start test databases"
+	@echo "  make test-cleanup             - Stop test databases"
+	@echo "  make clean                    - Clean test artifacts"

README.md

@@ -41,6 +41,14 @@ and the `amp` package. For example, you can run the `execute_query` app with the
 uv run apps/execute_query.py
 ```
 
+## Documentation
+
+### Features
+- **[Parallel Streaming Usage Guide](docs/parallel_streaming_usage.md)** - User guide for high-throughput parallel data loading
+- **[Parallel Streaming Design](docs/parallel_streaming.md)** - Technical design documentation for parallel streaming architecture
+- **[Reorganization Handling](docs/reorg_handling.md)** - Guide for handling blockchain reorganizations
+- **[Implementing Data Loaders](docs/implementing_data_loaders.md)** - Guide for creating custom data loaders
+
 # Self-hosted Amp server
 
 In order to operate a local Amp server you will need to have the files 
@@ -133,6 +141,19 @@ make test-iceberg     # Iceberg tests
 make test-lmdb        # LMDB tests
 ```
 
+## Feature-Specific Tests
+
+Run tests for specific features:
+```bash
+make test-parallel-streaming   # Parallel streaming integration tests (requires Amp server)
+```
+
+**Note**: Parallel streaming tests require an Amp server. Configure using environment variables in `.test.env`:
+- `AMP_SERVER_URL` - Amp server URL (e.g., `grpc://your-server:80`)
+- `AMP_TEST_TABLE` - Source table name (e.g., `eth_firehose.blocks`)
+- `AMP_TEST_BLOCK_COLUMN` - Block column name (default: `block_num`)
+- `AMP_TEST_MAX_BLOCK` - Max block for testing (default: `1000`)
+
 # Linting and formatting
 
 Ruff is configured to be used for linting and formatting of this project.