TESTING_STRATEGY.md

Comprehensive Testing Strategy for Distributed Guidance Template

Testing Philosophy

Two-Layer Testing Approach

1. Outer Layer (Template): Test cookiecutter mechanics, template generation, guidance distribution
2. Inner Layer (Generated Project): Test project functionality, guidance completeness, tooling integration

Test Categories

• Unit Tests: Individual component functionality
• Integration Tests: Cross-component interactions
• End-to-End Tests: Complete workflows from template to working project
• Guidance Tests: AI guidance completeness and accuracy
• Contract Tests: Interface compatibility between layers

Outer Layer Testing (Template Repository)

Test Structure

data-eng-template/
├── tests/
│   ├── __init__.py
│   ├── conftest.py                     # Test configuration and fixtures
│   ├── unit/
│   │   ├── test_cookiecutter_config.py # cookiecutter.json validation
│   │   ├── test_template_structure.py  # Directory structure validation
│   │   └── test_guidance_files.py      # CLAUDE.md content validation
│   ├── integration/
│   │   ├── test_template_generation.py # Full generation testing
│   │   ├── test_variable_resolution.py # Variable substitution testing
│   │   └── test_hook_execution.py      # post_gen_project.py testing
│   ├── e2e/
│   │   ├── test_devcontainer_startup.py # End-to-end DevContainer testing
│   │   └── test_generated_project_health.py # Generated project validation
│   └── fixtures/
│       ├── test_configs/               # Test cookiecutter configurations
│       └── expected_outputs/           # Expected generation results
├── pytest.ini                         # Pytest configuration
├── requirements-test.txt               # Testing dependencies
└── .github/workflows/test-template.yml # CI/CD pipeline

Test Coverage Areas

• Template Generation: All cookiecutter configurations generate successfully
• Variable Validation: All required variables present and properly typed
• File Structure: Generated projects have correct directory structure
• Guidance Distribution: All CLAUDE.md files created with proper content
• Hook Execution: post_gen_project.py creates .env files and initializes git
• Template Syntax: No Jinja template conflicts or syntax errors

Inner Layer Testing (Generated Project)

Test Structure

{{cookiecutter.repo_slug}}/
├── tests/
│   ├── __init__.py
│   ├── conftest.py                     # Test fixtures and configuration
│   ├── unit/
│   │   ├── dags/
│   │   │   ├── test_dag_structure.py   # DAG definition validation
│   │   │   └── test_dag_dependencies.py # Task dependency validation
│   │   ├── dbt/
│   │   │   ├── test_model_compilation.py # dbt model compilation
│   │   │   └── test_model_structure.py   # Model naming and structure
│   │   ├── transforms/
│   │   │   ├── test_data_models.py     # SQLModel validation
│   │   │   └── test_validators.py      # Pydantic validator testing
│   │   └── scripts/
│   │       └── test_utilities.py      # Script utility testing
│   ├── integration/
│   │   ├── test_airflow_dbt.py        # Airflow-dbt integration
│   │   ├── test_database_connectivity.py # Database connection testing
│   │   └── test_pipeline_flow.py      # Bronze→Silver→Gold flow
│   ├── guidance/
│   │   ├── test_claude_md_completeness.py # Guidance file validation
│   │   └── test_guidance_accuracy.py     # Content accuracy testing
│   └── e2e/
│       ├── test_devcontainer_startup.py  # DevContainer health
│       └── test_full_pipeline.py         # Complete pipeline execution
├── pytest.ini
└── requirements-test.txt

Test Implementation Priority

Phase 1: Critical Infrastructure

1. Template generation and variable resolution
2. Basic project structure validation
3. Guidance file creation and content

Phase 2: Functional Testing

1. DAG structure and dependency validation
2. dbt model compilation and structure
3. Database connectivity and basic operations

Phase 3: Integration Testing

1. Airflow-dbt integration workflows
2. End-to-end pipeline execution
3. DevContainer startup and service health

Phase 4: Guidance and Documentation

1. CLAUDE.md completeness and accuracy
2. Cross-reference validation between guidance files
3. AI interaction context validation