Move final check documentation in python docstring

Author: lmz

README.md

@@ -186,14 +186,14 @@ The data processing pipeline consists of modular steps that transform raw survey
 4. **[Link Trips](https://bayareametro.github.io/travel-diary-survey-tools/pipeline_steps/link_trips/)** - Aggregates individual trip segments into complete journey records by detecting mode changes and transfers
 5. **[Detect Joint Trips](https://bayareametro.github.io/travel-diary-survey-tools/pipeline_steps/detect_joint_trips/)** - Identifies shared household trips using spatial-temporal similarity matching
 6. **[Extract Tours](https://bayareametro.github.io/travel-diary-survey-tools/pipeline_steps/extract_tours/)** - Builds hierarchical tour structures (home-based tours and work-based subtours) from linked trips
-7. **Weighting** *(placeholder)* - Calculates expansion weights to match survey sample to population targets
+7. **[Weighting](https://bayareametro.github.io/travel-diary-survey-tools/pipeline_steps/weighting/)** *(placeholder)* - Calculates expansion weights to match survey sample to population targets
 8. **Format Output** - Transforms canonical data to model-specific formats (DaySim, ActivitySim, etc.)
     - **[DaySim Format](https://bayareametro.github.io/travel-diary-survey-tools/pipeline_steps/format_output/daysim/)** - Formats data for DaySim model input
     - **[CT-RAMP Format](https://bayareametro.github.io/travel-diary-survey-tools/pipeline_steps/format_output/ctramp/)** - Formats data for CT-RAMP model input
-9. **[Final Check](src/processing/final_check/README.md)** - Validates complete dataset against canonical schemas before export
+9. **[Final Check](https://bayareametro.github.io/travel-diary-survey-tools/pipeline_steps/final_check/)** - Validates complete dataset against canonical schemas before export
 10. **[Write Data](https://bayareametro.github.io/travel-diary-survey-tools/pipeline_steps/read_write/)** - Writes processed tables to output files with optional validation
 
-Each step README provides detailed documentation on:
+Each step links to documentation generated by the step's docstring, and provides detailed documentation on:
 - Input/output data requirements
 - Core algorithm and processing logic
 - Configuration parameters

docs/pipeline_steps/final_check.md

@@ -0,0 +1,11 @@
+# Final Check
+
+::: processing.final_check.final_check
+    options:
+      show_root_heading: true
+      show_root_toc_entry: false
+      members:
+        - final_check
+      filters:
+        - "!^logger$"
+        - "!^_"

mkdocs.yml

@@ -72,6 +72,7 @@ nav:
       - Format Output:
           - DaySim Format: pipeline_steps/format_output/daysim.md
           - CT-RAMP Format: pipeline_steps/format_output/ctramp.md
+      - Final Check: pipeline_steps/final_check.md
 
 markdown_extensions:
   - admonition

src/processing/final_check/README.md

@@ -2,64 +2,9 @@
 
 # Final Check Pipeline Step
 
-This module performs final validation checks on the complete processed dataset to ensure data quality and schema compliance. It is basically a dummy module to run Pydantic validation on all tables at the end of the pipeline.
+This module performs final validation checks on the complete processed dataset to ensure data quality and schema compliance before export.
 
-## Pipeline Steps
+For detailed API documentation including validation algorithm, error handling, and implementation notes, see: [Final Check API Documentation](https://bayareametro.github.io/travel-diary-survey-tools/pipeline_steps/final_check/)
 
-### `final_check`
-
-Runs comprehensive validation on all canonical survey tables at the end of the pipeline.
-
-**Inputs:**
-- `households`: Processed household table (pl.DataFrame)
-- `persons`: Processed person table (pl.DataFrame)
-- `days`: Processed person-day table (pl.DataFrame)
-- `unlinked_trips`: Processed unlinked trip records (pl.DataFrame)
-- `linked_trips`: Processed linked trip records (pl.DataFrame)
-- `tours`: Processed tour records (pl.DataFrame)
-
-**Outputs:**
-- Dictionary containing the same validated tables:
-  - `households`
-  - `persons`
-  - `days`
-  - `unlinked_trips`
-  - `linked_trips`
-  - `tours`
-
-**Core Algorithm:**
-
-**Pydantic Model Validation:**
-1. This step is decorated with `@step(validate_input=True, validate_output=True)`
-2. The pipeline framework automatically validates all input/output against Pydantic data models
-3. Validation checks:
-   - **Schema Compliance:** All required columns present with correct data types
-   - **Value Constraints:** Numeric ranges, categorical values, enum memberships
-   - **Referential Integrity:** Foreign keys match (person_id → persons, hh_id → households, etc.)
-   - **Business Rules:** Domain-specific constraints (e.g., depart_time < arrive_time)
-
-**Custom Validation Space:**
-- The function body is intentionally simple (pass-through)
-- Pydantic handles validation automatically at model instantiation
-- This space *could* be extended with additional custom checks not covered by models:
-  - Cross-table consistency checks
-  - Statistical outlier detection
-  - Survey-specific business rules
-  - Data quality metrics logging
-- However, validation logic should ideally be implemented in Pydantic models themselves for reusability
-
-**Validation Failure Handling:**
-- If validation fails, raises `DataValidationError` with detailed error messages
-- Error messages indicate:
-  - Which table failed validation
-  - Which rows/columns have issues
-  - What constraint was violated
-- Pipeline execution halts on validation failure
-
-**Notes:**
-- This is the last checkpoint before data export
-- Ensures output meets canonical data specifications
-- Validation errors caught here prevent invalid data from reaching models/analyses
-- Pydantic models defined in `src/data_canon/models/` provide the validation rules
-- Comprehensive logging helps diagnose data quality issues
-- Pass-through design allows validation to occur transparently
+The documentation includes:
+- `final_check()` - Comprehensive validation pass-through for all canonical tables

src/processing/final_check/final_check.py

@@ -1,4 +1,53 @@
-"""Final validation step for the entire dataset."""
+"""Final Validation Step.
+
+Performs final validation checks on the complete processed dataset to ensure data
+quality and schema compliance before export. This is a pass-through validation step
+that leverages the `@step()` decorator's automatic Pydantic model validation.
+
+!!! Algorithm
+
+    # Pydantic Model Validation
+    
+    1. This step is decorated with `@step(validate_input=True, validate_output=True)`
+    2. The pipeline framework automatically validates all input/output against Pydantic
+       data models
+    3. Validation checks:
+        - **Schema Compliance:** All required columns present with correct data types
+        - **Value Constraints:** Numeric ranges, categorical values, enum memberships
+        - **Referential Integrity:** Foreign keys match (person_id → persons,
+          hh_id → households, etc.)
+        - **Business Rules:** Domain-specific constraints (e.g., depart_time < arrive_time)
+
+    # Custom Validation Space
+    
+    - The function body is intentionally simple (pass-through)
+    - Pydantic handles validation automatically at model instantiation
+    - This space *could* be extended with additional custom checks not covered by models:
+        - Cross-table consistency checks
+        - Statistical outlier detection
+        - Survey-specific business rules
+        - Data quality metrics logging
+    - However, validation logic should ideally be implemented in Pydantic models
+      themselves for reusability
+    
+    # Validation Failure Handling
+    
+    - If validation fails, raises `DataValidationError` with detailed error messages
+    - Error messages indicate:
+        - Which table failed validation
+        - Which rows/columns have issues
+        - What constraint was violated
+    - Pipeline execution halts on validation failure
+
+!!! Notes
+
+    - This is the last checkpoint before data export
+    - Ensures output meets canonical data specifications
+    - Validation errors caught here prevent invalid data from reaching models/analyses
+    - Pydantic models defined in `src/data_canon/models/` provide the validation rules
+    - Comprehensive logging helps diagnose data quality issues
+    - Pass-through design allows validation to occur transparently
+"""
 
 import logging
 
@@ -18,21 +67,41 @@ def final_check(
     linked_trips: pl.DataFrame,
     tours: pl.DataFrame,
 ) -> dict[str, pl.DataFrame]:
-    """Run validation checks on the entire dataset.
+    """Run comprehensive validation on all canonical survey tables.
+
+    This is a pass-through function that relies on the `@step()` decorator to
+    perform automatic Pydantic model validation on both inputs and outputs.
+    Validation checks schema compliance, value constraints, referential integrity,
+    and business rules.
 
     Args:
-        households: The households dataframe
-        persons: The persons dataframe
-        days: The days dataframe
-        unlinked_trips: The unlinked trips dataframe
-        linked_trips: The linked trips dataframe
-        tours: The tours dataframe
+        households: Processed household table with all required fields
+        persons: Processed person table with all required fields
+        days: Processed person-day table with all required fields
+        unlinked_trips: Processed unlinked trip records with all required fields
+        linked_trips: Processed linked trip records (journey-level) with all
+            required fields
+        tours: Processed tour records with all required fields
 
     Returns:
-        The validated dataset
+        Dictionary containing the same validated tables:
+        
+            - households: Validated household table
+            - persons: Validated person table
+            - days: Validated person-day table
+            - unlinked_trips: Validated unlinked trip records
+            - linked_trips: Validated linked trip records
+            - tours: Validated tour records
 
     Raises:
-        DataValidationError: If pydantic validation fails
+        DataValidationError: If pydantic validation fails on any table. Error
+            message indicates which table, row, column, and constraint failed.
+
+    Notes:
+        - Pydantic handles validation automatically at model instantiation
+        - This is the final quality checkpoint before data export
+        - Custom validation logic can be added here if needed, but should
+          ideally be implemented in Pydantic models for reusability
     """
     logger.info("Starting final validation checks")