Skip to content

Add model validation diagnostics#29

Merged
mdales merged 12 commits intomainfrom
sw-model-validation
Nov 1, 2025
Merged

Add model validation diagnostics#29
mdales merged 12 commits intomainfrom
sw-model-validation

Conversation

@shaneweisz
Copy link
Copy Markdown
Collaborator

@shaneweisz shaneweisz commented Oct 31, 2025

Summary

Adds extra output files that make it easier to do understand and interpret the Dahal et al model validation results.

Changes

  • Added 4 new output files alongside the outliers CSV:

    • summary.md - Overview statistics and outlier counts by class
      Screenshot 2025-10-31 at 5 12 53 pm
    • model_coefficients.csv - the fitted model coefficients for each taxonomic class, helps with interpretability:
      Screenshot 2025-10-31 at 5 25 21 pm
    • random_effects.csv - Family-level random effects
      Screenshot 2025-10-31 at 5 26 22 pm
    • species_with_no_aoh.csv - Species that produced zero AOH area
  • Added diagnostic columns to the outliers CSV:

    • outlier_type (over-predicted/under-predicted/normal)
    • explanation (human-readable description)
    • Comparison metrics vs class means
      For example, the below screenshot suggests that erroneous elevation ranges are the explanation for some of these amphibians being flagged:
      Screenshot 2025-10-31 at 5 29 22 pm

Testing

Tested using aohs.csv produced in the STAR pipeline.

shaneweisz and others added 10 commits October 29, 2025 14:53
- Organize outputs into model_validation/ subdirectory
- Add diagnostic columns for outlier analysis
- Refactor code into focused functions
- Generate README documenting validation results
- Simplify coefficients table format

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Introduce aoh-prevalence-diagnostician subagent for detailed outlier explanations
- Create diagnose-aoh-prevalence command to streamline user interaction
- Update model validation functions to include random effects in outputs
- Enhance validation pipeline with comprehensive tests for new features

🤖 Generated with [Claude Code](https://claude.com/claude-code)
- Simplify YAML frontmatter to standard format
- Add edge case handling for species not found or not flagged
- Add example calculation format for clarity
- Improve argument mapping between command and subagent
- Add usage examples to command documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Add a check for the DATADIR environment variable to ensure data directory is set
- Update steps for locating validation outputs to use the DATADIR path
- Improve error handling for missing validation files
- Renumber steps for clarity and consistency

🤖 Generated with [Claude Code](https://claude.com/claude-code)
- Streamline the diagnostic steps for analyzing outlier species, emphasizing data extraction and calculation processes.
- Introduce a structured report format for presenting findings, including model predictions and biological interpretations.
- Enhance clarity by renumbering steps and consolidating instructions for user interaction and output generation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@shaneweisz shaneweisz changed the title Sw model validation Add model validation diagostics Oct 31, 2025
@shaneweisz shaneweisz changed the title Add model validation diagostics Add model validation diagnostics Oct 31, 2025
Copy link
Copy Markdown
Contributor

@mdales mdales left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, particularly splitting up the logic into more manageable bits where in future we might want to play.

This directory contains the following scripts:

* `collate_data.py` - Then you generate a series of AOH GeoTIFFs, besides each one is a JSON file that contains information required for validation. This script takes a folder containing the AOH output of a run and collates all those JSON files into a single CSV file that can be used for a validation run.
* `collate_data.py` - When you generate a series of AOH GeoTIFFs, besides each one is a JSON file that contains information required for validation. This script takes a folder containing the AOH output of a run and collates all those JSON files into a single CSV file that can be used for a validation run.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a right of passage for working on any code base I've touched - I'm sorry :)

@mdales mdales merged commit 6fc2c22 into main Nov 1, 2025
1 check passed
@mdales mdales deleted the sw-model-validation branch November 1, 2025 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants