Allow truth data to have smaller domain than prediction models in BMC

[Copilot]

Plan to Fix Domain Intersection Issue ✅ COMPLETE

Problem: Currently, load_data() uses an inner join on all models including the truth data, which forces the truth data to be present at all domain points. This unnecessarily restricts the model domains when truth data has a smaller domain.

Solution: Modify load_data() to:

• Add optional parameter truth_column_name to identify which model is the truth data
• Separate models into regular models and truth model
• Calculate common domain intersection for regular models only
• Left join truth data onto the common model domain (keeping all model domain points)
• Ensure backward compatibility by making the parameter optional

Changes completed:

• Modified Dataset.load_data() method signature to accept truth_column_name parameter
• Updated loading logic to handle truth data separately using left join
• Added comprehensive tests to verify truth data can have smaller domain (HDF5 & CSV)
• Added integration tests demonstrating full BMC workflow with smaller truth domains
• Verified backward compatibility (all existing tests pass)
• Created example script demonstrating the new feature
• Updated documentation (docs/usage.md and docs/index.md) with examples and usage guide

Documentation Updates:

• Updated usage guide with truth_column_name parameter examples
• Added notes explaining the behavior when truth has smaller domain
• Added complete example workflow demonstrating the feature
• Updated index.md to highlight flexible truth data capability
• Added tips for filtering data when training with smaller truth domain

Testing: All 29 tests pass, including:

• 3 new unit tests for smaller truth domain functionality
• 2 new integration tests showing complete BMC workflow
• All 24 existing tests (verifying backward compatibility)

Key Benefits:
✓ Truth data can have fewer domain points than models
✓ Train BMC on subset where truth is available
✓ Make predictions on full model domain
✓ Backward compatible - existing code works unchanged
✓ Works with both HDF5 and CSV formats
✓ Fully documented with examples

Original prompt

When loading data and using in the BMC, the models being mixed need to have a common domain. The truth data, called truth_column_name in the BMC module, can be smaller than the model domains. Propose a fix for this without introducing breaking changes.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[kylegodbey]

@copilot update the examples and the usage guide in the docs to reflect this behavior

[Copilot]

@copilot update the examples and the usage guide in the docs to reflect this behavior

Updated the documentation in commit COMMIT_HASH:

Changes made:

1. docs/usage.md - Comprehensive updates:

   • Added truth_column_name parameter to the main example
   • Added note boxes explaining the smaller domain behavior
   • Added complete example workflow section demonstrating the feature
   • Updated all code examples to show models_list excluding truth column
   • Added tips for filtering data when training with smaller truth domain

2. docs/index.md - Added feature highlight:

   • Listed "Flexible Truth Data" as a key feature

The usage guide now clearly demonstrates:

• How to use truth_column_name parameter
• What happens when truth has smaller domain
• How to filter training data appropriately
• Complete end-to-end workflow example