Ingestion Validation

[Mesh-ach]

changes

• Refactored validation pipeline to support:
  • Unified schema validation across cohort, course, and finance datasets.
  • Column normalization and fuzzy matching using aliases and canonical names.
  • Optional vs. required field distinctions with soft vs. hard failure handling.
• Replaced rigid schema checks with dynamic schema loading from JSON specs (base_schema.json and optional institution-specific extensions).
• Introduced Pandera-based validation using dynamically constructed regex-based DataFrameSchema.
• Error reporting via a custom HardValidationError class.
• Centralized logic for model inference, schema merging, and failure classification.
• Updated validation.py and tests to support the new logic and structure.

context

Previously, schema validation was tightly coupled to static column definitions. This limited flexibility and introduced maintenance overhead as models evolved.

This PR introduces a scalable, extensible schema-driven approach for validating CSV uploads:

• Schemas are centrally managed via JSON.
• Optional finance fields are now treated as non-blocking, while a core subset remains required (e.g., Pell).
• Designed to handle mixed-model files and future schema evolution with minimal code changes.

questions

No questions at this time

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1209728085135512