TDOE#230: Fix Student Program Dedupe, and add Program Status staging models.

[AngelicaLastra]

Warning

• This PR depends on a companion PR (PR#190) in edu_edfi_source that adds k_student_program and the new program_participation_statuses staging models. Do not merge until that PR is approved and a versioned release is available.

Description & motivation

This PR is a subset of TN'S edu_wh PR #230 , pulled forward separately because this piece is ready to ship while the rest of that work is still being scoped. It is the edu_wh companion to the edu_edfi_source PR that introduces k_student_program.

It propagates the new k_student_program surrogate key into all eight program association fact tables, adds a new fct_student_program_participation_status cross-program fact table, and expands fct_student_program_service to include services from the generic program association endpoint. ed_org_id is intentionally not exposed as a standalone column in any of the fact tables since it is fully encoded within k_student_program.

Main highlights:

• k_student_program added to all 8 program association fact tables, surfaced as a NOT NULL column and declared in YAML column lists and constraints blocks.
• fct_student_program_participation_status is a new union-style fact table (same pattern as fct_student_program_service) that stacks participation status rows from all enabled program types into a single table.
• fct_student_program_service now also unions in stg_ef3__stu_program__program_services, covering services from the generic program association endpoint that were previously missing.

PR Merge Priority:

• Low
• Medium
• High

Changes to existing files:

Parent program association fact table SQLs (cherry-picked from PR #230):

• Added stage.k_student_program to each formatted CTE SELECT list, and added alter column k_student_program set not null to each post_hook. No other logic was changed:
  • fct_student_program_association.sql
  • fct_student_special_education_program_association.sql
  • fct_student_cte_program_associations.sql
  • fct_student_homeless_program_association.sql
  • fct_student_language_instruction_program_association.sql
  • fct_student_migrant_education_program_associations.sql
  • fct_student_school_food_service_program_associations.sql
  • fct_student_title_i_part_a_program_association.sql:

Parent program association fact table YMLs (cherry-picked from PR #230):

• Same 8 models above: k_student_program added to the column list in each YAML.

fct_student_program_service.sql (cherry-picked from PR #230):

• Added stg_ef3__stu_program__program_services as an unconditional source in the union, so services from the generic program association endpoint are now included.
• Added stacked.k_student_program to the subset CTE SELECT list.
• Added stacked.cip_code (gated by src:program:cte:enabled) to the subset CTE SELECT list.
• Updated post_hook primary key from (k_student, k_program, program_enroll_begin_date, program_service) --> (k_student_program, program_service).

Bug fixes caught in review:

• Chose to remove ed_org_id from the final SELECT of all fact models, as we don't usually bring forward raw ID fields, those tend to only exist within dim tables. The k_student_program key contains the ID field as part of its source, so we can retain the proper grain of the table through that key (and k_lea/k_school).
• fct_student_program_participation_status.sql:
  • The Food Service block was referencing stg_ef3__stu_school_food_service__program_services (the services table) instead of stg_ef3__stu_school_food_service__program_participation_statuses.
  • Also removed a stray alter column ed_org_id set not null post_hook entry.
• fct_student_program_service.sql:
  • Removed a stray alter column ed_org_id set not null post_hook entry for the same reason.
• fct_student_program_service.yml:
  • Updated constraints primary key from [k_student, k_program, program_enroll_begin_date, program_service] --> [k_student_program, program_service] to match the post_hook.
  • Updated the unique_combination_of_columns test to match.
  • Added stg_ef3__stu_program__program_services to the description's source model list.
  • Corrected a pre-existing typo in the Title I link (stg_ef3__stu_title_i__ --> stg_ef3__stu_title_i_part_a__).
• fct_student_special_education_program_association.yml:
  • Removed ed_org_id from the unique_combination_of_columns test.
• All 8 typed program association YMLs + fct_student_program_service.yml:
  • Removed ed_org_id from column lists across the board. It was added in the cherry-picked work but is not selected in any of the SQL outputs since it is encoded in k_student_program.
  • Updated the primary key from (k_student, k_program, program_enroll_begin_date) --> (k_student_program) k_student_program; the surrogate is the correct grain identifier for these tables.
• fct_student_program_association.yml:
  • Corrected the primary key description from `k_student, k_program` --> `k_student, k_program, program_enroll_begin_date` to match the actual constraint and test.

New files created:

• fct_student_program_participation_status.sql (cherry-picked from PR Feature/tdoe cds enhanced and everwhere 0.6.2 #230): Union fact table that stacks program_participation_statuses records from all enabled program types, plus the generic program association endpoint unconditionally. Follows the same Jinja union pattern as fct_student_program_service: a list of staging relations is built at compile time, unioned with dbt_utils.union_relations, and a subset CTE selects the shared columns. rain is one record per student, year, program, ed org, program enrollment begin date, participation status, and status begin date (k_student_program + participation_status + status_begin_date). Joins to dim_program to filter on valid programs; FK to dim_student is declared via post_hook for Snowflake metadata.
• fct_student_program_participation_status.yml: Documentation for the new model. Uses the same any_var enabled logic as fct_student_program_service. Includes constraints, unique_combination_of_columns test on k_student_program + participation_status + status_begin_date, and full column list.

Tests and QC done:

• dbt compile clean against local edu_edfi_source development branch
• dbt run --select tag:special_ed tag:cte tag:homeless tag:language_instruction tag:migrant_education tag:food_service tag:title_i passes
• fct_student_program_participation_status uniqueness test passes with real data
• Row counts on program fact tables verified before/after k_student_program addition