Feature/tdoe cds enhanced and everwhere 0.6.2

[smckee-tnedu]

Description & motivation

The current EA CDS implementation is only on some tables probably because it was added only at the time it was needed by some customer. TDOE has added CDS to all dims and facts. Additionally, TDOE has enhanced the CDS implementation to be more flexible, but the current CDS implementation still works. So this should be backward compatible. The new impl allows a developer to directly control the joins and the output columns, as well as what the output column should be aliased as AND a default value if there is no row for the join.

IMPORTANT: This PR DEPENDS ON the enhancements to the Student Program Association tables and related tables. The reason for the dependency is because some of the current tables related to that contain a bug dealing with ed_org_id, which warranted fixes across the board, introducing a new PK for many of those tables. That PK, in many cases, is what the old CDS impl would have relied upon.

Here is an example of a CDS that is configured both ways:

  'edu:stu_demos:custom_data_sources': 
    cds_student_additional_columns:
      state_student_id:
        where: stateStudentId
      date_entered_us:
        where: dateenteredus
    cds_student_statuses:
      joins: 
        - cds_student_statuses.k_student = stg_student.k_student
      cds_additional_cols:
        tdoe_severity_code: { as: tdoe_severity_code, default: 0 }
        tdoe_severity: { as: tdoe_severity, default: "'good'" }

Here is an example of a CDS that simply adds columns to a dim/fact. The CDS model still has to exist, but it can be mainly empty. This would produce a cross-join... so know what you're doing:

  'edu:subgroup:custom_data_sources':
    cds_default_severity_columns:
      joins: [true]
      cds_additional_cols:
        tdoe_severity_code: { as: tdoe_severity_code, default: 0 }
        tdoe_severity: { as: tdoe_severity, default: "'good'" }

Breaking changes introduced by this PR:

There should not be any breaking changes introduced by this PR. But as the existing CDS code was physically moved into macros and, in some cases, changed in what CTE the code was located there COULD be breaking changes. However, before this impl we were using some originally CDSs and when systematically modified everything, my original CDS impls were still working.

PR Merge Priority:

• Low
• Medium
• High

Changes to existing files:

macros/accordion_columns.sql : Enhanced this macro to be able to deal with trailing commas. This enhancement is backwards compatible with the current codebase.
models/core_warehouse/dim_assessment.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_calendar_date.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_class_period.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_classroom.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_cohort.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_course.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_course_section.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_discipline_incident.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_esc.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_grading_period.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_graduation_plan.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_lea.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_learning_standard.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_network.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_objective_assessment.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_parent.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_program.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_school.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_school_calendar.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_session.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_staff.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_student.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/dim_subgroup.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_course_transcripts.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_staff_school_association.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_staff_section_association.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_academic_record.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_assessment.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_cohort_association.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_cte_program_associations.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_cte_program_associations.yml : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_daily_attendance.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_diploma.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_discipline_actions.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_discipline_actions_summary.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_discipline_incident_behaviors.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_discipline_incident_non_offenders.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_discipline_incident_summary.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_gpa.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_grades.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_homeless_program_association.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_homeless_program_association.yml : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_language_instruction_program_association.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_language_instruction_program_association.yml : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_learning_standard_grades.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_migrant_education_program_associations.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_migrant_education_program_associations.yml : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_objective_assessment.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_parent_association.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_program_association.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_program_association.yml : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_program_service.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_program_service.yml : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_school_association.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_school_attendance_event.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_school_food_service_program_associations.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_school_food_service_program_associations.yml : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_section_association.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_section_attendance_event.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_special_education_program_association.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_special_education_program_association.yml : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_subgroup.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_title_i_part_a_program_association.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_title_i_part_a_program_association.yml : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/msr_student_cumulative_attendance.sql : Modified to support original CDS impl and TDOE CDS impl.

New files created:

macros/custom_data_sources_macros.sql : This is a new set of macros for all CDS implementations, both EA's original design plus TDOE's implementation.
models/build/edfi_3/students/bld_ef3__student__disabilities.sql : New build model which contains all student's disabilities.
models/build/edfi_3/students/bld_ef3__student__wide_disability_designations.sql : New build model which contains all the disability designations for all student's disabilities.
models/core_warehouse/fct_student_disability.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_disability.yml : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_program_participation_status.sql : Modified to support original CDS impl and TDOE CDS impl.
models/core_warehouse/fct_student_program_participation_status.yml : Modified to support original CDS impl and TDOE CDS impl.

Tests and QC done:

TDOE uses both types of CDS extensively across basically all dims and facts. This is all working for us.

edu_wh PR Review Checklist:

Make sure the following have been completed before approving this PR:

• Description of changes has been added to Unreleased section of CHANGELOG.md. Add under ## New Features for features, etc.
• Code has been tested/checked for Databricks and Snowflake compatibility - EA engineers see Databricks checklist here
• Reviewer confirms the grain of all tables are unchanged, OR any changes are expected, communicated, and this PR is flagged as a breaking change (not for patch release)
• If a new configuration xwalk was added:
  • The code is written such that the xwalk is optional (preferred), and this behavior was tested, OR
  • The code is written such that the xwalk is required, and the required xwalk is added to edu_project_template, and this PR is flagged as breaking change (not for patch release)
  • A description for the new xwalk has been added to EDU documentation site here
• If a new configuration variable was added:
  • The code is written such that the variable is optional (preferred), and this behavior was tested, OR
  • The code is written such that the variable is required, and a default value was added to edu_project_template, and this PR is flagged as breaking change (not for patch release)
  • A description for the new variable has been added to EDU documentation site here

[AngelicaLastra]

@smckee-tnedu Thank you for all the effort that went into this code and enhancements, all the code was implemented in such a thoughtful way, I appreciate that. To make the review for this work easier, and have the pieces move forward without blocking each other, I decided it would be best to break this PR into three different ones:

1. PR#249:
   a. Adds k_student_program key to all fct program tables.
   b. Adds new model, fct_student_program_participation_status.
   c. Adds stg_ef3__stu_program__program_services to stack on fct_student_program_service model.

2. PR#250:
   a. Adds build models and fact model changes for Student Disability model refactor.

3. PR#251:
   a. CDS logic for all dim and fact tables.

I will keep you updated as the new PRs open up and as they progress through the review.

[AngelicaLastra]

@smckee-tnedu Update:

• PR#249 based on your student program enhancements has been approved! I will ping you with any updates when it gets merged.