Check YAML, Spec, Coefficients Settings Before Model Runs (#950)

* feat: adds initial setting_checker module to abm/models

* chore: add .vscode settings to gitignore

* feat: adds first test case for prepopulating settings pydantic model (accessibility)

* feat: adds initial spec checker - wip

* feat: update settings checker to load spec and evaluate coefficients. add second model to validate

* fix: adds missing arg to eval_coefficients

* feat: adds atworksubtour_frequency to settings checker

* feat: moves loading spec and coefficients to independent functions

* refactor: renames main settings checker function. moves loading model settings to independent function

* refactor: moves load, spec, and coef eval checks to an independent function called from main checker. run formatter

* feat: adds load SPEC check for preprocessor settings

* chore: run formatter

* feat: initial checking for model setting with nested coefficients

* feat: adds additional model settings. force try_load_spec to return an empty dataframe, even if no top level spec exists

* feat: adds additional model settings to check. adds special handler for disaggregate accessibility (needs more testing)

* feat: adds settings check for initalize_landuse

* refactor: reorders checking for joint tour composition settings

* feat: adds setting check for joint tour destination

* feat: adds settings check for joint tour frequency composition

* feat: adds settings check for joint tour frequency

* feat: adds settings check for joint tour participation

* feat: adds setting check for joint tour scheduling

* feat: adds settings checks for workplace location and school location

* feat: adds setting check for mandatory tour frequency

* feat: adds settings check for non mandatory tour destination

* feat: adds settings check for parking location choice

* feat: adds settings check for school escorting

* wip: adds stop_frequencies to components dict, but requires a separate flow for nested specs

* feat: adds setting checks for summarize

* feat: adds settings check for telecommute frequency

* feat: adds setting check for tour mode choice

* feat: adds setting check for tour od choice

* feat: adds setting check for tour scheduling probabilistic

* feat: adds setting check for transit pass ownership

* feat: adds setting check for transit pass subsidy

* feat: adds settings check for trip departure choice

* feat: adds setting check for trip destination

* feat: adds setting check for write trip matrices

* feat: adds setting check for trip mode choice

* feat: adds settings checks for trip purpose and destination

* feat: adds setting check for trip purpose

* feat: adds setting check for vehicle allocation

* feat: adds setting check for vehicle type choice

* feat: adds setting check for work from home

* docs: remove one comment

* refactor: removes spurious debug point around cdap spec load

* refactor: refactors setting checking to separate read/evaluate. checks now return result of tuple (Dataframe, Exception | None) for collection of errors.

* feat: adds setting check for intialize households

* feat: adds setting check for mandatory and non-mandatory tour scheduling

* feat: adds partial setting check for non-mandatory tour frequency (not fully implemented due to NESTED_SPEC)

* refactor: raise RunTimeError instead of custom

* feat: adds module-specific file handler to logging

* docs: remove one comment

* refactor: add more specific return type

* refactor: renames try_load_and_check to try_load_and_check_spec_coefs and directly injects model settings, separing check for reading  settings yaml files. useful for checking nested model settings

* feat: adds initial handling for SPEC_NEST configs. includes some prequisite refactors to error collection

* refactor: fix logging

* docs: update one comment

* refactor: remove commented raise statements used for debugging

* chore: run black formatter

* refactor: only ever evaluate spec/coef if both are available

* chore: better logging in model settings load

* feat: make setting checker optional

* feat: adds setting check for input checker

* feat: adds setting check for shadow pricing yaml

* chore: remove some outdated comments

* feat: adds setting checks for intialize los

* refactor: changes return type from empty df -> None when spec/coefs are unavaialable

* feat: adds logic for checking settings with templated coefficients

* chore: run formatter

* chore: fix alphabetization of components registry

* feat: check settings for write_data_dictionary

* feat: allow for detailed checks for SPEC_SEGMENTS with PTYPE against main model spec

* feat: allow for arbitrary loading of spec files from any subsettings in model settings

* feat: allow custom checks for unusual spec/coefficient pairs

* feat: adds custom SettingCheckerError exception for improved logging

* refactor: better checks and warnings when SPEC or COEFFICENT fields are
missing values.

because the fields in the settings classes are inherited, it is not
usually possible for the settings checker to determine if a path to an
external file is actually required for a particular model component.

the workaround is to issue a specific warning that a path *may* be
expected and users should check the YAML file.

the ultimate solution would be to define more robust Pydantic data
models that ensure that fields are marked as required when appropriate.

* chore: fix spelling.

* refactor: renames COMPONENTS_TO_SETTINGS -> CHECKER_SETTINGS and feeds to checker entrypoint as argument.

* refactor: single entry for settings checker. import extension check settings as dict.

* chore: run black formatter

* refactor: remove commented code

* fix: skip errors for write_data_dictionary if optional yaml file is not loadable

* chore: run black formatter

* fix(tests): remove unused COEFFICIENTS from MWCOG test model

* fix(tests): re-add deleted test_mtc.py test script

* fix: make model registry for checking a copy

* fix(test): remove unused general SPEC in non_mandatory_tour_scheduling.yaml from MWCOG example model

---------

Co-authored-by: Jeffrey Newman <[email protected]>

Author: andkay

.gitignore

@@ -49,6 +49,9 @@ coverage.xml
 .project
 .pydevproject
 
+# ides
+.vscode/
+
 # Rope
 .ropeproject