Estimation Enhancements (#917)

* multiprocess initial commit

* blacken

* parquet format for EDBs

* adding pkl, fixing edb concat and write

* fixing double naming of coefficient files

* blacken

* fixing missing cdap coefficients file, write pickle function

* combact edb writing, index duplication, parquet datatypes

* sorting dest choice bundles

* adding coalesce edbs as its own step

* CI testing initial commit

* infer.py CI testing

* estimation sampling for non-mandatory and joint tours

* adding survey choice to choices_df in interaction_sample

* adding option to delete the mp edb subdirs

* changes supporting sandag abm3 estimation mode

* running test sandag example through trip dest sample

* Estimation Pydantic (#2)

* pydantic for estimation settings

* allow df as type in config

* fix table_info

* repair for Pydantic

* df is attribute

* Estimation settings pydantic update

* new compact formatting

* handling multiple columns for parquet write

* dropping duplicate columns

* actually removing duplicate columns

* dfs with correct indexes and correct mp sorting

* ignore index on sort for mp coalesce edbs

* updating estimation checks to allow for non-zero household_sample_size

* Re-estimation (#3)

* pydantic for estimation settings

* allow df as type in config

* fix table_info

* auto ownership

* repair for pydantic

* update for ruff

* updated for simple models

* repair for Pydantic

* simple simulate and location choice

* df is attribute

* scheduling

* stop freq

* test locations

* cdap

* nonmand_and_joint_tour_dest_choice

* nonmand_tour_freq

* fix ci to stop using mamba

* test updates

* use larch6 from pip

* use numba for stop freq

* fix for pandas 1.5

* fix stop freq test for numba

* Sharrow Cache Dir Setting (#893)

* setting necessary filesystem changes from settings file

* set for multiprocessing

* repair github actions

* github action updates (#903)

* script to make data

* unified script for making data

* remove older

* bug

* doc note

* load from parquet if available

* add original alt ids to EDB output when using compact

* fix MP race

* script arg to skip to EDB

* clean up CDAP and blacken

* refactor model_estimation_table_types

change to estimation_table_types, to avoid pydantic namespace clash

* repair drop_dupes

* blacken

* location choice with compact

* choice_def for compact

* spec changes for simple-simulate

* re-estimation demo for auto ownership

* clean up status messages

* change name to stop pydantic warnings

* edit configs

* default estimation sample size is same as regular sample size

* allow location alts not in cv format

* dummy zones for location choice

* update scheduling model estimation

* various cleanup

* stop freq

* tidy build script

* update 02 school location for larger example

* update notebook 04

* editable model re-estimation for location choice

* fix test names

* update notebooks

* cdap print filenames as loading

* notebook 07

* tests thru 07

* notebooks 08 09

* build the data first

* runnable script

* change larch version dependency

* keep pandas<2

* notebooks 10 11

* notebook 12

* remove odd print

* add matplotlib

* notebook 13 14

* test all the notebooks

* add xlsxwriter to tests

* notebook 15

* CDAP revise model spec demo

* notebook 16

* notebook 17

* longer timeout

* notebook 18

* notebook 19

* notebook 20

* smaller notebook 15

* configurable est mode setup

* notebook 21

* notebook 22

* config sample size in GA

* notebook 23

* updates for larch and graphviz

* change default to compact

* compare model 03

* test updates

* rename test targets

* repair_av_zq

* move doctor up

* add another repair

* oops

---------

Co-authored-by: David Hensle <[email protected]>

* Removing estimation.yaml settings that are no longer needed

* fixing unit tests, setting parquet edb default

* one more missed estimation.yaml

* using df.items for pandas 2 compatibility

* tidy doc

* updating edb file name for NMTF

* updating numba and pandas in the conda env files

* Improve test stability (#4)

* handle dev versions of Larch

* test stability

* pin multimethod < 2.0

* add availability_expression

* starting est docs

* Resolve package version conflicts (#923)

* limit multimethod version to 2.0 and earlier

* add multimethod version to other settings

* [makedocs] update installer download link

* [makedocs] update branch docs

* GitHub Actions updates (#926)

* use libmamba solver

* add permissions [makedocs]

* add write permission for dev docs [makedocs]

* conda-solver: classic

* trace proto tables if available, otherwise synthetic population (#901)

Co-authored-by: Jeffrey Newman <[email protected]>

* release instructions (#927)

* use libmamba solver

* add permissions [makedocs]

* add write permission for dev docs [makedocs]

* conda-solver: classic

* include workflow dispatch option for tests

* update release instructions

* add installer build to instructions

* Pin mamba for now, per conda-incubator/setup-miniconda#392

* conda-remove-defaults

* when no unavailability parameters are included

* some general estimation docs

* Use pandas 2 for docbuild environment (#928)

* fix link

* allow failure to import larch

* workflow

* blacken

* try some pins

* speed up docbuild

* use pandas 2 for docs

* oops wrong file

* restore foundation

* Update HOW_TO_RELEASE.md

* refactor(shadow_pricing.py): remove a duplicated `default_segment_to_name_dict` (#930)

* fix typo

* fixing disaggregate accessibility bug in zone sampler

* Revert "fixing disaggregate accessibility bug in zone sampler"

This reverts commit be5d093.

* notes on size terms

* clean up docbuild

* fix version check

* add some doc

* tidy

* estimation docs

* more on alternative avail

* model evaluation

* add doc on component_model

* documentation enhancements

* larch6 is now larch>6

* branch docs on workflow_dispatch

* missing doc section on model respec

---------

Co-authored-by: Yue Shuai <[email protected]>
Co-authored-by: David Hensle <[email protected]>
Co-authored-by: amarin <[email protected]>
Co-authored-by: Ali Etezady <[email protected]>
Co-authored-by: Sijia Wang <[email protected]>

* handling missing data or availability conditions

* add docs on locking size terms

* include constants in CDAP

* bump larch requirement

* require larch 6.0.40

* add xlsxwriter to envs

* require larch 6.0.41

* add links

* fix typos and formatting

* cdap hh and per parquet read match csv

* add missing x_validator for mode choice and nonmand tour freq

* add tour mode choice edit example

* add to docs

* union not addition on sets

* restore nb kernel

* blacken

* replacing conda with uv in estimation tests

* add requests to github-action dependencies

* running with created virtual env instead

* Fix estimation notebook tests (#8)

* Update scheduling.py

---------

Co-authored-by: Jeffrey Newman <[email protected]>
Co-authored-by: Yue Shuai <[email protected]>
Co-authored-by: amarin <[email protected]>
Co-authored-by: Ali Etezady <[email protected]>
Co-authored-by: Sijia Wang <[email protected]>

Author: 6 people

.github/workflows/core_tests.yml

@@ -321,6 +321,83 @@ jobs:
         run: |
           uv run pytest activitysim/estimation/test/test_larch_estimation.py --durations=0
 
+  estimation_notebooks:
+    needs: foundation
+    env:
+      python-version: "3.10"
+      label: win-64
+    defaults:
+      run:
+        shell: pwsh
+    name: Estimation Notebooks Test
+    runs-on: windows-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: "Set up Python"
+        uses: actions/setup-python@v5
+        with:
+          python-version-file: ".python-version"
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          version: "0.7.12"
+          enable-cache: true
+          cache-dependency-glob: "uv.lock"
+
+      - name: setup graphviz
+        uses: ts-graphviz/setup-graphviz@v2
+
+      - name: Install activitysim
+        run: |
+          uv sync --locked --group github-action
+
+      - name: Create Estimation Data
+        run: >
+          uv run --group github-action python activitysim/examples/example_estimation/notebooks/est_mode_setup.py 
+          --household_sample_size 5000
+
+      - name: Test Estimation Notebooks
+        run: >
+          uv run --group github-action pytest activitysim/examples/example_estimation/notebooks 
+          --nbmake-timeout=3000 
+          --ignore=activitysim/examples/example_estimation/notebooks/01_estimation_mode.ipynb 
+          --ignore-glob=activitysim/examples/example_estimation/notebooks/test-estimation-data/**
+
+  estimation_edb_creation:
+    needs: foundation
+    env:
+      python-version: "3.10"
+      label: win-64
+    defaults:
+      run:
+        shell: pwsh
+    name: estimation_edb_creation_test
+    runs-on: windows-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          version: "0.7.12"
+          enable-cache: true
+          cache-dependency-glob: "uv.lock"
+
+      - name: "Set up Python"
+        uses: actions/setup-python@v5
+        with:
+          python-version-file: ".python-version"
+
+      - name: Install activitysim
+        run: |
+          uv sync --locked --only-group github-action
+
+      - name: Test Estimation EDB Creation
+        run: |
+          uv run pytest activitysim/estimation/test/test_edb_creation/test_edb_formation.py --durations=0
+
   expression-profiling:
     needs: foundation
     env:
@@ -397,4 +474,4 @@ jobs:
           github_token: ${{ secrets.GITHUB_TOKEN }}
           # Token is created automatically by Github Actions, no other config needed
           publish_dir: ./docs/_build/html
-          destination_dir: develop
+          destination_dir: develop

activitysim/abm/models/cdap.py

@@ -195,7 +195,7 @@ def cdap_simulate(
         estimator.write_coefficients(coefficients_df, model_settings)
         estimator.write_table(
             cdap_interaction_coefficients,
-            "interaction_coefficients",
+            "cdap_interaction_coefficients",
             index=False,
             append=False,
         )
@@ -204,7 +204,7 @@ def cdap_simulate(
             spec = cdap.get_cached_spec(state, hhsize)
             estimator.write_table(spec, "spec_%s" % hhsize, append=False)
             if add_joint_tour_utility:
-                joint_spec = cdap.get_cached_joint_spec(hhsize)
+                joint_spec = cdap.get_cached_joint_spec(state, hhsize)
                 estimator.write_table(
                     joint_spec, "joint_spec_%s" % hhsize, append=False
                 )

activitysim/abm/models/disaggregate_accessibility.py

@@ -764,11 +764,12 @@ def get_disaggregate_logsums(
             state.filesystem, model_name + ".yaml"
         )
         model_settings.SAMPLE_SIZE = disagg_model_settings.DESTINATION_SAMPLE_SIZE
-        estimator = estimation.manager.begin_estimation(state, trace_label)
-        if estimator:
-            location_choice.write_estimation_specs(
-                state, estimator, model_settings, model_name + ".yaml"
-            )
+        # estimator = estimation.manager.begin_estimation(state, trace_label)
+        # if estimator:
+        #     location_choice.write_estimation_specs(
+        #         state, estimator, model_settings, model_name + ".yaml"
+        #     )
+        estimator = None
 
         # Append table references in settings with "proto_"
         # This avoids having to make duplicate copies of config files for disagg accessibilities

activitysim/abm/models/joint_tour_frequency.py

@@ -192,16 +192,19 @@ def joint_tour_frequency(
         print(f"len(joint_tours) {len(joint_tours)}")
 
         different = False
+        # need to check households as well because the full survey sample may not be used
+        # (e.g. if we set household_sample_size in settings.yaml)
         survey_tours_not_in_tours = survey_tours[
             ~survey_tours.index.isin(joint_tours.index)
+            & survey_tours.household_id.isin(households.index)
         ]
         if len(survey_tours_not_in_tours) > 0:
             print(f"survey_tours_not_in_tours\n{survey_tours_not_in_tours}")
             different = True
         tours_not_in_survey_tours = joint_tours[
             ~joint_tours.index.isin(survey_tours.index)
         ]
-        if len(survey_tours_not_in_tours) > 0:
+        if len(tours_not_in_survey_tours) > 0:
             print(f"tours_not_in_survey_tours\n{tours_not_in_survey_tours}")
             different = True
         assert not different

activitysim/abm/models/location_choice.py

@@ -19,7 +19,6 @@
 from activitysim.core.interaction_sample_simulate import interaction_sample_simulate
 from activitysim.core.util import reindex
 
-
 """
 The school/workplace location model predicts the zones in which various people will
 work or attend school.
@@ -140,7 +139,7 @@ def _location_sample(
 
     sample_size = model_settings.SAMPLE_SIZE
 
-    if estimator:
+    if estimator and model_settings.ESTIMATION_SAMPLE_SIZE >= 0:
         sample_size = model_settings.ESTIMATION_SAMPLE_SIZE
         logger.info(
             f"Estimation mode for {trace_label} using sample size of {sample_size}"
@@ -423,7 +422,7 @@ def location_presample(
 
     # choose a MAZ for each DEST_TAZ choice, choice probability based on MAZ size_term fraction of TAZ total
     maz_choices = tour_destination.choose_MAZ_for_TAZ(
-        state, taz_sample, MAZ_size_terms, trace_label
+        state, taz_sample, MAZ_size_terms, trace_label, model_settings
     )
 
     assert DEST_MAZ in maz_choices
@@ -512,38 +511,6 @@ def run_location_sample(
             trace_label=trace_label,
         )
 
-    # adding observed choice to alt set when running in estimation mode
-    if estimator:
-        # grabbing survey values
-        survey_persons = estimation.manager.get_survey_table("persons")
-        if "school_location" in trace_label:
-            survey_choices = survey_persons["school_zone_id"].reset_index()
-        elif ("workplace_location" in trace_label) and ("external" not in trace_label):
-            survey_choices = survey_persons["workplace_zone_id"].reset_index()
-        else:
-            return choices
-        survey_choices.columns = ["person_id", "alt_dest"]
-        survey_choices = survey_choices[
-            survey_choices["person_id"].isin(choices.index)
-            & (survey_choices.alt_dest > 0)
-        ]
-        # merging survey destination into table if not available
-        joined_data = survey_choices.merge(
-            choices, on=["person_id", "alt_dest"], how="left", indicator=True
-        )
-        missing_rows = joined_data[joined_data["_merge"] == "left_only"]
-        missing_rows["pick_count"] = 1
-        if len(missing_rows) > 0:
-            new_choices = missing_rows[
-                ["person_id", "alt_dest", "prob", "pick_count"]
-            ].set_index("person_id")
-            choices = choices.append(new_choices, ignore_index=False).sort_index()
-            # making probability the mean of all other sampled destinations by person
-            # FIXME is there a better way to do this? Does this even matter for estimation?
-            choices["prob"] = choices["prob"].fillna(
-                choices.groupby("person_id")["prob"].transform("mean")
-            )
-
     return choices
 
 

activitysim/abm/models/non_mandatory_tour_frequency.py

@@ -289,14 +289,22 @@ def non_mandatory_tour_frequency(
         )
 
         if estimator:
-            estimator.write_spec(model_settings, bundle_directory=True)
+            bundle_directory = True
+            # writing to separte subdirectory for each segment if multiprocessing
+            if state.settings.multiprocess:
+                bundle_directory = False
+            estimator.write_spec(model_settings, bundle_directory=bundle_directory)
             estimator.write_model_settings(
-                model_settings, model_settings_file_name, bundle_directory=True
+                model_settings,
+                model_settings_file_name,
+                bundle_directory=bundle_directory,
             )
             # preserving coefficients file name makes bringing back updated coefficients more straightforward
             estimator.write_coefficients(coefficients_df, segment_settings)
             estimator.write_choosers(chooser_segment)
-            estimator.write_alternatives(alternatives, bundle_directory=True)
+            estimator.write_alternatives(
+                alternatives, bundle_directory=bundle_directory
+            )
 
             # FIXME #interaction_simulate_estimation_requires_chooser_id_in_df_column
             #  shuold we do it here or have interaction_simulate do it?
@@ -435,8 +443,10 @@ def non_mandatory_tour_frequency(
     if estimator:
         # make sure they created the right tours
         survey_tours = estimation.manager.get_survey_table("tours").sort_index()
+        # need the household_id check below incase household_sample_size != 0
         non_mandatory_survey_tours = survey_tours[
-            survey_tours.tour_category == "non_mandatory"
+            (survey_tours.tour_category == "non_mandatory")
+            & survey_tours.household_id.isin(persons.household_id)
         ]
         # need to remove the pure-escort tours from the survey tours table for comparison below
         if state.is_table("school_escort_tours"):

activitysim/abm/models/school_escorting.py

@@ -503,7 +503,10 @@ def school_escorting(
                 coefficients_df, file_name=stage.upper() + "_COEFFICIENTS"
             )
             estimator.write_choosers(choosers)
-            estimator.write_alternatives(alts, bundle_directory=True)
+            if state.settings.multiprocess:
+                estimator.write_alternatives(alts, bundle_directory=False)
+            else:
+                estimator.write_alternatives(alts, bundle_directory=True)
 
             # FIXME #interaction_simulate_estimation_requires_chooser_id_in_df_column
             #  shuold we do it here or have interaction_simulate do it?

activitysim/abm/models/stop_frequency.py

@@ -191,9 +191,15 @@ def stop_frequency(
 
         if estimator:
             estimator.write_spec(segment_settings, bundle_directory=False)
-            estimator.write_model_settings(
-                model_settings, model_settings_file_name, bundle_directory=True
-            )
+            # writing to separte subdirectory for each segment if multiprocessing
+            if state.settings.multiprocess:
+                estimator.write_model_settings(
+                    model_settings, model_settings_file_name, bundle_directory=False
+                )
+            else:
+                estimator.write_model_settings(
+                    model_settings, model_settings_file_name, bundle_directory=True
+                )
             estimator.write_coefficients(coefficients_df, segment_settings)
             estimator.write_choosers(chooser_segment)
 
@@ -265,7 +271,11 @@ def stop_frequency(
 
         survey_trips = estimation.manager.get_survey_table(table_name="trips")
         different = False
-        survey_trips_not_in_trips = survey_trips[~survey_trips.index.isin(trips.index)]
+        # need the check below on household_id incase household_sample_size != 0
+        survey_trips_not_in_trips = survey_trips[
+            ~survey_trips.index.isin(trips.index)
+            & survey_trips.household_id.isin(trips.household_id)
+        ]
         if len(survey_trips_not_in_trips) > 0:
             print(f"survey_trips_not_in_trips\n{survey_trips_not_in_trips}")
             different = True