Skip to content

Commit 9181390

Browse files
jpn--jpn--
authored
Release notes v1.3 (#882)
* release notes * some older changes * numeric downcasting, protect_columns example * explicit chunk settings description * notes on new examples * Fix example docs page
1 parent d81f5f2 commit 9181390

File tree

5 files changed

+311
-1682
lines changed

5 files changed

+311
-1682
lines changed

‎docs/dev-guide/changes.md‎

Lines changed: 283 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,283 @@
1+
# Change Log
2+
3+
This document describes significant changes to ActivitySim. This includes
4+
major new features that may require modifications to existing model configurations
5+
or code to utilize, as well as breaking changes that may cause existing model
6+
configurations or code to fail to run correctly.
7+
8+
## v1.3
9+
10+
### New Canonical Examples
11+
12+
Beginning with version 1.3, ActivitySim provides two supported "canonical" example
13+
implementations:
14+
15+
- the [SANDAG Model](https://github.com/ActivitySim/sandag-abm3-example) is a two-zone
16+
model based on the SANDAG ABM3 model, and
17+
- the [MTC Model](https://github.com/ActivitySim/activitysim-prototype-mtc) is a
18+
one-zone model based on the MTC's Travel Model One.
19+
20+
Each example implementation includes a complete set of model components, input data,
21+
and configuration files, and is intended to serve as a reference for users to build
22+
their own models. They are provided as stand-alone repositories, to highlight the
23+
fact that model implementations are separate from the ActivitySim core codebase,
24+
and to make it easier for users to fork and modify the examples for their own use
25+
without needing to modify the ActivitySim core codebase. The examples are maintained
26+
by the ActivitySim Consortium and are kept up-to-date with the latest version of
27+
ActivitySim.
28+
29+
```{note}
30+
The two example models are not identical to the original agency models from which
31+
they were created. They are generally similar to those models, and have been calibrated
32+
and validated to reproduce reasonable results. They are intended to demonstrate the
33+
capabilities of ActivitySim and to provide a starting point for users to build their own
34+
models. However, they are not intended to be used as-is for policy analysis or forecasting.
35+
```
36+
37+
### Logging
38+
39+
The reading of YAML configuration files has been modified to use the "safe" reader,
40+
which prohibits the use of arbitrary Python code in configuration files. This is a
41+
security enhancement, but it requires some changes to the way logging is configured.
42+
43+
In previous versions, the logging configuration file could contain Python code to
44+
place log files in various subdirectories of the output directory, which might
45+
vary for different subprocesses of the model, like this:
46+
47+
```yaml
48+
logging:
49+
handlers:
50+
logfile:
51+
class: logging.FileHandler
52+
filename: !!python/object/apply:activitysim.core.config.log_file_path ['activitysim.log']
53+
mode: w
54+
formatter: fileFormatter
55+
level: NOTSET
56+
```
57+
58+
In the new version, the use of `!!python/object/apply` is prohibited. Instead of using
59+
this directive, the `log_file_path` function can be invoked in the configuration file
60+
by using the `get_log_file_path` key, like this:
61+
62+
```yaml
63+
logging:
64+
handlers:
65+
logfile:
66+
class: logging.FileHandler
67+
filename:
68+
get_log_file_path: activitysim.log
69+
mode: w
70+
formatter: fileFormatter
71+
level: NOTSET
72+
```
73+
74+
Similarly, previous use of the `if_sub_task` directive in the logging level
75+
configuration like this:
76+
77+
```yaml
78+
logging:
79+
handlers:
80+
console:
81+
class: logging.StreamHandler
82+
stream: ext://sys.stdout
83+
level: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [WARNING, NOTSET]
84+
formatter: elapsedFormatter
85+
```
86+
87+
can be replaced with the `if_sub_task` and `if_not_sub_task` keys, like this:
88+
89+
```yaml
90+
logging:
91+
handlers:
92+
console:
93+
class: logging.StreamHandler
94+
stream: ext://sys.stdout
95+
level:
96+
if_sub_task: WARNING
97+
if_not_sub_task: NOTSET
98+
formatter: elapsedFormatter
99+
```
100+
101+
For more details, see [logging](Logging).
102+
103+
### Chunking
104+
105+
Version 1.3 introduces a new "[explicit](Explicit-Chunking)" chunking mechanism.
106+
107+
Explicit chunking is simpler to use and understand than dynamic chunking, and in
108+
practice has been found to be more robust and reliable. It requires no "training"
109+
and is activated in the top level model configuration file (typically `settings.yaml`):
110+
111+
```yaml
112+
chunk_training_mode: explicit
113+
```
114+
115+
Then, for model components that may stress the memory limits of the machine,
116+
the user can specify the number of choosers in each chunk explicitly, either as an integer
117+
number of choosers per chunk, or as a fraction of the overall number of choosers.
118+
This is done by setting the `explicit_chunk` configuration setting in the model
119+
component's settings. For this setting, integer values greater than or equal to 1
120+
correspond to the number of chooser rows in each explicit chunk. Fractional values
121+
less than 1 correspond to the fraction of the total number of choosers.
122+
If the `explicit_chunk` value is 0 or missing, then no chunking
123+
is applied for that component. The `explicit_chunk` values in each component's
124+
settings are ignored if the `chunk_training_mode` is not set to `explicit`.
125+
Refer to each model component's configuration documentation for details.
126+
127+
Refer to code updates that implement explicit chunking for accessibility in
128+
[PR #759](https://github.com/ActivitySim/activitysim/pull/759), for
129+
vehicle type choice, non-mandatory tour frequency, school escorting, and
130+
joint tour frequency in [PR #804](https://github.com/ActivitySim/activitysim/pull/804),
131+
and all remaining interaction-simulate components in
132+
[PR #870](https://github.com/ActivitySim/activitysim/pull/870).
133+
134+
### Automatic dropping of unused columns
135+
136+
Variables that are not used in a model component are now automatically dropped
137+
from the chooser table before the component is run. Whether a variable is deemed
138+
as "used" is determined by a text search of the model component code and specification
139+
files for the variable name. Dropping unused columns can be disabled by setting
140+
[`drop_unused_columns`](activitysim.core.configuration.base.ComputeSettings.drop_unused_columns)
141+
to `False` in the [`compute_settings`](activitysim.core.configuration.base.ComputeSettings)
142+
for any model component, but by default this setting is `True`, as it can result in a
143+
significant reduction in memory usage for large models.
144+
145+
Dropping columns may also cause problems if the model is not correctly configured.
146+
If it is desired to use this feature, but some required columns are being dropped
147+
incorrectly, the user can specify columns that should not be dropped by setting the
148+
[`protect_columns`](activitysim.core.configuration.base.ComputeSettings.protect_columns)
149+
setting under [`compute_settings`](activitysim.core.configuration.base.ComputeSettings).
150+
This allows the user to specify columns that should not be dropped, even if they are
151+
not apparently used in the model component. For [example](https://github.com/ActivitySim/activitysim/blob/67820ad32789f59217608b5311e9a2a322d029ed/activitysim/examples/prototype_sandag_xborder/configs/tour_od_choice.yaml#L59-L61):
152+
153+
```yaml
154+
compute_settings:
155+
protect_columns:
156+
- origin_destination
157+
```
158+
159+
Code updates to drop unused columns are in
160+
[PR #833](https://github.com/ActivitySim/activitysim/pull/833) and to protect
161+
columns in [PR #871](https://github.com/ActivitySim/activitysim/pull/871).
162+
163+
### Automatic conversion of string data to categorical
164+
165+
Version 1.3 introduces a new feature that automatically converts string data
166+
to categorical data. This reduces memory usage and speeds up processing for
167+
large models. The conversion is done automatically for string columns
168+
in most chooser tables.
169+
170+
To further reduce memory usage, there is also an optional downcasting of numeric
171+
data available. For example, this allows storing integers that never exceed 255
172+
as `int8` instead of `int64`. This feature is controlled by the `downcast_int`
173+
and `downcast_float` settings in the top level model configuration file (typically
174+
`settings.yaml`). The default value for these settings is `False`, meaning that
175+
downcasting is not applied. It is recommended to leave these settings at their
176+
default values unless memory availability is severely constrained, as downcasting
177+
can cause numerical instability in some cases. First, changing the precision of
178+
numeric data could cause results to change slightly and impact a previous calibrated
179+
model result. Second, downcasting to lower byte data types, e.g., int8, can cause
180+
numeric overflow in downstream components if the numeric variable is used in
181+
mathematical calculations that would result in values beyond the lower bit width
182+
limit (e.g. squaring the value). If downcasting is desired, it is strongly recommended
183+
to review all model specifications for compatability, and to review model results
184+
to verify if the changes are acceptable.
185+
186+
See code updates in [PR #782](https://github.com/ActivitySim/activitysim/pull/782)
187+
and [PR #863](https://github.com/ActivitySim/activitysim/pull/863)
188+
189+
### Alternatives preprocessors for trip destination.
190+
191+
Added alternatives preprocessor in
192+
[PR #865](https://github.com/ActivitySim/activitysim/pull/869),
193+
and converted to separate preprocessors for sample (at the TAZ level) and
194+
simulate (at the MAZ level for 2 zone systems) in
195+
[PR #869](https://github.com/ActivitySim/activitysim/pull/869).
196+
197+
### Per-component sharrow controls
198+
199+
This version adds a uniform interface for controlling sharrow optimizations
200+
at the component level. This allows users to disable sharrow entirely,
201+
or to disable the "fastmath" optimization for individual components.
202+
Controls for sharrow are set in each component's settings under `compute_settings`.
203+
For example, to disable sharrow entirely for a component, use:
204+
205+
```yaml
206+
compute_settings:
207+
sharrow_skip: true
208+
```
209+
210+
This overrides the global sharrow setting, and is useful if you want to skip
211+
sharrow for particular components, either because their specifications are
212+
not compatible with sharrow or if the sharrow performance is known to be
213+
poor on this component.
214+
215+
When a component has multiple subcomponents, the `sharrow_skip` setting can be
216+
a dictionary that maps the names of the subcomponents to boolean values.
217+
For example, in the school escorting component, to skip sharrow for an
218+
OUTBOUND and OUTBOUND_COND subcomponent but not the INBOUND subcomponent,
219+
use the following settings:
220+
221+
```yaml
222+
compute_settings:
223+
sharrow_skip:
224+
OUTBOUND: true
225+
INBOUND: false
226+
OUTBOUND_COND: true
227+
```
228+
229+
The `compute_settings` can also be used to disable the "fastmath" optimization.
230+
This is useful if the component is known to have numerical stability issues
231+
with the fastmath optimization enabled, usually when the component potentially
232+
works with data that includes `NaN` or `Inf` values. To disable fastmath for
233+
a component, use:
234+
235+
```yaml
236+
compute_settings:
237+
fastmath: false
238+
```
239+
240+
Code updates that apply these settings are in
241+
[PR #824](https://github.com/ActivitySim/activitysim/pull/824).
242+
243+
### Configuration validation
244+
245+
Version 1.3 adds a configuration validation system using the Pydantic library.
246+
Previously, the YAML-based configuration files were allowed to contain arbitrary
247+
keys and values, which could lead to errors if the configuration was not correctly
248+
specified. The new validation system checks the configuration files for correctness,
249+
and provides useful error messages if the configuration is invalid. Invalid
250+
conditions include missing required keys, incorrect data types, and the presence
251+
of unexpected keys. Existing models may need to be cleaned up (i.e. extraneous settings
252+
in config files removed) to conform to the new validation system.
253+
254+
See [PR #758](https://github.com/ActivitySim/activitysim/pull/758) for code updates.
255+
256+
### Input checker
257+
258+
Version 1.3 adds an input checker that verifies that the input data is consistent
259+
with expectations. This tool can help identify problems with the input data before
260+
the model is run, and can be used to ensure that the input data is correctly
261+
formatted and complete.
262+
263+
See [PR #753](https://github.com/ActivitySim/activitysim/pull/753) for code updates.
264+
265+
### Removal of orca dependency
266+
267+
This new version of ActivitySim does not use `orca` as a dependency, and thus does
268+
not rely on orca’s global state to manage data. Instead, a new [`State`](activitysim.core.workflow.State)
269+
class is introduced, which encapsulates the current state of a simulation including
270+
all data tables. This is a significant change “under the hood”, which may be
271+
particularly consequential for model that use “extensions” to the ActivitySim framework.
272+
See [PR #654](https://github.com/ActivitySim/activitysim/pull/654) for code updates.
273+
274+
## v1.2
275+
276+
The [v1.2](https://github.com/ActivitySim/activitysim/releases/tag/v1.2.0) release
277+
includes all updates and enhancements complete in the ActivitySim Consortium's Phase 7
278+
development cycle, including:
279+
280+
- Sharrow performance enhancement
281+
- Explicit school escorting
282+
- Disaggregate accessibility
283+
- Simulation-based shadow pricing

‎docs/dev-guide/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,3 +35,4 @@ Contents
3535
../core
3636
../benchmarking
3737
build-docs
38+
changes

0 commit comments

Comments
 (0)