Multiple Zone Systems Tests

ActivitySim Multiple Zone Branch Performance Tests (DRAFT)

1. Objectives:

SANDAG staff (@wusun2) tested the performance of ActivitySim Multiple Zone branch. The objectives of this effort are:

• Evaluate the performance of implemented methods in ActivitySim Multiple Zone Systems by mimicking SANDAG model settings,
• Provide a peek at the overall ActivitySim performance, especially the performance of methods shared by both the Multiple Zone and master ActivitySim branches,
• Identify performance bottlenecks, and
• Summarize findings.

The focus of this effort is on testing performance of critical 'building blocks' that are called numerous times in utility computations, including retrieving zonal TAZ, TAP, and MAZ values, TAZ, TAP, and MAZ skim values, and the transit virtual path building (TVPB) procedure used to find best transit path (BTP).

2. Test Environment:

Intel® Xeon® Processor E5-2650 v3 @ 2.30GHz; 256 RAM; 20 cores; Windows Server 2012 R2 Std, 64-bit

3. Limitations:

Although the aim of this effort is to test scenarios that mimic SANDAG 2012 model settings, the constructed test scenarios don't fully reflect model complexity for the following reasons:

• Many UEC components such as logsums and zonal, household, and personal attributes are not included in the tests.
• Utility expression evaluation tests are not included, expect for TVPB.
• The best transit path finding test is a more comprehensive case somewhat like the implementation in CT-RAMP. However, the TVPB expression is simplified with no probability computations and no Monte Carlo choices.

Because of the above limitations, the performance of a real model run could be significantly different from that of the tested scenarios. Regardless, these tests server as an initial peek at the performance of ActivitySim.

4. Approaches:

This section describes the process of creating test scenarios. It starts with a breakdown analysis of SANDAG CT-RAMP model components (Table 1). Only San Diego resident models, the most time-consuming part, are included in this analysis. Unlike special market models, the resident models are more generic with similar structures shared by models of most other partner agencies.

For each model component in Table 1, we estimated the computational burden of retrieving skim data by counting the frequency of network LOS methods calls. We use UECs as the basis for estimating computational burden, which is affected by the numbers of decision making units (DMU), skims used in UEC, and alternatives. Each component is then ranked using low, medium, and high burden levels. At the end, mandatory tour mode, non-mandatory tour mode, trip mode, and stop location choices are assigned with 'high' burden status; work, school, and non-mandatory tour location choices are assigned with 'medium' burden status; all other components are assigned with 'low' burden status. The following rules are used in selecting test scenarios:

• Skip all 'low' burden components
• Work, school, and non-mandatory tour location choice are expected to have similar performances; pick work and school location choices for testing.
• Trip and tour mode choices share similar UECs; pick trip mode choice for testing because its computational burden is significantly larger than that of tour mode choice.
• Test stop location choice

Table 1: SANDAG CT-RAMP Resident Models

Model Components	DMU	#Units	Skims	Alts	Method	calls/alt	times called	** Burden**	Test	Note
Work location	#workers	1,313,256	OP_GP_DIST	30	taz_skims()	1	39,397,680	Med.	Yes
School location	#students	965,880	OP_GP_DIST	30	taz_skims()	1	28,976,400	Med.	Yes
Auto ownership	#households							Low	No	Fast
Transponder	#households							Low	No	Fast
Free parking	#workers in CBD							Low	No	Fast
CDAP	#households							Low	No	No Skims
Mandatory tour frequency	#persons with M tours	2,279,136	Distances to work/school	*	taz_skims()	1	2,279,136	Low	No
Mandatory tour mode	#Mandatory tours	1,913,084	100 auto skims	26	taz_skims(); TVPB	4		High	No
Mandatory TOD	#Mandatory tours	1,913,084	OP_GP_DIST	*	taz_skims()	1	1,913,084	Low	No
Join Tour frequency	#persons with J tours							Low	No	small size
Join Tour location	#joint tours							Low	No	small size
Join Tour Mode	#joint tours							Low	No	small size
Join Tour TOD	#joint tours							Low	No	small size
Non-Mandatory tour frequency	#persons with N tours							Low	No	No Skims
Non-Mandatory tour location	#Non-Mandatory tours	1,932,217	OP_GP_DIST	30	taz_skims()	1	57,966,510	Medium	No
Non-Mandatory tour mode	#Non-Mandatory tours	1,932,217	100 auto skims	26	taz_skims(); TVPB	4		High	No
Non-Mandatory TOD	#Non-Mandatory tours	1,932,217	OP_GP_DIST	*	taz_skims()	1	1,932,217	Low	No
AtWork tour frequency	#persons with AtWork tours							Low	No	small size
AtWork tour location	#At-Work tours							Low	No	small size
AtWork tour mode	#At-Work tours							Low	No	small size
AtWork TOD	#At-Work tours							Low	No	small size
Stop frequency	#tours	3,966,590	OP_GP_DIST	*	taz_skims()	1	3,966,590	Low	No
Stop location	#trips	10,146,986	distances: os;sd; od; tos; sto	30	get_maz_pairs()	5	1,522,047,900	High	Yes
trip mode choice	#trips	10,146,986	100 auto skims	26	taz_skims(); TVPB	4	See Table 5	High	Yes

5. Test Scenarios & Test Sizes:

The relevant method in mandatory tour mode choice is taz_skims(). The test size is set to 68,374,080 (Table 2), estimated from the numbers of workers and students, skims used in UEC, and choice alternatives. In SANDAG CT-RAMP, 30 location alternative samples are used in location choice models.

Table 2: Mandatory Tour Choice Test

# workers	1,313,256
#students	965,880
# skim calls in UEC per unit per alternative	1
skim	off peak distance
method	taz_skims()
#alternatives	30
Test Size	68,374,080

The relevant method in stop location choice is get_maz_pairs(). The test size is set to 1,522,047,900 (Table 3), estimated from the numbers of trips, skims used in UEC, and choice alternatives.

Table 3: Stop Location Choice Test

# trips	10,146,986
# skim calls in UEC per trip per alternative	5
skim	Distances: O to S, S to D, O to D, tour O to S, and S to tour D
method	get_maz_pairs()
#alternatives	30
Test Size	1,522,047,900

The test size estimation for trip mode choice model is more nuanced. Trip mode choice UEC utilizes auto, walk, bike, and transit skims (Table 4). There are 100 auto skims alone. However, the utility computation of each auto mode utilizes only a few relevant skims specific to that mode. For example, the only skim needed for computing walk mode utility is walk time. For transit modes, instead of transit skims, the utility computation relies on the BTP computed from the TVPB procedure.

Table 4: Trip Mode Choice Skims

		Method	Skims
#trips	10,146,986
# auto skim calls in UEC per trip per alternative	4	taz_skims()	time and distance by in/out time period
# walk skim calls	1	get_maz_pairs()	walk time
# bike skim calls	1	get_maz_pairs()	bike time
# TVPB Calls	1	TVPB	best transit path
# auto alternatives	14

The burden of computing mode choice utility is mitigated because many trips only have access to a subset of the 26 mode alternatives due to the maximum allowed walk, bike, walk to transit, and drive to transit time thresholds. However, this also makes it difficult to estimate how many modes are available to a trip because mode availability depends on trip origin and destination. In this effort, we constructed 4 hypothetical trip mode choice scenarios (Table 5) that represents various levels of mode availability to trips:

• 10% trips have access to transit; 25% trips have access to walk and bike; 100% trips have access to drive
• 25% trips have access to transit; 50% trips have access to walk and bike; 100% trips have access to drive
• 33% trips have access to transit; 60% trips have access to walk and bike; 100% trips have access to drive
• 50% trips have access to transit; 75% trips have access to walk and bike; 100% trips have access to drive

The frequency of method calls (Table 5) are derived from the total number of trips and mode availability assumptions above. For example, taz_skims(), get_maz_pairs() and TVPB are called 568,231,216 (, 5,073,493 (100% of all 10,146,986 trips have access to drive mode, each trip has 14 auto mode alternatives, and each alternative uses 4 skims), 2,536,747 bike and walk distance calls each), and 1,014,699 times respectively in scenario 1.

Table 5: Trip Mode Choice Test Scenarios

** **	Method	Trip Mode 1	Trip Mode 2	Trip Mode 3	Trip Mode 4
transit available trips		10%	25%	33%	50%
walk available trips		25%	50%	60%	75%
bike available trips		25%	50%	60%	75%
drivable trips		100%	100%	100%	100%
# auto skims calls	taz_skims()	568,231,216	568,231,216	568,231,216	568,231,216
# walk skim calls	get_maz_pairs()	2,536,747	5,073,493	6,088,192	7,610,240
# bike skim calls	get_maz_pairs()	2,536,747	5,073,493	6,088,192	7,610,240
# TVPB Calls	TVPB	1,014,699	2,536,747	3,348,505	5,073,493

Based on the analysis in tables 2, 3 and 5, the test sizes of relevant methods in mandatory location, stop location, and trip mode choice models are summarized in Table 6.

Table 6: Test Scenarios & Test Sizes

Model Component	Relevant Method	Test Size
Mandatory tour mode choice	taz_skims()	68,374,080
Stop location choice	get_maz_pairs()	1,522,047,900
Trip mode choice 1	taz_skims()	568,231,216
	get_maz_pairs()	5,073,493
	best transit path	1,014,699
Trip mode choice 2	taz_skims()	568,231,216
	get_maz_pairs()	10,146,986
	best transit path	2,536,747
Trip mode choice 3	taz_skims()	568,231,216
	get_maz_pairs()	12,176,383
	best transit path	3,348,505
Trip mode choice 4	taz_skims()	568,231,216
	get_maz_pairs()	15,220,479
	best transit path	5,073,493

6. Test Results:

Table 7 is the data loading performance summary. In this test, population, land use, and walk and bike impedance data are loaded. The SANDAG model uses 80 skims, including 70 auto skims and 10 transit skims. In this test, only 6 skims are loaded including AM SOV, PM SOV, AM local bus, AM premium bus, PM local bus and PM premium skims. Data loading performs well, taking less than 30 seconds.

Table 7: Data Loading Performance

Test Item	Runtime (sec)
Load TAZ skims	16.3
Load TAP skims	10.5
Load network LOS	2.9
Total	29.7

Table 8 is a performance summary of each individual method by increasing test size from 10,000 to 50,000,000.

Table 8: Individual Method Performance

	Runtime (sec)
Test Size	** 10,000**
get_taz()	0.008
get_tap()	0.007
get_maz()	0.023
taz_skims()	0.004
tap_skims()	0.014
get_maz_pairs()	2.867
get_maz_tap_pairs()	0.084
get_taps_mazs()	0.118

Table 9 is a performance summary of each individual method per unit (10000 as one unit) derived from Table 8. Method taz_skims() consistently performs well. Method get_taps_mazs() also maintains a stable performance as test size increases. For all the other methods, as test size increases, the performance deteriorates.

Table 9: Individual Method Performance per Unit (10000)

	Runtime/10000 (sec)
	10,000
get_taz()	0.008
get_tap()	0.007
get_maz()	0.023
taz_skims()	0.004
tap_skims()	0.014
get_maz_pairs()	2.867
get_maz_tap_pairs()	0.084
get_taps_mazs()	0.118

Table 10 is a performance summary of methods that are relevant to each model component.

Table 10: Method Performance by Model Component

Model Component	Relevant Method	Test Size	Runtime (secs)	Runtime(hrs)
Mandatory tour mode choice	taz_skims()	68,374,080	24.01	0.01
Stop location choice	get_maz_pairs()	1,522,047,900
Trip mode choice scenario 1	taz_skims()	568,231,216	209.3	0.06
	get_maz_pairs()	5,073,493	35.6	0.01
	best transit path	1,014,699	23391.2	6.50
Total			23636.1	6.57
Trip mode choice scenario 2	taz_skims()	568,231,216	209.3	0.06
	get_maz_pairs()	10,146,986	83.4	0.02
	best transit path	2,536,747	killed at 12th hr
Total
Trip mode choice scenario 3	taz_skims()	568,231,216	209.3	0.06
	get_maz_pairs()	12,176,383	114	0.03
	best transit path	3,348,505	NA
Total
Trip mode choice scenario 4	taz_skims()	568,231,216	209.3	0.06
	get_maz_pairs()	15,220,479	161.4	0.04
	best transit path	5,073,493	NA
Total

7. Findings:

• Although only 6 out of the 80 skims are loaded in the test, data loading performs well taking less than 30 seconds.
• In mandatory tour location choice, taz_skims() calls performs well taking 24 seconds.
• In stop location choice, get_maz_paris() calls take ******, clearly a bottle neck.
• In trip mode choice, the TVPB calls takes 6.5 hours even in the most conservative trip mode scenario, clearly a bottleneck. We were not able to test the TVPB performance for the other three trip mode scenarios because the runtime was too long.
• The following findings are summarized from the individual method testing results:
  • As test size increases, the per 10000 unit runtime increase significantly for get_taz(), get_tap(), get_maz(), tap_skims(), and get_maz_tap_pairs(); per 10000 unit runtime fluctuates for get_maz_tap_pairs(); per 10000 unit runtime are stable for get_skims() and get_taps_mazs().
  • Method taz_skims() consistently performs well even when test size is large.
  • Method get_taps_mazs() maintains a stable performance as test size increases.
  • For all the other methods, as test size increases, the performance deteriorates.
  • The less than ideal performance of tap_skims(), in comparison with taz_skims(), is particularly interesting. If tap_skims() can be implemented in a way similar to taz_skims(), the overall ActivitySim performance could be improved.