Overflow error when coalescing memory logs

[JoeJimFlood]

Describe the bug
An ABM3 validation run failed due to the following error:

Traceback (most recent call last):
  File "C:\abm_runs\car\2035_1530\src\asim\simulation.py", line 24, in 
    sys.exit(run(args))
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\activitysim\cli\run.py", line 415, in run
    mem.consolidate_logs(state)
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\activitysim\core\mem.py", line 90, in consolidate_logs
    df["time"] = time_bin(
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\activitysim\core\mem.py", line 45, in time_bin
    return pd.to_datetime(bin, unit="s", origin="unix")
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\pandas\core\tools\datetimes.py", line 1067, in to_datetime
    values = convert_listlike(arg._values, format)
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\pandas\core\tools\datetimes.py", line 407, in _convert_listlike_datetimes
    return _to_datetime_with_unit(arg, unit, name, utc, errors)
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\pandas\core\tools\datetimes.py", line 512, in _to_datetime_with_unit
    arr = cast_from_unit_vectorized(arg, unit=unit)
  File "conversion.pyx", line 149, in pandas._libs.tslibs.conversion.cast_from_unit_vectorized
  File "<__array_function__ internals>", line 200, in round_
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\numpy\core\fromnumeric.py", line 3763, in round_
    return around(a, decimals=decimals, out=out)
  File "<__array_function__ internals>", line 200, in around
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\numpy\core\fromnumeric.py", line 3337, in around
    return _wrapfunc(a, 'round', decimals=decimals, out=out)
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\numpy\core\fromnumeric.py", line 57, in _wrapfunc
    return bound(*args, **kwds)
FloatingPointError: overflow encountered in multiply

Looking through the code, it would appear as though this is happening while combining each of the memory logs into the omnibus memory log. From what I can tell, the reported time in one of the the files is causing an overflow error. I saw that all of the needed output files had been created, so I suggested the modeler running the model run resume it after ActivitySim model, which worked just fine (the results were close to similar scenarios that had been run).

To Reproduce
I'm not completely sure how to reproduce it. I tried copying all of the logfiles into a separate folder and running a Python script that replicated what ActivitySim was doing, but that worked just fine (I still have the folder saved and can share it with anyone who would like to see it for debugging purposes).

Expected behavior
The memory logs should be combined into the omnibus memory log without any issue.

[aber-sandag]

I've just encountered the same error in a separate ABM3 run. I believe the source is a non-reproducible bug in the pandas to_datetime function, see discussion here:

pandas-dev/pandas#58419

pandas-dev/pandas#61022

This bug will be fixed in Pandas 3, but there are some workarounds mentioned in that discussion that could be applied in ActivitySim. The to_datetime function is called here: https://github.com/ActivitySim/activitysim/blob/main/activitysim/core/mem.py#L45

My error traceback:

  File "C:\abm_runs\aber\bike_model_test_2035nb_v2\src\asim\simulation.py", line 24, in 
    sys.exit(run(args))
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\activitysim\cli\run.py", line 415, in run
    mem.consolidate_logs(state)
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\activitysim\core\mem.py", line 90, in consolidate_logs
    df["time"] = time_bin(
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\activitysim\core\mem.py", line 45, in time_bin
    return pd.to_datetime(bin, unit="s", origin="unix")
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\pandas\core\tools\datetimes.py", line 1067, in to_datetime
    values = convert_listlike(arg._values, format)
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\pandas\core\tools\datetimes.py", line 407, in _convert_listlike_datetimes
    return _to_datetime_with_unit(arg, unit, name, utc, errors)
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\pandas\core\tools\datetimes.py", line 512, in _to_datetime_with_unit
    arr = cast_from_unit_vectorized(arg, unit=unit)
  File "conversion.pyx", line 149, in pandas._libs.tslibs.conversion.cast_from_unit_vectorized
  File "<__array_function__ internals>", line 200, in round_
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\numpy\core\fromnumeric.py", line 3763, in round_
    return around(a, decimals=decimals, out=out)
  File "<__array_function__ internals>", line 200, in around
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\numpy\core\fromnumeric.py", line 3337, in around
    return _wrapfunc(a, 'round', decimals=decimals, out=out)
  File "C:\Anaconda3\envs\asim_140\lib\site-packages\numpy\core\fromnumeric.py", line 57, in _wrapfunc
    return bound(*args, **kwds)
FloatingPointError: overflow encountered in multiply

[JoeJimFlood]

This issue came up again for us over the weekend. We're planning on starting our regional transportation runs this week so I'm going to write a memo for our application modelers on what to do if they encounter the error.

[jpn--]

Seems like the easiest solution here might be to add an option to not trace memory usage at all, rather than trying to hack around a bug in pandas. In "production" model runs, we don't usually care anymore about carefully monitoring memory usage.

[JoeJimFlood]

One thing I've considered was to add a Boolean setting that would determine whether or not to run this line. I was thinking about submitting a pull request for that, though maybe we'd want to include the option to not create other logs that aren't needed in application.