Skip to content
This repository was archived by the owner on Jul 8, 2025. It is now read-only.

interpolate_df() changes datetime unit of date column #212

@capelastegui

Description

@capelastegui

We have seen forecasts crashing with the following error:

  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/anticipy/forecast.py", line 928, in run_forecast
    return run_forecast_single(df_y,
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/anticipy/forecast.py", line 1386, in run_forecast_single
    df_data = pd.concat(l_df_data, sort=False, ignore_index=True)
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/reshape/concat.py", line 393, in concat
    return op.get_result()
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/reshape/concat.py", line 680, in get_result
    new_data = concatenate_managers(
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/internals/concat.py", line 189, in concatenate_managers
    values = _concatenate_join_units(join_units, copy=copy)
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/internals/concat.py", line 486, in _concatenate_join_units
    concat_values = concat_compat(to_concat, axis=1)
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/dtypes/concat.py", line 132, in concat_compat
    return cls._concat_same_type(to_concat_eas)
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/arrays/datetimelike.py", line 2251, in _concat_same_type
    new_obj = super()._concat_same_type(to_concat, axis)
  File "/home/jupyter_user/workspace/nsa.notebooks-voice/virtual/lib/python3.9/site-packages/pandas/core/arrays/_mixins.py", line 230, in _concat_same_type
    return super()._concat_same_type(to_concat, axis=axis)
  File "arrays.pyx", line 190, in pandas._libs.arrays.NDArrayBacked._concat_same_type
ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 141 and the array at index 1 has size 616```

We have traced this back to a bug in interpolate_df() that changes the datetime unit of the date column. In this case, the input data had a date column with dtype datetime64[us], and interpolate_df() returned a date with dtype datetime64[ns]. The crash happened when concatenating both series together.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions