BUG: Pandas squashes 1-dimensional Numpy array with shape (1,) down to a 0-dimensional array

[sunghjung3]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

df = pd.DataFrame(columns=['x'])
arr0 = np.array([1., 2.])
df.loc[0, "x"] = arr0
print(df.loc[0, "x"].shape)  # prints (2,)
arr1 = np.array([1.])
df.loc[1, "x"] = arr1
print(df.loc[1, "x"].shape)  # prints (). Should print (1,)

Issue Description

When a 1-dimensional Numpy array that only has 1 entry is stored to a DataFrame, Pandas seems to automatically make it a 0-dimensional array.

This is extremely inconvenient when the entries of both df.loc[0, "x"] and df.loc[1, "x"] are accessed later like df.loc[1, "x"][i] just for it to cause Numpy to raise an IndexError because df.loc[1, "x"] is now a 0-dimensional array, not a 1-dimensional array.

NOTE: This was not an issue in Pandas 2.0.0, but it is in version 2.2.2. I am not sure exactly which version in between this was introduced.

FYI: If a higher dimensional array with 1 entry is stored (e.g. np.array([[1.]])), then the shape is correctly preserved. It is only an issue when the array is 1-dimensional.

Expected Behavior

print(df.loc[1, "x"].shape) should print (1,).

Installed Versions

INSTALLED VERSIONS

commit : d9cdd2e
python : 3.11.9.final.0
python-bits : 64
OS : Linux
OS-release : 4.18.0-477.15.1.el8_8.x86_64
Version : #1 SMP Wed Jun 28 15:04:18 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.2
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0
setuptools : 70.3.0
pip : 24.0
Cython : None
pytest : 8.2.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.4
IPython : 8.26.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.6.1
gcsfs : None
matplotlib : 3.9.1
numba : 0.60.0
numexpr : 2.10.0
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.14.0
sqlalchemy : None
tables : 3.9.2
tabulate : None
xarray : None
xlrd : None
zstandard : 0.22.0
tzdata : 2024.1
qtpy : None
pyqt5 : None

[Animesh-Shukla]

can I take this issue ?

[Animesh-Shukla]

take

[Siddharth-Latthe-07]

@sunghjung3 I think it seems to be a regression in recent versions of pandas. It appears that a 1-dimensional numpy array with a single entry is being automatically converted to a 0-dimensional array when stored in a DataFrame.
Temporary Workaround:

1. Until the issue is resolved in future pandas releases, you can explicitly check and reshape the arrays as needed.

import pandas as pd
import numpy as np

df = pd.DataFrame(columns=['x'])
arr0 = np.array([1., 2.])
df.loc[0, "x"] = arr0
print(df.loc[0, "x"].shape)  # prints (2,)

arr1 = np.array([1.])
df.loc[1, "x"] = arr1 if arr1.shape != () else arr1.reshape(1,)
print(df.loc[1, "x"].shape)  # prints (1,)

Hope this helps,
Thanks

[sunghjung3]

@Siddharth-Latthe-07 I've already tried this, and it doesn't work. It still prints (). Please try it yourself.

[Siddharth-Latthe-07]

oh, I see then can we create a custom function to ensure that the shape of the numpy array is preserved when you store it in the DataFrame. This function will explicitly check the shape of the array and reshape it if necessary before storing it in the DataFrame.
Thoughts ? @sunghjung3

[Siddharth-Latthe-07]

something like this:-

def safe_loc_assign(df, index, column, value):
    if isinstance(value, np.ndarray) and value.shape == ():
        value = value.reshape(1,)
    df.at[index, column] = value

[sunghjung3]

@Siddharth-Latthe-07 I don't get the logic of your function.

I'm assuming that we would call this function like safe_loc_assign(df, 1, "x", arr1). And inside this function, it checks for value.shape == (). But the shape of the value argument (which is arr1) is not (), so this if statement isn't even utilized.

And the result is still the same; it prints ().

Also, even if I change this if-statement to value.shape == (1,), it still doesn't work.

[rhshadrach]

Result of a git-bisect:

a1c6fe0620f4322dc9b6c2265e3602c373f2f06c is the first bad commit
commit a1c6fe0620f4322dc9b6c2265e3602c373f2f06c
Author: MeeseeksMachine <39504233+meeseeksmachine@users.noreply.github.com>
Date:   Wed Apr 26 20:28:43 2023 +0200

    Backport PR #52906 on branch 2.0.x (DEPS: Address numpy deprecation of len 1 arrays assignment) (#52933)
    
    Backport PR #52906: DEPS: Address numpy deprecation of len 1 arrays assignment
    
    Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>

 pandas/core/indexing.py                     |  7 +++++++
 pandas/core/internals/base.py               |  4 ++++
 pandas/core/internals/blocks.py             |  4 +++-
 pandas/io/parsers/base_parser.py            | 10 +++++++++-
 pandas/tests/groupby/test_groupby_dropna.py |  2 +-
 5 files changed, 24 insertions(+), 3 deletions(-)
bisect found first bad commit

PR: #52906

cc @mroeschke

[sunghjung3]

@rhshadrach Has this been fixed?

[rhshadrach]

No - and it's not clear what the resolution will be.