BUG: Pandas Overrides Default NumPy Function's Behavior

[ZenithClown]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

df = pd.DataFrame(data = {"requirement" : [10, 20, 30], "cost" : [4.5, 5.5, np.nan], "allocation" : [0.1, 0.5, 0.4]})

# the default behavior of np.prod() returns nan when encountered
# the alternate function np.nanprod() considers nan as 1 and returns the value
df["total"] = df.apply(lambda x : np.prod(x), axis = 1)

Issue Description

The pandas module overrides the default behavior of a numpy function like np.prod() with the internal default behavior of pd.prod() instead.

Expected Behavior

The default behavior must be preserved as this creates a confusion when referring the documentation of numpy vs the observed result.

Installed Versions

INSTALLED VERSIONS

commit : 2e218d1
python : 3.10.9.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : Intel64 Family 6 Model 140 Stepping 2, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_India.1252

pandas : 1.5.3
numpy : 1.23.5
pytz : 2022.7
dateutil : 2.8.2
setuptools : 65.6.3
pip : 22.3.1
Cython : None
pytest : 7.1.2
hypothesis : None
sphinx : 5.0.2
blosc : None
feather : None
xlsxwriter : 3.1.2
lxml.etree : 4.9.1
html5lib : None
pymysql : 1.4.6
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.10.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : 1.3.5
brotli :
fastparquet : None
fsspec : 2022.11.0
gcsfs : None
matplotlib : 3.7.0
numba : 0.56.4
numexpr : 2.8.4
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : 15.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.0
snappy :
sqlalchemy : 1.4.39
tables : 3.7.0
tabulate : 0.8.10
xarray : 2022.11.0
xlrd : 2.0.1
xlwt : None
zstandard : 0.19.0
tzdata : 2024.1

[Edricklam123]

Hi this looks like a numpy implementation issue for pandas dataframe instead.

It seems numpy treats pandas series differently to a numpy.ndarray, for your example, if you use
df["total"] = df.apply(lambda x : np.prod(x.values), axis = 1) or
df["total"] = df.apply(lambda x : x.prod(skipna=False), axis = 1) if you want to stick with pandas

Both of them will return the result that you expected because it passes a np array to np.prod(). Please consider adding the .values to ensure alignment of input output between pandas and numpy library.

In fact there are some feature in numpy does not support the pandas implementation, and there are also some features, like you said, have a different default parameter between the two.

An example would be:

df.iloc[0].std() # >> 4.0541131377075965
np.std(df.iloc[0]) # >> 3.510964967071019
df.iloc[0].std(ddof=0) # >> 3.510964967071019

The .std() function in pandas has default ddof=1 (a sample std), but that of numpy is 0.

[rhshadrach]

Thanks for the report. pandas is calling np.prod, which then differs back to pandas. This is standard NumPy behavior when acting on any non-NumPy object (such as a pandas Series). If you'd like a change in behavior here, it'd have to be in NumPy, but I don't believe it should be changed.

Nothing pandas can do here; closing.