BUG: ewm with adjust=False and ignore_na=False does not properly take NaNs into account

[btglau]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
df = pd.DataFrame({'B': [0, 1, 2, None, 4]})
     B
0  0.0
1  1.0
2  2.0
3  NaN
4  4.0
df.ewm(alpha=2/3, adjust=False, ignore_na=False).mean()
          B
0  0.000000
1  0.666667
2  1.555556
3  1.555556
4  3.650794

Issue Description

According to the documentation, for adjust=False the formula for the recursive mean should be:

y_t = (1-alpha)y_{t-1} + alpha x_t

And for ignore_na=False the absolute positions should yield a factor of (1-alpha)^2 and alpha, which does not give the same answer as the invocation above:

alpha = 2/3
yprev = 1.555556
xt = 4
(1-alpha)**2*yprev + alpha*xt
2.839506222222222

In contrast, ignore_na=True does give the right value:

alpha = 2/3
yprev = 1.555556
xt = 4
(1-alpha)*yprev + alpha*xt
3.185185333333333

Expected Behavior

With ignore_na=False, the correct value in index 4 should be 2.839506222222222, not 3.650794.

Installed Versions

INSTALLED VERSIONS

commit : bdc79c1
python : 3.11.5.final.0
python-bits : 64
OS : Darwin
OS-release : 23.4.0
Version : Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.1
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.8.2
setuptools : 68.2.2
pip : 23.3.1
Cython : 3.0.10
pytest : 7.4.0
hypothesis : None
sphinx : 5.0.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.3
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : 8.20.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.2
bottleneck : 1.3.7
dataframe-api-compat : None
fastparquet : None
fsspec : 2023.10.0
gcsfs : None
matplotlib : 3.8.4
numba : 0.59.1
numexpr : 2.8.7
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 14.0.2
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : 2023.10.0
scipy : 1.13.0
sqlalchemy : 2.0.25
tables : 3.9.2
tabulate : 0.9.0
xarray : 2023.6.0
xlrd : 2.0.1
zstandard : 0.22.0
tzdata : 2023.3
qtpy : 2.4.1
pyqt5 : None

[gboeker]

I have confirmed this bug exists on the main branch of pandas.

>>> import pandas as pd
>>> df = pd.DataFrame({'B': [0, 1, 2, None, 4]})
>>> df
     B
0  0.0
1  1.0
2  2.0
3  NaN
4  4.0
>>> df.ewm(alpha=2/3, adjust=False, ignore_na=False).mean()
          B
0  0.000000
1  0.666667
2  1.555556
3  1.555556
4  3.650794

>>> alpha = 2/3
>>> yprev = 1.555556
>>> xt = 4
>>> (1-alpha)**2*yprev + alpha*xt
2.839506222222222

>>> alpha = 2/3
>>> yprev = 1.555556
>>> xt = 4
>>> (1-alpha)*yprev + alpha*xt
3.185185333333333

[pmhatre1]

I believe it's Nan causing an issue here. Shall I add an if statement to consider Nan as 0 or shall we throw an error here that Nan shouldn't be used to calculate in homogeneous_func() in rolling.py?

Also I was trying to enter in func(x, start, end, min_periods, *numba_args) in rolling.py to check how the values are calculated but could not get in this function? Any help in this regards.

[btglau]

According to the documentation, if ignore_na=False then NaN should not be considered numerically, but should factor in for calculating the weights. It seems like your first proposal is in line with the documented behaviour.

[torext]

Apologies if I'm being dense, but to me it seems Pandas is doing the correct thing here. When the documentation refers to "weights" there it's clearly referring to non-normalised weights; hence those weights of $(1-\alpha)^2$ and $\alpha$ need to be normalised before being applied (i.e. divided by their sum). Indeed below a manual computation of my reading of the docs matches what Pandas does (both for Series and DataFrames, just to be sure):

In [1]: x = pd.Series([0, 1, 2, None, 4])

In [2]: alpha = 2 / 3

In [3]: w = pd.Series((1 - alpha)**np.arange(5)[::-1])

In [4]: w.iloc[1:] *= alpha

In [5]: x.ewm(alpha=alpha, adjust=False, ignore_na=False).mean()
Out[5]: 
0    0.000000
1    0.666667
2    1.555556
3    1.555556
4    3.650794
dtype: float64

In [6]: x.mul(w / w.where(x.notna()).sum()).sum()
Out[6]: np.float64(3.6507936507936507)

In [7]: x.to_frame().ewm(alpha=alpha, adjust=False, ignore_na=False).mean()
Out[7]: 
          0
0  0.000000
1  0.666667
2  1.555556
3  1.555556
4  3.650794