Overflow handling in `statistics.mean()` and `statistics.fmean()`

[olliefr]

I'm not sure if the following should be considered a bug or a documentation issue.

In the example that follows, statistics.mean() and statistics.fmean() are called with the same floating-point values.

The former evidently takes some effort to avoid the overflow while the latter does not.

If this works as intended, shouldn't this be mentioned in the statistics docs in some way? Because judging from this (arguably limited) example, when given the same input mean() appears to handle pathological cases better than fmean().

Python 3.12.8 | packaged by Anaconda, Inc. | (main, Dec 11 2024, 16:31:09) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import statistics, sys

>>> statistics.mean([sys.float_info.max] * 2)
1.7976931348623157e+308

>>> statistics.fmean([sys.float_info.max] * 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/oliver/miniconda3/lib/python3.12/statistics.py", line 510, in fmean
    total = fsum(data)
            ^^^^^^^^^^
OverflowError: intermediate overflow in fsum

[AA-Turner]

cc @rhettinger

[skirpichev]

The former evidently takes some effort to avoid the overflow while the latter does not.

mean() just do exact math (with Fraction's) and then convert back to an appropriate input type (this time - to a float).

It's possible to use a more specialized algorithm (currently it does roughly fsum(data)/len(data)) for fmean() to avoid some overflow conditions, e.g. sumprod() works in your case:

>>> math.sumprod([sys.float_info.max]*2, [0.5]*2)
1.7976931348623157e+308

But this will be not without a speed penalty. I'm not sure it's appropriate.

[olliefr]

@skirpichev if I understood you correctly, this is what happens:

When I run statistics.mean([sys.float_info.max] * 2) the input to statistics.mean() is a list of floats which is cast to a list of Fractions before the mean is calculated. And since a fraction is just two (long) integers to represent the numerator and the denominator, the sum is successful because Python's (long) integers are limited only by available memory. Then the result gets divided by two and returned as a float?

[skirpichev]

Correct.

Then the result gets divided by two and returned as a float?

It's important, that it divided also exactly. So, result will be Fraction(sys.float_info.max) - and that will not overflow, when converted back to a float. (Conversion is using the true division of the numerator and the denominator.)

[olliefr]

Thank you for the detailed explanation, @skirpichev!

@rhettinger I'm not sure what can be done about it other than maybe adding a note to fmean() docs to say that for some edge cases it can be less robust to overflows than mean()? I'm leaving it to your professional judgement. Please feel free to just close this issue if no action needs to be taken.