Decimal strict mode should prevent Decimal("0.89") == 0.89

[timkay]

Bug report

Bug description:

Mixing Decimals and floats will often get you the wrong answer. By default, the Decimal library allows such behavior:

>>> import decimal
>>> decimal.Decimal(0.1)
Decimal('0.1000000000000000055511151231257827021181583404541015625')
>>> decimal.Decimal("0.1") < 0.1
True
>>> decimal.Decimal("0.1") == 0.1
False

Fortunately, you can turn on floating point strict mode, where mixing Decimals and floats is prohibited:

>>> decimal.getcontext().traps[decimal.FloatOperation] = True
>>> decimal.Decimal(0.1)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
decimal.FloatOperation: [<class 'decimal.FloatOperation'>]
>>> decimal.Decimal("0.1") < 0.1
Traceback (most recent call last):
  File "<console>", line 1, in <module>
decimal.FloatOperation: [<class 'decimal.FloatOperation'>]

HOWEVER, for some reason, == and != are allowed in floating point strict mode, and produce wrong answers:

>>> decimal.Decimal("0.1") == 0.1
False
>>>

When floating point strict mode is on, why is == and != still allowed?

    if isinstance(other, float):
        context = getcontext()
        if equality_op:
            context.flags[FloatOperation] = 1
        else:
            context._raise_error(FloatOperation,
                "strict semantics for mixing floats and Decimals are enabled")
        return self, Decimal.from_float(other)

This code would be better without the if equality_op::

    if isinstance(other, float):
        context = getcontext()
        context._raise_error(FloatOperation,
            "strict semantics for mixing floats and Decimals are enabled")
        return self, Decimal.from_float(other)

Why are == and != allowed in floating point strict mode?

CPython versions tested on:

3.12

Operating systems tested on:

Linux

[skirpichev]

you can turn on floating point strict mode, where mixing Decimals and floats is prohibited

Not at all:
"If the signal is not trapped (default), mixing floats and Decimals is permitted in the Decimal constructor, create_decimal() and all comparison operators. Both conversion and comparisons are exact. Any occurrence of a mixed operation is silently recorded by setting FloatOperation in the context flags. Explicit conversions with from_float() or create_decimal_from_float() do not set the flag.

Otherwise (the signal is trapped), only equality comparisons and explicit conversions are silent. All other mixed operations raise FloatOperation."

HOWEVER, for some reason, == and != are allowed in floating point strict mode, and produce wrong answers:

Why do you think it's a wrong answer?! 1/10 != (binary approximation of)0.1; the 0.1 can't be represented exactly as binary floating-point number. See https://docs.python.org/3/tutorial/floatingpoint.html#floating-point-arithmetic-issues-and-limitations

So, current behaviour seems to be consistent and well documented. Your only argument against is wrong. And this will break backward compatibility.

To change things you need more arguments. Probably, this should be discussed first on https://discuss.python.org/

[mdickinson]

for some reason, == and != are allowed in floating point strict mode

There's a more general design principle at work here (though it's not one I think I've seen articulated clearly in the docs), not related to float or Decimal specifically: as a rule, an == comparison between two hashable objects shouldn't raise an exception. The reason is that equality checks are performed implicitly and unpredictably in hash-based collections like dict and set.

If == raised as you suggest, then it would be possible to create the set {Decimal("0.47013"), 8946670875749133.0}, but creating a set {Decimal("0.47012"), 8946670875749133.0} would raise (because the hashes of the two elements in the set match, and so an equality test would be invoked to determine whether the elements are actually different, and that equality test would raise). I think that would be rather surprising.

>>> from decimal import Decimal
>>> s = {Decimal("0.47012"), 8946670875749133.0}
>>> list(map(hash, s))
[8946670875749133, 8946670875749133]

[skirpichev]

There's a more general design principle at work here (though it's not one I think I've seen articulated clearly in the docs), not related to float or Decimal specifically: as a rule, an == comparison between two hashable objects shouldn't raise an exception.

Perhaps, most close to documenting it is a quote from here: "When no appropriate method returns any value other than NotImplemented, the == and != operators will fall back to is and is not, respectively."

It doesn't say why this default does exists, however. But I'm not sure it worth.

I think this can be closed.

[timkay]

There's a more general design principle at work here

The general design principle is that floating point strict mode should prevent coders from making mistakes. Sure it's not always a mistake to say Decimal(0.89), if that's what you really intended, but that's almost never what people really want.

>>> Decimal(0.89)
Decimal('0.89000000000000001332267629550187848508358001708984375')

Decimal has lots of use cases, and a common one is to handle dollars-and-cents. If you say

account.balance < amount

You can run into trouble because

>>> Decimal("0.89") < 0.89
True

This sort of problem is flagged by turning on floating point strict mode, and the runtime will flag that statement as incorrect.

The exact same argument applies to equality. For many use cases, equality between Dollar and float is a coding error, and floating point strict mode should catch it.

The idea that things like sets use equality implicitly is interesting. Problem is, most people would be surprised by

>>> 0.89 in set([Decimal("0.89")])
False

What is the use case where equality makes sense between Decimal and float?

As for breaking changes, we could introduce a new flag, StrictMode, which would do what FloatOperation does, but includes equality.

[skirpichev]

What is the use case where equality makes sense between Decimal and float?

When they are mathematically equal.

>>> import decimal
>>> decimal.Decimal(0.1) == 0.1
True
>>> decimal.Decimal(0.89) == 0.89
True

We don't loose anything, unless you apply context settings with insufficient precision:

>>> (+decimal.Decimal(0.1)) == 0.1
False
>>> decimal.Decimal(0.1).as_integer_ratio()
(3602879701896397, 36028797018963968)
>>> (0.1).as_integer_ratio()
(3602879701896397, 36028797018963968)
>>> (+decimal.Decimal(0.1)).as_integer_ratio()
(1000000000000000055511151231, 10000000000000000000000000000)
>>> decimal.getcontext().prec=100
>>> (+decimal.Decimal(0.1)).as_integer_ratio()
(3602879701896397, 36028797018963968)

we could introduce a new flag, StrictMode, which would do what FloatOperation does, but includes equality.

Mark remind us above why we can't raise an exception here.

So, in this case you suggest to fall back on is and set 0.1 != decimal.Decimal(0.1). What we gain here?

[timkay]

I wasn't suggesting that we fall back on is. I was responding to a previous point that equality is used implicitly sometimes, and that's why equality is white listed even in floating point strict mode. But it doesn't make any sense, as that example shows.

To your point, I ask, what is the use case for saying Decimal(0.1)? The correct use is Decimal("0.1").

[skirpichev]

I wasn't suggesting that we fall back on is.

Then what? Currently Decimal's comparison method does implicit conversion and then do comparison. We have options: 1) implement comparison method in some other way (how?) or 2) just return NotImplemented - then Python will fall back to is.

If you rule out 2) - please explain how we should implement 1) instead.

what is the use case for saying Decimal(0.1)?

To convert losslessly binary floating-point number to it's Decimal equivalent. Or in other words: to construct Decimal instances from some binary fractions ($N/2^M$).

The correct use is Decimal("0.1").

It depends. To achieve the previous goal e.g. for 0.1 - the correct use will be something like:

>>> x = 0.1
>>> with localcontext() as ctx:
...     ctx.prec = 55  # assuming IEEE doubles
...     n, d = x.as_integer_ratio()
...     a = Decimal(n)/d
...     
>>> a  # == Decimal(x)
Decimal('0.1000000000000000055511151231257827021181583404541015625')

Using a string in the Decimal constructor - the correct way to enter a decimal fraction.

[tim-one]

Elaborating on what @mdickinson said, there's an even more general design principle at work: starting with the introduction of the classes in the datetime module, the intent has been that == and != never raise an exception when applied to objects of different types. That was late in Python 2's life, and was in a sense a prototype for a more sweeping rule in Python 3. The idea being that equality is a much simpler concept than ordering (although don't tell philosophers that 😉), and e.g. it's just plain dead obvious that datetime.dateime.now() != K for any integer K.

It was appreciated at the time that this would simplify reasoning about sets and dicts, but the latter isn't really what drove it. Instead the latter was taken as confirmation of that it was a Good Idea in general.

[skirpichev]

Other than raising an exception (that, I think, excluded by Mark's arguments) - the only option is to make a context-dependent equality.

I think this is silly, we might end up e.g. with a set, that after changing context settings - has equal elements.

[tim-one]

@mdickinson wrote:

If == raised as you suggest, then it would be possible to create the set {Decimal("0.47013"), 8946670875749133.0}, but creating a set {Decimal("0.47012"), 8946670875749133.0} would raise

Perhaps when FloatOperation was introduced, mixed equality comparison could have raised too on "practicality beats purity" grounds, but now there's also backwards compatibility to preserve. As Mark's example shows, behavior in existing code could change if FloatOperation changed to include raising on mixed equality. A simpler example illustrates simpler potential breakage:

f = 0.0
d = Decimal(0)
if f == d:
    # whatever

That works fine today regardless of FloatOperation state. Changing it to raise if FloatOperation is enabled would be a breaking change.

In any case, behavior that's working as designed and documented isn't properly called "a bug" in the issue tracker. I suggest changing it to a feature request, and instead ask for a new exception to be introduced. one that acts like FloatOperation but also raises on mixed (in)equality comparisons. I'd probably be +0 on that, but, as Mark's example shows, it would come with its own surprises.

[timkay]

@skirpichev, I asked what is the use case for Decimal(0.1), and you replied, that it's to load a Decimal instance with the value 0.1000000000000000055511151231257827021181583404541015625. That's not a use case. What would be a situation where doing so would be useful?

@mdickinson points out that {Decimal("0.47012"), 8946670875749133.0} would raise an exception. I ask you for a use case, where mixing Decimal and float is useful.

There is a reason that Decimal exists, and it is to allow decimal computations without worrying about floating point rounding. If you care about that then you would use Decimal and avoid float. Mixing the two leads to incorrect answers. Raising on equality would prevent that.

Put a different way, IF you want to mix Decimal and float, then why would you want to prevent comparing Decimal < float ? What is the purpose of floating point strict mode? If you want to mix Decimal and float, don't turn on strict mode. The compromise that is strict mode now, it isn't useful.

[tim-one]

@timkay:

I asked what is the use case for Decimal(0.1), and you replied, that it's to load a Decimal instance with the value 0.1000000000000000055511151231257827021181583404541015625. That's not a use case.

It's not your use case, but I've frequently done such things, because the ability is quite useful for people analyzing floating-point error propagation. Decimal isn't their focus. Decimal is just a tool that makes it much easier for non-experts to "see" what's really happening in their float code. For example, I've frequently used this in public replies to Stackoverflow questions about float behavior.

Read the room here? For reasons of backward compatibility alone, I expect there's essentially no chance this will change. But I think you could make a decent case for adding a new exception (say, FloatOperationEx or FloatOperationStrict) that acts as you want.

The compromise that is strict mode now, it isn't useful.

It doesn't do everything you want, but does do a great deal of you want. It does do everything I want of it (and I do use it when appropriate), but then I'm always acutely aware of how mixing numeric types works in Python. "Isn't useful" is too strong.