Skip to content

Quantity.__format__() can sometimes produce wrong resultsΒ #11

@llucax

Description

@llucax

What happened?

The main issue seems to be that we are doing the calculation about which exponent to use based on the base_value, but then when we convert the base_value to string, the float string conversion does some rounding, that could change how many significant digits we end up with.

For example: if base_value for power is 999.9999850988388, then we consider it is smaller than the next possible exponent (1kW) so we use just the base unit (W), but when doing str(base_value), this number ends up displayed as "1000", so we return "1000 W" when we should have returned "1 kW". This specific case is now resolved, but there are other that prevent, in particular, for this invariant to always be true: Power.from_string(str(power)) == power.

What did you expect instead?

Power.from_string(str(power)) == power always.

Affected version(s)

No response

Affected part(s)

Core components (data structures, etc.) (part:core)

Extra information

The current implementation to handle these weird corner cases is quite complex, and it doesn't seem to be handling all of them. An, less efficient, alternative would be to just do a string round-trip before calculating what's the most appropriate exponent to use, so convert to string, convert again to float, and only then check which exponent is more appropriate, then convert again to string.

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority:lowThis should be addressed only if there is nothing else on the tabletype:bugSomething isn't working

    Type

    No type

    Projects

    Status

    To do

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions