Skip to content

'trunc' rounding behaviour does not match documentation #100

@rkrajnc

Description

@rkrajnc

When using negative numbers with 'trunc' rounding mode, rounding produces results that don't match the documentation in this part:

In short, the fractional part of the signed number x that is not supported, is discarded.

Example code:

from fxpmath import Fxp

FXP_FMT_IN  = Fxp(None, signed=True, n_word=24, n_frac=21, rounding="trunc", overflow="wrap")
FXP_FMT_OUT = Fxp(None, signed=True, n_word=12, n_frac=11, rounding="trunc", overflow="wrap")

raw_val = -2095500 #0xe00674
fxp_val = Fxp(raw_val, raw=True, like=FXP_FMT_IN)

out_raw_val = raw_val >> 10
out_fxp_val = fxp_val.like(FXP_FMT_OUT)

out_raw_val and out_fxp_val.raw() should be the same, but are not.

I suggest removing the quoted statement from the documentation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions