-
Notifications
You must be signed in to change notification settings - Fork 25
Open
Description
When using negative numbers with 'trunc' rounding mode, rounding produces results that don't match the documentation in this part:
In short, the fractional part of the signed number x that is not supported, is discarded.
Example code:
from fxpmath import Fxp
FXP_FMT_IN = Fxp(None, signed=True, n_word=24, n_frac=21, rounding="trunc", overflow="wrap")
FXP_FMT_OUT = Fxp(None, signed=True, n_word=12, n_frac=11, rounding="trunc", overflow="wrap")
raw_val = -2095500 #0xe00674
fxp_val = Fxp(raw_val, raw=True, like=FXP_FMT_IN)
out_raw_val = raw_val >> 10
out_fxp_val = fxp_val.like(FXP_FMT_OUT)
out_raw_val and out_fxp_val.raw() should be the same, but are not.
I suggest removing the quoted statement from the documentation.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels