vl53l0x: Perform 'abs' operation before scaling and dividing mcps dif… #10
Conversation
…f value If the computed value is negative, then it's important to flip the sign before doing the division as that will not have the same result on an unsigned value as it would on a signed value in the same bits. Signed-off-by: Keith Packard <[email protected]>
|
Thanks for this fix. This seems valid but the original code is extracted from a library provided by ST. |
There was a problem hiding this comment.
Advise from ST domain expert is to reject proposal:
"the fix is correct from mathematical perspective, but physically, xtalk can't be higher than signal otherwise status i invalid. Besides since function used for pulse standard deviation estimation isn't in the datasheet, ST can't guarantee perf on it".
@keith-packard Is that fine with you ?
|
Their comment doesn't make much sense to me; if they're so sure the difference will always be positive, then why is there an abs in this code at all? In any case, the only reason I saw this bug was because I was working on zephyrproject-rtos/zephyr#45179. If you want to leave the code as-is, that's entirely up to you. |
2 participants
…f value
If the computed value is negative, then it's important to flip the
sign before doing the division as that will not have the same result
on an unsigned value as it would on a signed value in the same bits.
Signed-off-by: Keith Packard [email protected]