Skip to content

f32 -> u8 conversion should multiply by 256-ε instead of by 255 #13

@nigeltao

Description

@nigeltao

The Rust code has SIMD and non-SIMD versions of this conversion
(255.0 * y) as u8
from an f32 in the range [0.0, 1.0] to a u8 in the range [0, 255].

Due to rounding errors, this sometimes means that a pixel well inside a glyph ends up with an 0xfe value instead of 0xff, since its floating point value is slightly less than 1.

In my Go port, I embiggened the constant from 255.0 to 255.99998, or 0x437fffff as a float32 bit pattern, which eliminated these central "0xfe"s. See the comment at the bottom of google/font-go@7c31afd

I think that the Rust code should do the same.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions