Tight decoding of 10-bit-per-pixel HDR framebuffer fails

[bk138]

To repro:

In .config/sway/config:

output HEADLESS-1 {
pos 0,0
mode 2560x1440@75Hz
render_bit_depth 10
}

Then WLR_BACKENDS=headless sway and WAYLAND_DISPLAY=wayland-1 wayvnc 0.0.0.0 5900 -L debug.

In libvncclient, getting "Incorrect data received from the server", other encodings work.

Moved here from bk138/multivnc#286