Commit 935dc6a
committed
vertexfilter: Adjust encodeFilterColor alpha quantization
Because the reconstruction of alpha involves replicating the low bit
instead of replicating the high bit, for minimum error we should
quantize the value as if it had K bits instead of K-1, and drop the low
bit.
This improves the maximum error by up to 40% depending on the bit count;
the effect on average error is more muted and generally is around 10% or
lower.1 parent 24a4ced commit 935dc6a
1 file changed
+1
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1469 | 1469 | | |
1470 | 1470 | | |
1471 | 1471 | | |
1472 | | - | |
| 1472 | + | |
1473 | 1473 | | |
1474 | 1474 | | |
1475 | 1475 | | |
| |||
0 commit comments