meshletcodec: Revise vertex reference encoding for more compression#1012
Merged
meshletcodec: Revise vertex reference encoding for more compression#1012
Conversation
Instead of encoding the vertex reference codes as 0/1/2/4 bytes, we encode them as 0/1/2/3 bytes. 4 byte reference encoding is very wasteful; even on very large meshes it's almost never seen, especially if the applications follow the best practice of rebasing the meshlet vertex references (which is usually free because the offset can be folded into the rest of the packing). Because we need the encoding to be fully general, we need to be able to encode 4-byte deltas if we encounter them. For that, we limit the encoding such that if one delta requires 4 bytes (or all 4 deltas require 3 bytes), we encode all deltas in this group using 4 bytes. While this could result in larger encodings in rare cases, usually the savings make up for it.
Instead of packing two-bit codes sequentially into the control byte, we pack the low bits into low nibble, and high bits into high nibble. This does not affect the encoded size, but it improves the statistics for general purpose compressors when meshletcodec output is compressed further, resulting in ~0.1 bits/triangle gains for free.
While this is not fully general and won't support meshlets above 128v/t, practically speaking it's sufficient, and saves a byte in the meshlet stream encoding - making packed data size a little closer to raw data size. Of course, applications can choose any framing or none at all depending on what works best.
zeux
changed the title
Change the code to be more aligned with table building: the code is always 2-bit, but the length that's recovered from it is 0-4; additionally, with this we don't need kLengths lookup because it's an identity.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
The data format continues to be highly experimental. Until a library point release, data compatibility is not guaranteed.
Vertex reference deltas should almost never need 32 bits; even if source references are 32-bits, any reasonable locality preprocessing will rarely result in long distance jumps. However, for some types of preprocessing, deltas that don't fit into 16 bits can be more common. We adjust the encoding to use 3-byte references in a group, unless all 4 references need the full width, in which case they are upgraded to 4 bytes to allow arbitrary data to still be represented.
Also, the vertex control byte now encodes low control bits separately from high control bits. On typical data that reduces the integer range (most bytes have zero high nibble) which improves entropy compression when compressing the encoded output with a general purpose compressor. Together, the changes get ~0.1 bits/triangle back both pre- and post-deflate on average.
The demo code is now also using a slightly smaller header for the stream, however this part is always up to the application and isn't always required.
This contribution is sponsored by Valve.