Complete binary data handling overhaul

[davidrios]

Hello. I've implemented a complete binary data handling overhaul and applied it to m2 loading on my repository. File parsing and writing is mostly declarative with the rust type system, and it also has native handling of different file versions. I think this can also be applied to all crates, so I implemented it in a new workspace crate called wow-data. I also removed lots of useless testing code and comments. I think it's probably a good idea to incorporate it in the main repo.

Here's an example:

// declaring a versioned property:
#[derive(Debug, Clone, WowHeaderR, WowHeaderW)]
#[wow_data(version = M2Version)]
pub enum M2ParticleEmitterBlending {
    Classic {
        blending_type: u16,
        emitter_type: u16,
    },

    #[wow_data(read_if = version >= M2Version::TBC)]
    Later {
        blending_type: u8,
        emitter_type: u8,
        particle_color_index: u16,
    },
}

#[derive(Debug, Clone, WowHeaderR, WowHeaderW)]
#[wow_data(version = M2Version)]
pub struct M2ParticleEmitterHeader {
    pub flags: M2ParticleFlags,
    pub position: C3Vector,
    pub texture_index: u16,
    pub model_filename: WowCharArray,

    #[wow_data(versioned)]
    pub blending_type: M2ParticleEmitterBlending,
}

This is all the code that's needed to define the structs, parsing and writing is automatically implemented using the derives, but it's also completely customizable and extendable. This is all the code that's need to parse and write now:

// any `Read`er will do
let header: M2ParticleEmitterHeader = reader.wow_read()?;
// alternatively:
let header = M2ParticleEmitterHeader::wow_read(&mut reader)?;
// writing:
writer.wow_write(&header)?;

Example of the entirety of the parsing code for the model data now, which reads everything according to https://wowdev.wiki/M2: https://github.com/davidrios/warcraft-rs/blob/refactor-data-handling/file-formats/graphics/wow-m2/src/model.rs

Main implementation of shared types and traits: https://github.com/davidrios/warcraft-rs/blob/refactor-data-handling/file-formats/wow-data/src/types.rs
Derive procmacros : https://github.com/davidrios/warcraft-rs/blob/refactor-data-handling/file-formats/wow-data-derive/src/lib.rs

[danielsreichenbach]

This is definitely a solid refactor. Considering the quirky nature of the WoW formats, I am seeing a few loose ends - from a quick birds eye view - here that I consider showstoppers:

• we do need chunk awareness, and versioning on chunk level
• ordering of chunks is not guaranteed
• how would we handle unknown chunks? (since we eventually would want to expand to current retail M3 format at some point)

If you want to continue with this, I would definitely be interested in seeing more!

[danielsreichenbach]

I am wondering if using "nom" would be better because it seems to be more suitable for the kind of chunk soup we do find in WoW formats. I could also see a combo of nom and the derive macro approach working really well here.

[davidrios]

That nom package is definitely interesting, and I think it will help. I can also easily see the system I created being adapted to that, the versioning handling logic is very suited for that kind of thing. I'll be continuing this development on my side anyway, so I propose I start working on the main repo to not have a constant fork, of course submitting PRs just in case. There's really only one thing that's different now, and that is the trimmed debug output that I implemented and you didn't backport last time. I really want this implemented, it's really annoying to constantly having thousands of lines of debug output while I'm developing, especially since it's really easy to see if the data is correct by inspecting just the first few elements of an array. I can definitely accept that it's disabled rather than enabled by default, but it being there is something that is non negotiable for me. What do you think?

[davidrios]

Also, I will be deleting all useless comments and tests. Correct me if I'm wrong, but from what I can see most of the code from this package seems to be generate by AI, and it generates a lot of trash. We should really put a gemini.md/claude.md/etc somewhere with some guidelines that prevent that kind of thing. Comments repeating exactly what's right beneath it in code form are just the worst, it's just in-built tech debt and I really don't want to see them

[davidrios]

We should also have a testing system that uses real files, instead of using half-baked manually crafted ones. Of course we can't use game files for copyright reasons, but as we implement the parsing and writing we can easily create real files with a similar structure as the ones from the game and use those for subsequent tests.
And about conversions, I won't be touching those, as that is a very complex and specialized use-case. The system I created already has conversion built-in, there's an example in wow-data, and it's relatively easy to do, so that's probably better suited to be done on an ad-hoc basis by anyone who's actually using said conversion. We just need to provide some documentation with guidance and leave that to other contributors.