Skip to content

New tooling: dataproc library + resdat packer#852

Draft
lhearachel wants to merge 7 commits intopret:mainfrom
lhearachel:dataproc
Draft

New tooling: dataproc library + resdat packer#852
lhearachel wants to merge 7 commits intopret:mainfrom
lhearachel:dataproc

Conversation

@lhearachel
Copy link
Collaborator

@lhearachel lhearachel commented Dec 11, 2025

There are two pieces to this PR:

  1. Commit the dataproc library source code into the tree, which is a successor to datagen and our use of rapidjson.
  2. A C utility to pack resdat and cldat files from an input JSON structure.

dataproc

This is a middleware library that acts as a director on top of a generic interface for accessing elements from a plain-text data structure. Right now, the only implementation is for JSON, but any language could theoretically be used, if the appropriate interface is fully implemented. yyjson serves as the back-end for processing JSON files and is vendored into the source tree. The vendored files include a patch that I applied on top of the base source code to add support for spanning offsets on each value-node in the parsed document.

I'm open to discussion about pulling yyjson from upstream and storing my changes as a patch file instead.

The library comes with smart reporting of errors for free. For instance, given the following function call:

get_u8(data, ".base_stats.defense");

And the following data file snippet:

{
    "base_stats": {
        "hp": 90,
        "attack": 92,
        "defense": 512
    }
}

The following error will be written to the console:

data.json:5:20: error: expected '.base_stats.defense' to be an integer in the range [0,255]
    5 |         "defense": 512,
      |                    ^~^

This behavior can be suppressed during initialization. More details are in the README.md.

resdat

This is a small tool that illustrates some basic usage of the dataproc library. It takes a JSON file and processes it into the packed binary structures called resdat and cldat in the source code. These binary files are used to load "packs" of resources in unison and assign them to various sprite templates.

This PR also includes an unpack of the resdat and cldat files used by the party menu application, as a reference for their structure. I will unpack the other resdat and cldat files as part of this PR, but I wanted to get this up for review in case folks wanted to discuss ahead of time.

I'll also be asking for help porting the existing datagen tools to this new library once it's merged.

Copy link
Contributor

@Gudf Gudf Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: It would be nice to have generated constants to refer to the cldat entries.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to defer that as part of a follow-up PR after all of these files have been unpacked.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went ahead and wound up doing this anyways. Not too hard.

@lhearachel lhearachel force-pushed the dataproc branch 2 times, most recently from 10abefb to 164b9d1 Compare December 15, 2025 04:57
@lhearachel
Copy link
Collaborator Author

I've included unpacked data files for the resdat and cldat used by the Party Menu, Shop Menu, Town Map, and Poffin Case. All others require unpacking NARCs that we have yet to unpack.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants