Package schema

[jbragg]

Resolves an encountered problem where some scored data was set to a default value (number of tokens = 0), the field was dropped during serialization (due to exclude_defaults=True in model_dump_json), the HuggingFace dataset inferred a null value, and de-serialization in the leaderboard app failed (the number of tokens field expected an int).

This PR

• Adds functionality to produce a fixed schema file (dataset_infos.json) which after being uploaded to the root of the results HuggingFace repo should obviate the need for HuggingFace schema inference.
• Removes the exclusions of default and None values during serialization. These exclusions were put in place with the goal of avoiding auto schema inference problems, which are no longer relevant (and which caused problems like the one described above).

[rodneykinney]

LGTM

[AmberRose2]

Looks good, thank you! I think this also makes it a lot clearer for users!

[jbragg]

It turned out that HF wasn't reading from dataset_infos.json so I switched to adding the schema info to the README, which seems to work