Consider renaming test case configuration file suffix

[thorehusfeldt]

In 2025-09 we put test case configuration files in a file called <testcase>.yaml., where <testcase is the name of the testcase, such as 001.

I suggest instead put this information into a file that has a different suffix that .yaml. For instance,

• 001_test_case.yaml
• 001.test_case.yaml
• 001.config.yaml
• 001.test_case_config.yaml
• 001.metadata.yaml

or whatever we can agree on.

The reason for this is to allow tooling to recognise such files using standard glob patterns. For instance,

• A LSP-aware editor (VS Code, NeoVim, etc.) will negotiate the correct JSON schema via schemastore, which matches on glob patterns. There is no way to tell the editor to please use this grammar for out test case configuration files because we cannot positively identify them (in particular, distinguish them from test_group.yaml)
• BAPCtools may want to validate test case configuration files against a schema to ensure test data integrity; the dispatcher works on file names, mappting (say) problems.yaml files to the schema #Problem, test_group.yaml files to the schema #test_group_configuration. But it cannot just map *.yaml (or even data/**/*.yaml) to the schema #test_case_configuration.

There could be more, the two cases above just happened to me today alone.

[niemela]

<testcase>.config.yaml sounds very reasonable to me.

Note that choosing that name is basically the "opposite" of doing #568 , so if we so this we should probably just close that one.

[thorehusfeldt]

A reason to not use *config.yaml is to avoid generic filematch patterns, see for instance the considerations at schemastore.org: https://github.com/SchemaStore/schemastore/blob/master/CONTRIBUTING.md#avoid-generic-filematch-patterns .

Aside, we’re already pushing it with problem.yaml, but apparently that one is still “free” (in the sense that no other .yaml-tradition in the history of civilisation has “claimed” it.) I’ll make a request for mapping *problem.yaml to “our” schema early in the new year.

We will not be able to hog *config.yaml, but of course that’s less of a problem because we can match data/**/*config.yaml, but even that sounds like a very generic glob pattern that may already have been taken. For that reason alone, *tcconfig.yaml or (even better) *test_case.yaml would be safer.

[niemela]

@thorehusfeldt I honestly feel that I'm not enough into the world of YAML, and more specifically YAML schemas to even understand your comment. Is there any explanation, elaboration on why it is actually good to avoid "generic" (whatever that is supposed to mean exactly) names? This feels to me to be specific to the workflows and standard practices to the YAML ecosystem (which of course makes it relevant to us when we use YAML files), but I don't at all understand why?

And, semantically, I hate 01-small.test_case.yaml. To me 01-small.in reads as "the input file of the test case called 01-small". So, 01-small.test_case.yaml would read to me as "the test case file of the test case called 01-small". (I'm definitely nitpicking now...)

[RagnarGrootKoerkamp]

This feels to me to be specific to the workflows and standard practices to the YAML ecosystem

Indeed. There is a database that enabled automatic yaml verification/auto-completion based on specific filenames. But data/**/*.yaml is a super generic pattern so we can't automatically enable it for such files.

I do agree that for us, <testcase>.yaml is nice while <testcase>.config.yaml is a bit much.

@thorehusfeldt Isn't there also some option for magic comments like #! https://path/to/schema.yaml or #! problem-package-format/testcase? Then we could just make sure to include that on top of all sample/skel files.

[niemela]

So, is this automatic YAML verification magic thingy completely context free? That seems (to me) to be mostly useless most of the time? Because in the contest of "this is a problem package" both problem.yaml, test_group.yaml, and data/**/*.yaml are plenty specific. And having globally unique file name (patterns) over all files in all systems and all contexts just sounds... terrible ... to me?

But, I have to admit that I'm talking from a position of ignorance here. I don't use a lot of YAML tools...

[RagnarGrootKoerkamp]

So, is this automatic YAML verification magic thingy completely context free?

indeed. For stuff like problem.yaml and generators/generators.yaml and submissions/submissions.yaml that's apparantly good enough.

So <testcase>.yaml is then, I believe, the first one where the name is not a sufficiently-good global identifier.

[thorehusfeldt]

Here is the globally unique(!) JSON schema catalog. Believe it or not, but this works and is an important part of many editors.

https://github.com/SchemaStore/schemastore/blob/master/src/api/json/catalog.json

You can scroll until you find problempackage generators. I'll add problempackage problem in a few days.

[niemela]

Here is the globally unique(!) JSON schema catalog. Believe it or not, but this works and is an important part of many editors.

Ok, I believe you. But if you didn't tell me that it works well, I would never think it would.

Also, it feels crazy that we can get "generators.yaml". If we have "problem.yaml" that's less surprising.

You can scroll until you find problempackage generators. I'll add problempackage problem in a few days.

👍

[thorehusfeldt]

Ok, I believe you. But if you didn't tell me that it works well, I would never think it would.

Something similar is happening in a modern IDE. The editor has an internal registry of file type what it matches by globbing against a registry. That’s how it would decides that “oh, this currently opened problem.yaml looks as if it wants to be treated as problem metadata file for the famous problem package. I better fire up type checking, validation, autocompletion, and tooltips for that.”

The central registry at schemastore.org is just where most modern IDEs look.

We can maybe glob for data/{secret, sample, invalid_inputs, ...}/**/*.yaml, but this would still clash with test_group.yaml, because (at least to my knowledge) one cannot write exclusive globs.

Hence my suggestion to change the suffix of <test_case_configuration>.yaml. Even 001.tc.yaml would solve this. (But 001.test_case.yaml would be better.)