Make `problem.yaml` available to generators, validators, and various statements

[thorehusfeldt]

Aside about \constant{name} in LaTeX:

What we seem to want to support is a mechanism for the Latex source to refer to values defined in problem.yaml. I clean way to do that (which would require minimal effort do define) is \problem{foo} where foo is a key in problem.yaml.

@thorehusfeldt This is a new (and novel) idea. I'm not sure at all if I like it or not, but I think it should be a separate issue. Was this a real suggestion, or more of a random thought? If the former, we should create a new issue for it.

I will close this issue now though.

Originally posted by @niemela in #344

Continuing this line of thought.

Several mechanisms in the problem package want access to values in problem.yaml. Here are just three, different, examples:

• input_validator/validate/ wants to know constants.max_n.value
• the LaTeX problem statement to know title.en

We currently have different ways of facilitating this, such as LaTeX macros and template substitution. But we can’t do this:

from bapctoolslib import problem # imaginary module that just parses ../../problem.yaml and puts it into a useful namespace

max_n = problem.constants.max_n # OMG! No substitution! The linter is happy! I can run my code from CLI

Or, without a library,

from ruamel.yaml import YAML

yaml = YAML(typ="safe")  # safe loader -> plain Python types
with open("../problem.yaml", "r") as f:
    prroblem = yaml.load(f)

max_n = problem.constants.max_n

Whatever mechanism a language or framework uses to open a YAML file can be used. No substitution is performed. If BAPCtools wants to write neat libraries or .h header files that hide the YAML parsing, they can do that. Or not. A LaTeX class file would do the same. Or an upcoming typst library for making really sexy solution slides that read problem.source. (Alas, YAML can’t include YAML, otherwise that would be perfect.)

The only change to the specification is for sandboxing:

• the problem.yaml must be is available to the code in the same relative position as in the problem package.

[niemela]

• "this code" is input and output validators, right?
• "same relative position" would be "directly in the working directory"? Or do you mean, "where problem.yaml is in the package if you assume that you are running the validators in their respective directories in the problem package"?
• generators are not specified in the spec, so we probably shouldn't require one random thing about them?

[thorehusfeldt]

"this code" is input and output validators, right?

No, I mean more. I can see, for instance, a typst library for solution slides wanting to access problem.yaml. Certainly a LaTeX class that expects the code in problem.en.tex and the constants in ../problem.yaml. Or an output visualiser. Or /submissions/accepted/thore.py. During problem development construction I can write such libraries or programmes that I can run locally (the python example above is one concrete case, but I can come up with many more), but which won’t work when my code is (correctly) sandboxed away.

Note that this in particular allows a clean and general way to allow programmes in /submission to use (or ignore) problem.constants in a way that keeps the source code valid CPP or Python or whatever (because we avoid substitutions.)

So “the code” is includes input and output validators, submissions, and source files for statements and slides. BAPCtools would naturally add answer validators and generators.

same relative position" would be "directly in the working directory"?

For statements, its ../problem.yaml. For input validators its ../problem.yaml or ../../problem.yaml, depending on how I organise that directory. For submissions/accepted/thore.py its ../../problem.yaml. My above Python snipped is an example. I want

with open("../problem.yaml", "r") as f:
    prroblem = yaml.load(f)

to work even when my programme is sandboxed.

generators are not specified in the spec,

You’re right.

[niemela]

During problem development construction I can write such libraries or programmes that I can run locally (the python example above is one concrete case, but I can come up with many more),

Exactly, so we don't need to (and IMO shouldn't) specify how problem.yaml is accessed during development, because it's just there where it's defined to be.

but which won’t work when my code is (correctly) sandboxed away.

Right. Why would LaTeX and typst run in the sandbox? Why would LaTeX and typst run other than during development and installation?

So “the code” is includes input and output validators, submissions, and source files for statements and slides. BAPCtools would naturally add answer validators and generators.

Why would submissions want access to problem.yaml? That seems like a very bad idea to me? Also, this is not in line with the current consensus in #344 , which is that constants will not be replaced in submissions. (If you think that's the wrong direction, please comment over there as well).

For input validators its ../problem.yaml or ../../problem.yaml

This is problematic, don't you think? Same is true for output validators.

specifically, this means that the relative position of problem.yaml and other files that will be available in the working directory (e.g. .files) will differ for different validators.

, depending on how I organise that directory

Rather, depending on whether the validator consists of one or multiple files.

[thorehusfeldt]

Right. Why would LaTeX and typst run in the sandbox? Why would LaTeX and typst run other than during development and installation?

I think BAPCtools certainly copies the statement directory and nothing else somewhere else and does all the compilation there. You can’t \includegraphics{../data/sample/001.png}. (At least that’s what I think happens. Can’t remember that verifyproblem does.) The point is to guarantee the existence of ../problem.yaml at the expected place.

Why would submissions want access to problem.yaml?

Because that’s where problem.constants are. This entire suggestion started under “rethink constants”, so that’s what I’m doing here. It is very much not in line with the current consensus; indeed I am actively rethinking that.

My suggestion allows authors to write code during development that

1. they can run without substitution, directly from the command line or their IDE. (By comparison, substitutions enforce a workflow, where a tool first must change the source code.)
2. is valid python / cpp / whatever. (In particular, their IDE doesn’t complain about type or syntax errors, the linter is happy, the autoformatter works, etc.)

Library authors (such as validation.h used by many validators) can offer a namespace with the correctly inferred values from problem.yaml. I could just write int max_n = constants.max_n; in the C++ code of my validator, instead of int max_n = {% max_n %}; (or whatever the template syntax is.)

For input validators its ../problem.yaml or ../../problem.yaml
This is problematic, don't you think? Same is true for output validators.

No. We just need to promise that source code is processed in sandboxes that retain problem.yaml in the same relative position to the source code as in the problem package.

Rather, depending on whether the validator consists of one or multiple files.

Well, /submissions/accepted/thore.py uses ../../problem.yaml, /input_validator/validate.py uses ../problem.yaml, etc. It’s the depth of the source code in the problem package.

[thorehusfeldt]

After phone conversation with @niemela : Maybe the reference issue (about “how far up” the problem.yaml actually is located) can be solved by symlinking problem.yaml from in the problem package. Then a (say) input validator could look like this:

problem.yaml
input_validators/
  +-my_validator/
      +-validation.h
      +-validator.cpp
      +-problem.yaml -> ../../problem.yaml

A (much-used) validation.h header file reads the values of problem.constant from problem.yaml (which is just a symbolic link two levels up) and makes those values available to the bespoke validator.cpp programme. No templating/substitution is ever performed. All we need to guarantee is that the symbolic link work (for instance by copying problem.yaml into the working directory.)

[RagnarGrootKoerkamp]

I do think this is nice conceptually, but I'm not quite convinced yet the benefits outweight the effort of making it work after writing this I now like it more :)

1. they can run without substitution, directly from the command line or their IDE.

ohhhh, only just now I realize this point. You want the latex/validators to pull the constants from the problems.yaml instead of pushing them (via substitution), so that they work in isolation.

Hmmm; for latex we already have a bespoke compilation system, and validators are pretty much never run in isolation, so I'm not sure how much value that provides in practice?

I do agree that especially with typst this would be quite convenient probably.

symlinking problem.yaml

Hmm this feels quite verbose. I'd rather just say that validators can read problem.yaml from ../../problem.yaml?

2. is valid python / cpp / whatever

Yeah in the end I think this is a main benefit.

One more benefit, that actually quite excites me:

3. the header could automatically match variable names to variables. So we could auto n = read_int("n");, and the header could automatically parse n_min and n_max from the constants.

[niemela]

Hmm this feels quite verbose. I'd rather just say that validators can read problem.yaml from ../../problem.yaml?

Yes, but depending on the whether the validator is a single file of multiple this will be ../problem.yaml or ../../problem.yaml. This is very annoying and ugly.

3. the header could automatically match variable names to variables. So we could auto n = read_int("n");, and the header could automatically parse n_min and n_max from the constants.

I don't understand this point. Could you explain/elaborate?

[RagnarGrootKoerkamp]

Hmm this feels quite verbose. I'd rather just say that validators can read problem.yaml from ../../problem.yaml?

Yes, but depending on the whether the validator is a single file of multiple this will be ../problem.yaml or ../../problem.yaml. This is very annoying and ugly.

Oh ugh... I think the 'can run validator in isolation' benefit is quite small here, so I'd also be very happy to just say that validators can read problem.yaml from the current working directory.

Ah but: input/answer validators are only run by problemetters ('us') so we can be flexible, but output validators are run by domjudge as well. In that setting, making problem.yaml available in CWD or so is OK, but a symlink wouldn't work (and so we'd replace it by a copy of the file on export).

One more idea (not sure if good; probably overkill): add an environment variable with the path to the file, and default to cwd?

3. the header could automatically match variable names to variables. So we could auto n = read_int("n");, and the header could automatically parse n_min and n_max from the constants.

I don't understand this point. Could you explain/elaborate?

If the validator/header has access to the entire problem.yaml / the full set of constants (as opposed to only a subset of values via {{n_max}} substitution), it can automatically extract the values for n_min and n_max from that when the user calls read_int("n"), and error out if they're not set.

That way, we can 'force' the user into setting these variables, and then also encourage reusing them in latex.

[thorehusfeldt]

ohhhh, only just now I realize this point. You want the latex/validators to pull the constants from the problems.yaml instead of pushing them (via substitution), so that they work in isolation.

Push/pull is not how I think of it, but yes: I suggest to give various pieces of source code (validators, statements, generators, but maybe not author submissions in /submissions) access to problem metadata. This could work ad-hoc and bespoke, but mainly I believe that various “helper libraries” such as validation.h or bapc.cls or a hypothetical problempackagelib.py would extract that information and make it available to author-written source code in a clean fashion if the author so desires (“pulling”).

It would then not be the responsibility of the specification to determine which current programming languages get access to useful information in which fashion (say, by specifying – as we do now – that latex expects problem.name.en in \problemtitle{} but constants.max_n.tex or constants.max_n in a different way). (“pushing”)

For me the main benefit is to ensure a more pleasant authoring experience (because IDEs “just work” and I can compile and run my code like I want), remove some details from the specification, make it more future-proof (again, typst is the current use case), and increase expressiveness (a LaTeX statement for the Dutch version could now actually refer to the English word for Stroopwafels in a parenthetical remark):

Alice really likes \problem{constants.delicacy.nl} (in English, \problem{constants.delicacy.en}), and…

[niemela]

@thorehusfeldt I like what you're trying to to, but...

a LaTeX statement for the Dutch version could now actually refer to the English word for Stroopwafels in a parenthetical remark:

Alice really likes \problem{constants.delicacy.nl} (in English, \problem{constants.delicacy.en}), and…

That really seems like bad use of this?

[mzuenni]

I don't really want to argue against or in favor, but this was mentioned:

again, typst is the current use case

I would say this is a pretty week use case right now. Not because of typst but because of how we handled statement files in the past.

For LaTeX we do not provide a "complete" LaTeX file but rather a snippet. The build system is then responsible for providing an environment in which the statement can be built... and therefore the build system is also responsible for making constants, titles, samples, etc. available (in LaTeX words: the problemset.cls is not part of the spec).

The same can certainly be done for typst, i.e., the typst statement should be just the text. The buildsystem would again provide everything else.

One independent point: Properly parsing yaml is quite annoying. In Python you only get YAML 1.1 with pyyaml. In C++ you would need your own header. In pdfTeX if there is no package, I see no hope. In markdown this is simply impossible.