Thoughts about support for multiple (human) languages

[thorehusfeldt]

The commit thorehusfeldt@84fdb98 is a first attempt to support other languages than English.

Currently it allows

bt pdf --language da

to turn problem_statement/problem.da.tex into problem.pdf, including reading the Danish problem name from problem.yaml according to the specification, see

• Problem metadata
• Problem statement

The next steps require some thought:

1. How should this be handled at contest level? For instance, should a nl contest gracefully fall back to en for problem D if the nl statement is not available? (I think yes.) There are lots of extra contest-related files and infrastructure that I honestly don’t yet understand.
2. Should the pdfs instead now be called problem.da.pdf etc? I like this. It could also allow bt pdf --all or something like that. (Or have bt pdf produce all languages it can find by default.) Opinions are welcome.
3. The file name convention for solution.tex is not ready for this. The obvious solution is to expect solution.nl.tex (with solution.tex as an alias for solution.en.tex)
4. Should problem_default_language be introduced and be configurable (instead hardcoding a preference for en)? I think it’s not worth doing.
5. Minor: How do we communicate command and thousands separator conventions? Should there be yet another yaml file, maybe in problem_statement? This is a minor issue because it only applies to PDF printouts and typically isn’t moved to the judging platform anyway. I think it’s not worth doing.
6. Export: I have no idea what’s going on with exporting to various other formats.
7. Constraints: the constraints logic should be extended to understand many languages. (This would be extremely valuable; having inter-statement consistency checks automated for multi-lingual development is in my experience very useful.)

[RagnarGrootKoerkamp]

Code looks pretty decent already! Sweet!

Some initial thoughts and replies.

• We should be backwards compatible with default/unspecified language, and just write problem.pdf in that case.
• Not specifying a language should probably build all languages. Indeed we can add by pdf --all and bt pdf --language xy that can override each other, with --all the default. (So that one can optionally be set in local config, and no further problem_default_language should be needed.)

1. Contests: Yes, all the contest infra is custom since the problem-package spec is only for single problems. There's problems.yaml which lists directories, labels, colours, time limits, and problem names again (This should also be a map then maybe; not sure which of that and <problemname>/problem.yaml takes precedence.) I would start without fallback for non-existing languages and only compile the problems that are available for the given language, maybe printing warnings/errors for missing langs? Mixing languages doesn't sound like something you should want?
2. I agree, output should be problem.pdf for problem.tex, and problem.xy.pdf for problem.xy.tex.
3. solution.tex is not part of the spec anyway, but yes we could treat it the same if you feel the need for this.
4. thousands separators... tricky. Are you thinking of passing a flag to latex/babel to get this right? Typically these separators are hand written in the problem.xy.tex anyway right?
5. Export: I also don't really know. I suppose Kattis supports it; domjudge not sure.
6. Constraints: I tihnk there is c++/latex specific regexes, but they don't depend on English I think, so that could just work?

[thorehusfeldt]

As far as I can tell, the current version of BAPCtools does not recognise problem.tex as a valid problem statement anyway; it expects problem.en.tex. So backwards compatibility is not much of an issue, and we can have the rule

problem.XY.tex produces problem.XY.pdf (rather than, as now, problem.en.tex produces problem.pdf)

The only issue I can see is that a file named problem.pdf may be expected by some tools. I suggest the following

1. if there is only a single language (typically, but not necessarily en) then problem.XY.pdf is also copied to problem.pdf. [In particular, the paths problem.pdf and problem.en.pdf both exist and have the same contents.] Could be a symlink rather than a copy.
2. if there is only a single language (typically, but not necessarily en) then problem.XY.pdf is moved to problem.pdf. [In an all-Dutch contest, problem.nl.tex produces problem.pdf, but not problem.nl.pdf]
3. if it exists, problem.en.pdf is moved to problem.pdf (no matter if there are other languages around). [No file problem.en.pdf ever exists.]
4. forget about it, and never try to produce problem.pdf (alternatively, add --output-name to bt pdf)

My order of preference is 4 > 2 > 3 > 1.

[RagnarGrootKoerkamp]

I agree that the simplicity of making problem.xy.pdf only is nice.

The only issue then remains the bt export for domjudge/kattis.
Domjudge expects a single problem.pdf, so I'd suggest to have bt export --language xy, which adds problem.pdf to the zip for each problem in the given language, and is only required if more than one language is found.

[thorehusfeldt]

Ah, I don’t know Domjudge well enough. Will do.

The Kattis export I did check by exporting and running verifyproblem on the result. Verifyproblem likes the problem.XY.tex files as they are (with the problemlabel macro replaced) so there was nothing to change.

[thorehusfeldt]

problem language attributes:

According to the specification, the problem.yaml metadata key languages exists and is about programming languages.

My commit thorehusfeldt@84fdb98 introduces the attribute (human) language to class Problem, but in hindsight that is not well-chosen:

1. From the discussion above, we won’t be privileging a single statement language anyway, so it makes no sense for a Problem to have a language.
2. The confusion between (programming) languages and (human) language is too great.

Instead, there should be Problem.statement_languages, even though it’s clunky. bt pdf should make all of them, unless otherwise instructed by --language, kept in config.args.language.

[thorehusfeldt]

Made it so in faa99a998185ebb24913b61c3526a906f16b16c2

[mzuenni]

I would suggest to make problem.pdf the concatenation of all language specific problem statements.
This has some benefits in my opinion:

• In the case where there is only one language nothing changes (exept that the .lang.pdf is also created) thus also ensures backwards compability
• It is the only (?) way to get all languages in other tools like DOMjudge
• It gives a nice overview over all languages

[thorehusfeldt]

Excellent solution. I have the ball.

[thorehusfeldt]

Actually @mzuenni and @RagnarGrootKoerkamp disagree on this (I have no opinion either way). To spell the two policies out (and give then names):

• merge: The file problem.pdf always is a concatenation of problem.*.pdf.
• move_when_unique: If len(problem.statement_languages) == 1 (for instance, because only en exists, or because --language en was specified) then and only then is problem.en.tex moved to problem.pdf.
• copy_when_unique: If len(problem.statement_languages) == 1 (for instance, because only en exists, or because --language en was specified) then and only then is problem.en.tex copied to problem.pdf
• always_keep: never produce problem.pdf; instead always keep the en.pdf form even when there’s only one language

These policies make sense for bt pdf (on both contest and problem levels), bt solutions and bt export. With minor changes they apply to generated problem* solution* and contest* PDFs.

Observations:

• Policy move_when_unique is backwards compatible; merge and copy_when_unique produces a superset of the old set of files.
• Policy keep_langs won’t work for bt export to DOMjudge (because it expects problem.pdf).
• Policy merge is the only way to get multiple language statements on the DOMjudge
• Policy merge with --language has the same effect as move_when_unique.

An alternative is to provide --cleanup-language-statements, which removes problem.en.pdf whenever problem.pdf was produced (by merge or copy policies).

Thore’s opinions (weakly held)

• I would like to not have different policies for, say, bt export and bt pdf.
• I don’t like copy_when_unique, it feels dirty to keep the same file with two different names.
• I prefer move-when-unique (for compatibility) or always-keep (it’s cleanest and retains all the information; renaming files for various exports is a banality that shouldn’t influence the architecture).

[thorehusfeldt]

Implemented move-when-unique out of conservatism.

[RagnarGrootKoerkamp]

Thanks! I think I like move-when-unique best also.

[thorehusfeldt]

bt solutions currently kind-of works, in particular if you have solution.xy.tex for every problem.xy.tex. If you only have solution.en.tex it is renamed to solution.pdf – the same logic as for statements.

The question is how problem_statement/solution.tex should be handled to ensure backwards compatibility. (This situation is different from statements because the BAPC file naming convention already expects problem.en.tex.)

1. ignore: Merely warn about problem_statement/solution.tex being present, but don’t use it. This won’t create solutions.pdf.
2. rename: Path.rename every problem_statement/solution.tex to problem_statement/solution.en.tex the first time bt sees it, log this, but leave the file renamed.
3. renameandback: Path.rename every problem_statement/solution.tex to problem_statement/solution.en.tex, warn about it, compile, and move it back. This seems very fragile and a lot of code.

[RagnarGrootKoerkamp]

I think it makes sense for solutions to be consistent with statements, so rename SGTM.

[RagnarGrootKoerkamp]

Output could be move-when-unique or solution.en.pdf. Since the solution is not part of any spec renaming that to include the language is fine with me, but probably consistency with the problem.pdf move-when-unique is preferable

[RagnarGrootKoerkamp]

(Note: Giving an error on solution.tex with info that they should be renamed, instead of implementing the rename in code, is also OK with me)