Skip to content

Allow secret test cases to use .statement and .download #516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Allow secret test cases to use .statement and .download #516

Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
17c54e3
Allow secret test cases to use .statement and .download
hairez Sep 7, 2025
6add832
Clarify download rules for full‑feedback test cases
hairez Sep 7, 2025
44d35c8
Update 2025-09.md
hairez Sep 8, 2025
6f1647f
Update 2025-09.md
hairez Sep 8, 2025
667a359
Update spec/2025-09.md
hairez Sep 9, 2025
cfd06d4
Update spec/2025-09.md
hairez Sep 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 21 additions & 4 deletions spec/2025-09.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -645,12 +645,14 @@ For each test case:
- `output_validator_args` defines arguments passed to the output validator for the test case.
- `input_visualizer_args` defines arguments passed to the input visualizer for the test case.
- `output_visualizer_args` defines arguments passed to the output visualizer for the test case.
- When `full_feedback` is `true`, somebody whose submission didn't pass case should be shown:
- the given input,
- the produced output (stdout),
- When `full_feedback` is `true`, the following should be made available to a solver that didn't pass the test case:
- contents of `.in.statement` if it exists, otherwise the given `.in` file.
- the produced output (stdout),
- any error messages (stderr),
- the illustration created by the output visualizer (if applicable),
- the expected output.
- contents of `.ans.statement` if it exists, otherwise the `.out` if it exists, else the `.ans` file.

The test case with full feedback should also be available for download; the rules determining which files are provided are specified in [Test Cases with Full Feedback](#test-cases-with-full-feedback).
Comment on lines +654 to +655
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree. full_feedback test cases differ from sample in that are not given apriori, only if you fail on them.

Suggested change
The test case with full feedback should also be available for download; the rules determining which files are provided are specified in [Test Cases with Full Feedback](#test-cases-with-full-feedback).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I think I meant to describe that the test case with full feedback should be available for download, only when the feedback is given.

The test case would've already be shown in that case, and allowing it to be downloaded would just make it easier.

Do you agree with this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that it is a very good idea, I think we should be careful with requiring specific methods for delivering data. E.g. one could imagine a contest system that makes the files available on disk. That would technically not be "available for download".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The test case with full feedback should also be available for download; the rules determining which files are provided are specified in [Test Cases with Full Feedback](#test-cases-with-full-feedback).
For a submission that fails a test case with `full_feedback` set to `true`, the relevant feedback files for that test case should be available for download for that submission. These files are not made available otherwise; the rules determining which files are provided are specified in [Test Cases with Full Feedback](#test-cases-with-full-feedback).

- `hint` provides feedback for solving a test case to, for example, somebody whose submission didn't pass.
- `description` conveys the purpose of a test case.
It is an explanation of what aspect or edge case of the solution the input file is meant to test.
Expand Down Expand Up @@ -769,6 +771,21 @@ However, since only the `.out` files are validated it is advised to use these if

Validation can be customized by specifying `input_validator_args` and `output_validator_args` in `data/sample/test_group.yaml`.

### Test Cases with Full Feedback
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section seems to mostly talk about sample. Specifically the sections on downloads which doesn't really say that all full_feedback cases should be downloadable (and I don't think they should), but the text makes this unclear.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that was unclear, I have changed some things (and some mistakes where it mentioned sample).

Although, I think it should be the following: The test cases with full_feedback should be downloadable for a specific submission, if the submission failed on that test case.

I agree that the test cases secret should not be available for download otherwise. (Is that what you meant?)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.


If a test case has `full_feedback` set to `true` (see [Test Case Configuration](#test-case-configuration)), then by default, the `.in` and `.ans` pair for this particular testcase should be shown as feedback.
If a `.out` file exists the `.out` file is shown instead of the `.ans` file as feedback.
This behavior can be customized by creating files with extension `.in.statement` and `.ans.statement`.
If one of these files exists, its contents replaces that of the file with the same name -- except the `.statement` extension.

Note that it is an error to provide both a `.out` and a `.ans.statement` file.

By default, the `.in`, `.ans`, and `.files` files in `data/sample` are available for download.
Note that the content of `.in.statement` replaces that of `.in` and that the content of `.out` or `.ans.statement` replace that of `.ans` for the download.
This behavior can be further customized by providing files with the extension `.in.download` or `.ans.download`.
If one of these files exists, its contents replaces that of the file with the same name -- except the `.download` extension -- for the problem download.

If you want to make other files -- like testing tools -- available for download, you can use [attachments](#attachments).

## Generators

Expand Down