Skip to content

Improve file handling for stdin#608

Merged
niknetniko merged 6 commits intomasterfrom
enhancement/stdin
Feb 16, 2026
Merged

Improve file handling for stdin#608
niknetniko merged 6 commits intomasterfrom
enhancement/stdin

Conversation

@niknetniko
Copy link
Member

@niknetniko niknetniko commented Jan 24, 2026

This PR implements the stdin part of #600 , based on the ideas from #577.
For example:

tabs:
- tab: stdin
  contexts:
  - testcases:
      - stdin: 
          path: "hello.txt"
        stdout: "hello world!\n"

will now result in:

{"command": "start-judgement"}
{"title": "stdin", "command": "start-tab"}
{"command": "start-context"}
{"description": {"description": "$ submission < hello.txt", "format": "console"}, "command": "start-testcase"}
{"expected": "hello world!\n", "channel": "stdout", "command": "start-test"}
{"generated": "hello world!\n", "status": {"enum": "correct"}, "command": "close-test"}
{"command": "close-testcase"}
{"data": {"files": [{"path": "hello.txt"}]}, "command": "close-context"}
{"command": "close-tab"}
{"command": "close-judgement"}

Technical

  • Support defining a file for stdin
    • Support redirect for stdin
    • Supports "dynamically generated" files, ie. files whose content is in the test suite
  • Support update file definitions (using content and path)
  • Cleans up and fixes some things around cattrs decorators, and adds a bunch of tests for that

@niknetniko niknetniko self-assigned this Jan 24, 2026
@niknetniko niknetniko added enhancement New feature or request file support labels Jan 24, 2026
@niknetniko niknetniko marked this pull request as ready for review January 31, 2026 19:12
Copy link
Contributor

@TomNaessens TomNaessens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Had a look, mostly from a sanity check perspective and didn't spot anything weird or out of place.

@niknetniko niknetniko mentioned this pull request Feb 15, 2026
@niknetniko niknetniko merged commit 3329a84 into master Feb 16, 2026
11 of 12 checks passed
@niknetniko niknetniko deleted the enhancement/stdin branch February 16, 2026 18:19
@bmesuere bmesuere requested a review from Copilot February 17, 2026 12:39
@bmesuere
Copy link
Member

I know it's already merged, but to be sure we didn't miss any obvious things I requested a copilot review just now.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements file handling for stdin with support for file redirects and dynamically generated files, addressing issue #600. The implementation allows test suites to specify stdin using file paths (e.g., stdin: {path: "hello.txt"}), with support for:

  • Simple file references where content is read from a file
  • Dynamically generated files where content is specified inline or copied from another file to a different name
  • Display of file redirects in test descriptions (e.g., $ submission < hello.txt)

The PR also refactors the parsing decorators (@fallback_field and @ignore_field) to support decorator stacking via a new _chain_structure_hook mechanism, ensuring top-to-bottom execution order.

Changes:

  • Core data model updated: TextData.data renamed to TextData.content, added TextData.path field, introduced ContentPath class
  • Parsing decorators refactored to support chaining without requiring converter parameter
  • Execution logic enhanced to create dynamically generated files at runtime
  • DSL parser extended to handle !path YAML tag and file-based stdin definitions
  • Comprehensive test coverage added for legacy field support, decorator chaining, and stdin file handling

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tested/testsuite.py Renamed data to content, added ContentPath class, added path field and is_dynamically_generated() method to TextData
tested/parsing.py Refactored decorators to use _chain_structure_hook for composable hooks, removed converter parameter requirement
tested/judge/planning.py Added DynamicallyGeneratedFile class and get_dynamically_generated_files() method
tested/judge/execution.py Implemented file creation logic for dynamically generated files
tested/languages/generation.py Added stdin file redirect display support ($ submission < file.txt)
tested/languages/preparation.py Updated to use content instead of data
tested/dsl/translate_parser.py Added parsing logic for stdin file definitions, !path tag support, ContentPathString class
tested/dsl/schema.json Added fileData definition for stdin, updated to allow path or content fields
tested/dsl/schema-strict.json Extended fileData to support path type in content field
tested/configs.py Removed converter parameter from @fallback_field
tested/oracles/common.py Removed converter parameter from @fallback_field
tests/tested-draft7.json Added "path" to simpleTypes enum for custom YAML tag support
tests/test_testsuite_legacy.py New tests for legacy field support and decorator ordering
tests/test_parsing.py New comprehensive tests for parsing decorators
tests/test_dsl_legacy.py New tests for legacy DSL field support
tests/test_suite.py Updated assertions to use content instead of data
tests/test_oracles_builtin.py Updated to use content instead of data
tests/test_functionality.py Updated to use content instead of data
tests/test_dsl_yaml.py Updated to use content, added stdin file handling tests
tests/test_io_exercises.py Added tests for stdin file handling
tests/exercises/echo/evaluation/plan.yaml New test plan with stdin file examples
tests/exercises/echo/evaluation/plan-dynamic.yaml New test plan with dynamically generated file examples
tests/exercises/echo/evaluation/input.txt Input file for testing

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

# If we have both stdin and arguments, we use a here-document.
if case.input.arguments and stdin:
if stdin and isinstance(stdin, Path):
text = f"${args} < {stdin}"
Copy link

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The format string should be f"$ {args} < {stdin}" not f"${args} < {stdin}". The dollar sign in shell prompts is conventionally followed by a space before the command. This inconsistency with line 128 (args = f"$ {command}") makes the output look incorrect.

Suggested change
text = f"${args} < {stdin}"
text = f"$ {args} < {stdin}"

Copilot uses AI. Check for mistakes.
Comment on lines +190 to +203
if isinstance(dynamically_generated_file.content, ContentPath):
_logger.debug(
f"Copying input file %s to %s",
dynamically_generated_file.content.path,
destination,
)
source_file = (
bundle.config.resources / dynamically_generated_file.content.path
)
shutil.copy2(source_file, destination)
else:
_logger.debug(f"Creating dynamically generated file %s", destination)
destination.parent.mkdir(parents=True, exist_ok=True)
destination.write_text(dynamically_generated_file.content)
Copy link

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The destination.parent.mkdir(parents=True, exist_ok=True) call should also be made before copying the file in the ContentPath branch (line 199). If the destination path includes subdirectories (e.g., "subdir/input.txt"), the copy will fail because the parent directory doesn't exist. Move this mkdir call before the if-else block or add it to both branches.

Copilot uses AI. Check for mistakes.
Comment on lines +643 to +645
assert (
False
), f"Invalid stdin content is required but got {type(raw_content)}"
Copy link

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error message "Invalid stdin content is required but got..." is misleading when neither content nor path is provided. Consider improving it to something like "stdin requires either 'content' or 'path' field, but neither was provided" to better reflect the actual error condition.

Suggested change
assert (
False
), f"Invalid stdin content is required but got {type(raw_content)}"
if raw_content is None and raw_path is None:
assert False, (
"stdin requires either 'content' or 'path' field, "
"but neither was provided"
)
else:
assert (
False
), (
"Invalid stdin configuration: expected 'content' as a string "
f"or 'path' as a string, but got content type {type(raw_content)} "
f"and path type {type(raw_path)}"
)

Copilot uses AI. Check for mistakes.
chvp added a commit that referenced this pull request Feb 19, 2026
This reverts commit 3329a84, reversing
changes made to 72d62b7.
chvp added a commit that referenced this pull request Feb 19, 2026
Revert "Merge pull request #608 from dodona-edu/enhancement/stdin"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request file support

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants