How to test the tested-universal-judge locally

[Brechtvsk]

I'm a teacher and I use dodona to teach my studends how to program. I create my own exercises and tests and would like to use the universal-judge (rather than the csharp judge). I've pulled the dodona/dodona-tested image on docker, I've also build my own image starting from the source code of the universal judge. When I open my exercise folder in a container based on these images i get an error when running python -m tested,module tested not found. Can anyone help on how i can get this working?
Do i need a docker compose file and mount both the judge source code as my own source code or ...

[jorg-vr]

Hi Brecht

The dockerfile only contains the dependencies, not the TESTed source itself.

You'll thus indeed have to include the TESTed source and your exercises within the docker environment. A docker compose file might be the simplest way to do this.

The TESTed repo also has a .devcontainer folder with a devcontainer.json definition using our docker image. I usually launch this devcontainer in vscode, which gives me an environment with both dependencies, dev-dependencies for running tests and the TESTed source code. But depending on your familiarity with devcontainers and/or docker-compose one method might be easier than the other.

[jorg-vr]

What commands do you currently use to run the docker container?

You can make folders from your own machine available in the docker using --volume, eg. docker run --volume <host-path>:<container-path>. This way you can make your exercises available in the container.
To mount multiple volumes, simply specify both: docker run --volume <host-path>:<container-path> --volume <host-path2>:<container-path2>

For TESTed to evaluate a submission with a certain test configuration, you need to give it some config data, see the README for all info that needs to be included in that config file. The paths specified here, should be the paths within the container.

Let me know If you have any more questions.

[Brechtvsk]

I tried using a devcontainer.json file toe spin up a container. But I haven't found out how I can set volume-bindings in a devcontainer. I've also searched for a way to start up an existing container and attach to it in a devcontainer.json, but havent found a way yet. Otherwise, runnen the tests locally works now

[Brechtvsk]

So they work locally now, but they don't work on dodona. I get "compilation error" without any information. It seems the run the testcases I've defined, but also no extra information is given why it fails

[jorg-vr]

For the devcontainer. I've TESTed your exercise repo with the following devcontainer.json:

{
    "name": "Exercise container",
    "image": "dodona/dodona-tested",
    "mounts": [
        "source=${localEnv:HOME}/Documents/universal-judge,target=/workspaces/universal-judge,type=bind,consistency=cached"
    ],
    "features": {
        // add git to the container
        "ghcr.io/devcontainers/features/git:1": {
            "version": "latest"
        }
    }
}

This works for me. The changes made to your file:

1. I've used our official published container dodona/dodona-tested, which should automatically be pulled (no need to build things yourself etc.)
2. I've updated the source path to match my local repo
3. I've added consistency=cached as recommended here

But, if you had a local docker container named universal-judge and your path was correct, I don't see why your existing configuration wouldn't already work.

I've also tested it on an exercise using the following script. (Provided as inspiration for future readers)

EXERCISE="/workspaces/6IW/Hoofdstuk 5 Iteratie/exercises/TestExercise/"
SUBMISSION="$EXERCISE""solution/solution.nl.cs"
TEST_SUITE="$EXERCISE""evaluation/tests.yaml"
PROGRAMMING_LANGUAGE="csharp"


JUDGE_LOCATION="/workspaces/universal-judge/"

cd $JUDGE_LOCATION
echo "$SUBMISSION"
python -m tested.cli -e "$EXERCISE" -s "$SUBMISSION" -t "$TEST_SUITE" -p "$PROGRAMMING_LANGUAGE"

More info about the tested.cli command can be found here

[jorg-vr]

I am willing to investigate why we provided no valid compilation bugs when the compilation failed on Dodona as this is a clear bug.
But currently your exercises seem to be working correctly.

Do you have a link to a still failing exercise, or could you tell me what you've changed in order to get it working?

[pdawyndt]

@jorg-vr discussion 114 explains how the issue was solved, and probably also why the issue occurs; I lately bumped into the same issue when starting a new exercise repository for TESTed; when using the default memory limit setting, the TESTed docker hits the memory limit for certain programming languages; so I think running the TESTed docker already comes close to the default memory limit and the compilation step of some programming language take up enough additional memory that the memory limit is exceeded and compilation does not succeed; what would improve the UX in this case:

• at least show a proper compilation error, which might already give a clue; I guess this requires TESTed to take responsibility of catching memory (and other) issues during compilation and properly reporting the compilation trace (no idea why TESTed could report the compilation error but not the compilation trace)
• reporting a memory limit exceeded instead of a compilation error, which would in this case more clearly indicate why the issue occurs
• split the TESTed docker into separate dockers for the individual programming languages, reducing the memory requirements for the individual dockers but requiring extra infrastructure to select an appropriate docker container based on the judge AND the programming language of the submitted solution if we want to support exercises that can be solved in different programming languages

[Brechtvsk]

@jorg-vr @pdawyndt Do i create a seperate bug report for the message so we can close this issue and also 114?
There is also some new information. When no memorylimit is set, no error is given execpt compilation error when manually setting memory_limit: 100000000 it gives an error out of memory

[jorg-vr]

These discussions can be closed. I've created an issue to follow up this bug internally. We'll notify you once it is fixed.

Just to be sure that I understood correctly: setting a sufficiently large memory limit allows you to create working exercises, which helps as a workaround for now?