This is the source distribution of Ostfalia-Praktomat, a simple programming test runner for various programming languages.

The code is a fork (2019) from the KIT Praktomat (https://github.com/KITPraktomatTeam/Praktomat). It adds the ProFormA interface (https://github.com/ProFormA/proformaxml) which enables Praktomat to be used as a test runner backend in learning management systems. Most of the unused code is removed in order to ease maintenance.

The ProFormA format for tasks is 2.0 and 2.0.1 with some limitations. The ProFormA format for HTTP request and response is 2.0 and 2.1.

The code is running as a 'docker composition'. Each testcase runs in its own docker container.

Programming Languages

The following programming languages and test frameworks are provided with the ProFormA interface.

Language	Test Frameworks
Java 21	JUnit 4.12/5, Checkstyle 8.23/8.29/10.1/10.17
C++	Googletest, Make/CMake
C	Googletest, CUnit, Make/CMake
Python 3.11	Unittest with Pip (Doctest preparation)
SetlX	Simple test, Syntax Check

For running SetlX tests (https://randoom.org/Software/SetlX/) you need to copy the setlx-2.7.x.jar into the extra folder and rename it to setlX-2.7.jar.

For test specific information look here.

Submission types

The following types of submission are supported:

• one single file (embedded or attached to request)
• several files (embedded or attached to request)
• zip file containing several files
• URI of SVN repository to export the submission from (credentials are stored in .env file)

Installation

ProFormA Praktomat requires docker (https://docs.docker.com/get-docker/).

Each test run in its own docker container. This requires the user in the main container to be member of the docker group of the host computer. This is achieved by adding him to a group whose ID is identical to the ID of the Docker group on the host.

So you need to figure out the group id of the docker group on your computer:

    grep "docker" /etc/group

Set this group id value in the docker-compose.yml file as DOCKER_GROUP_ID.

The file system used by docker should preferably not be "vfs" (virtual filesystem), otherwise the performance will be greatly reduced.

The file system can be determined as follows:

    docker info | grep Storage

Configuration

Mandatory: Credentials

Create an .env file containing credentials and other private data. A sample file is included as .env.example.

cp .env.example .env 

Modify credentials!

Adjust number of workers to fit your hardware (e.g. WORKERS = ).

Optional: Change Locale

By default Praktomat uses German locale which results in German Checkstyle messages. In order to adjust the locale settings, you can override the build argument LOCALE in the docker-compose.yml file. E.g. for American English set LOCALE to en_US.UTF-8:

build:
    ...
    args:
        LOCALE: en_US.UTF-8

Note: This does not work at the moment with the supplied Java dockerfile.

Optional: HTTPS

For enabling HTTPS (port 443) you must

• docker-compose.yml: comment in - /data/certs:/etc/nginx/certs under nginx
• put your certificate files into /data/certs (or adjust folder name in docker-compose.yml)
• nginx/nginx.conf: comment in 443 configuration
• nginx/nginx.conf: set servername and adjust certificate file names

Optional: Different Test Framework Versions

For using other test framework versions you need to modify the following files:

• URLs in Dockerfile (folder docker-sandbox-image)
• src/checker/checker/JUnitChecker.py
• src/checker/checker/CheckStyleChecker.py
• src/settings/docker.py

Optional: SetlX

The SetlX library is not installed automatically. It can be downloaded from https://randoom.org/Software/SetlX/. The jar file must be copied into the extra subfolder and renamed to setlX-2.7.jar.

Create Docker Containers

In order to build and start the docker composition simply run

docker compose build
docker compose up

ProFormA API (CURL)

The grading service is available on port 80

    http://localhost/api/v2/submissions

A typical grading HTTP request in CURL is

curl -X POST --form submission.xml=@submission.xml -F "{solutionfilename}=@{solutionfile}" -F "{taskfilename}=@{taskfile}" http://localhost/api/v2/submissions

with the following 'submission.xml'

<?xml version="1.0" encoding="utf-8"?>
<submission 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xmlns="urn:proforma:v2.1" xmlns:praktomat="urn:proforma:praktomat:v2.3">

    <external-task uuid="{UUID}">
        <uri>http-file:{taskfilename}</uri>
    </external-task>
    <external-submission>
        <uri>http-file:{solutionfilename}</uri>
    </external-submission>
    <lms url="{your URI}">
        <submission-datetime>{timestamp}</submission-datetime>
        <user-id>{user id}</user-id>
        <course-id>{course id}</course-id>
    </lms>
    <result-spec format="xml" structure="separate-test-feedback" lang="de">
        <student-feedback-level>{level}</student-feedback-level>
        <teacher-feedback-level>{level}</teacher-feedback-level>
    </result-spec>

</submission>"

submission.xml can be transferred as a separate file or simply as data. Files are sent as multipart/form-data. The task file can be a zipped file or a simple xml file.

Note that embedding the student submission file(s) into submission.xml as embedded-txt-file or embedded-bin-file is also possible.

A sample for a timestamp is:

    2022-04-03T01:01:01+01:00

Submission with more than one file

For submitting more than one file you have the following options:

1. create a list of embedded text or/and binary files in the files section in submission.xml.
2. external-submission: set http-file as file name list (comma separated without blanks) and use standard file upload
3. external-submission: compress all student submission files to a zip archive (with or without package paths for Java) and set http-file to appropriate filename.

Sample for http-file for Java submission files list:

    http-file:de/ostfalia/sample/User.java,de/ostfalia/sample/File.java

You can also omit the relative path for Java source files:

    http-file:User.java,File.java

Server sent events

In order to get progress information during upload and running tests another interface is provided.

    http://{serverhost}/api/v2/runtest

Same as http://{serverhost}/api/v2/submissions but sends server sent events while running tests.

    http://{serverhost}/api/v2/upload

Same as http://{serverhost}/api/v2/runtest but does not run tests. Only the task is uploaded into cache and a sandbox template is created if required for the programming language.

LON-CAPA API

Since 4.5.0 an HTTP interface for the legacy learning management system LON-CAPA is provided.

The URI is

/api/v2/loncapasubmission

The following form fields are expected:

• submission_filename: Submission filename
• task: base64 coded ProFormA task
• task_filename: Filename of ProFormA task

The student submission is automatically put by LON-CAPA into LONCAPA_student_response.

Maintenance

The Praktomat stores tasks and results in a database and in the filesystem. In order not to run out of disk space the whole system should be reset from time to time (e.g. before starting a new semester). This can easily be done by calling

docker compose down
docker dompose up 

There is no need to back-up anything!

Retrieve state information:

    http://{serverhost}/praktomat-info

Retrieve tasks overview:

    http://{serverhost}/tasks

From version 5.3.0 the docker compose file creates volumes for data and database. So data is persistent. If you want to cleanup data execute

docker volume rm xxx_data_volume
docker volume rm xxx_database

xxx depends on the installation folder (check 'docker volume ls').

Software Update

In case of a software update this is the recommended process:

1. docker compose down
2. update software (e.g. git pull)
3. sudo ./remove_migrations.sh (delete old build files)
4. docker dompose build
5. docker dompose up -d

If there are sandbox Dockerfiles to be updated all sandbox images can be recreated with

docker exec -u root -it praktomat3 python3 src/manage-docker.py admin_recreate_images

It is recommended to restart Praktomat afterwards (docker compose up -d), because otherwise the status (http://{serverhost}/praktomat-info) will be incomplete (images will be missing).

Troubleshooting

If you encounter responses containing errors like 'cannot allocate memory' then you should try and increase one of the following values

TEST_MAXMEM_DOCKER_DEFAULT
TEST_MAXMEM_DOCKER_JAVA
TEST_MAXMEM_DOCKER_PYTHON

in src/settings/docker.py for your programming language. However, memory shortages can also lead to other error messages.