Always use external ID in API

Fixes #2087

Author: nickygerritsen

ChangeLog

@@ -2,6 +2,8 @@ DOMjudge Programming Contest Judging System
 
 Version 8.4.0DEV
 ---------------------------
+ - Get rid of 'internal' data source mode, always requiring - but auto
+   generating - external ID's for all entities to simplify event logic.
 
 Version 8.3.0 - 31 May 2024
 ---------------------------

doc/manual/config-advanced.rst

@@ -16,30 +16,25 @@ path of your installation) as follows:
 - *Affiliation logos*: these will be shown with the teams that are
   part of the affiliation, if the ``show_affiliation_logos`` configuration
   option is enabled. They can be placed in
-  `public/images/affiliations/1234.png` where *1234* is the numeric ID
+  `public/images/affiliations/1234.png` where *1234* is the :ref:`external ID <external-ids>`
   of the affiliation as shown in the DOMjudge interface. There is a
   separate option ``show_affiliations`` that independently controls where
   the affiliation *names* are shown on the scoreboard. These logos should be
   square and be at least 64x64 pixels, but not much bigger.
 - *Team pictures*: a photo of the team will be shown in the team details
   page if `public/images/teams/456.jpg` exists, where *456* is the
-  team's numeric ID as shown in the DOMjudge interface. DOMjudge will not
+  team's :ref:`external ID <external-ids>` as shown in the DOMjudge interface. DOMjudge will not
   modify the photos in any way or form, so make sure you don't upload photos
   that are too big, since that will incur a lot of network traffic.
 - *Contest Banners*: a page-wide banner can be shown on the public scoreboard
   if that image is placed in `public/images/banners/1.png` where *1* is the
-  contest's numeric ID as shown in the DOMjudge interface. Alternatively, you
+  contest's :ref:`external ID <external-ids>` as shown in the DOMjudge interface. Alternatively, you
   can place a file at `public/images/banner.png` which will be used as a banner
   for all contests. Contest-specific banners always have priority. Contest
   banners usually are rectangular, having a width of around 1920 pixels and a
   height of around 300 pixels. Other ratio's and sizes are supported, but check
   the public scoreboard to see how it looks.
 
-.. note::
-
-  The IDs for affiliations, teams and contests need to be the *external ID*
-  if the ``data_source`` setting of DOMjudge is set to external.
-
 It is also possible to load custom CSS and/or JavaScript files. To do so, place
 files ending in `.css` under `public/css/custom/` and/or files ending in `.js`
 under `public/js/custom/`. See the Config checker in the admin interface for the

doc/manual/import.rst

@@ -13,6 +13,22 @@ installation.
 To use the CLI, you need to replace ``<WEBAPP_DIR>`` with the path to
 the ``webapp`` directory of the DOMserver.
 
+.. _external-ids:
+
+External ID's
+-------------
+
+Most entities in DOMjudge have an `eternal ID`. External ID's are used to link
+entities in DOMjudge to entities in an external system, e.g. the ICPC CMS. The
+API uses the external ID's to expose entities to other systems. When you create
+an entity in DOMjudge, specifying the external ID is optional; DOMjudge will
+use an automatically generated ID if you don't provide one. However, if this ID
+is not unique, you will get a message telling you that you need to provide your
+own external ID.
+
+When importing entities in bulk as described below, the external ID will be
+populated with the ID as specified in the files you import.
+
 Importing team categories
 -------------------------
 
@@ -33,9 +49,7 @@ fields:
 - ``sortorder`` (defaults to ``0``): the sort order of the team category to use
   on the scoreboard. Categories with the same sortorder will be grouped together.
 
-If the ``data_source`` setting of DOMjudge is set to external, the ``id`` field will be the
-ID used for the group. Otherwise, it will be exposed as ``externalid`` and a group ID will be
-generated by DOMjudge.
+The ``id`` field will be the ID used for the group.
 
 Example ``groups.json``::
 
@@ -73,9 +87,7 @@ Each of the following lines must contain the following elements separated by tab
 - the category ID. Must be unique
 - the name of the team category as shown on the scoreboard
 
-If the ``data_source`` setting of DOMjudge is set to external, the category ID field will be
-the ID used for the group. Otherwise, it will be exposed as ``externalid`` and a group ID will
-be generated by DOMjudge.
+The ``id`` field will be the ID used for the group.
 
 Example ``groups.tsv``::
 
@@ -114,9 +126,7 @@ fields:
 - ``formal_name``: the affiliation name as used on the scoreboard
 - ``country``: the country code in form of ISO 3166-1 alpha-3
 
-If the ``data_source`` setting of DOMjudge is set to external, the ``id`` field will be the
-ID used for the affiliation. Otherwise, it will be exposed as ``externalid`` and an affiliation
-ID will be generated by DOMjudge.
+The ``id`` field will be the ID used for the affiliation.
 
 Example ``organizations.json``::
 
@@ -168,11 +178,8 @@ fields:
 - ``organization_id``: the ID of the team affiliation this team belongs to
 - ``location.description`` (optional): the location of the team
 
-If the ``data_source`` setting of DOMjudge is set to external, the ``id`` field will be the
-ID used for the team and the ``group_ids`` and ``organization_id`` fields are the values as
-provided during the import of the other files listed above. Otherwise, the ``id`` will be
-exposed as ``externalid``, a team ID will be generated by DOMjudge and you need to use the
-ID's as generated by DOMjudge for ``group_ids`` as well as ``organization_id``.
+The ``id`` field will be the ID used for the team and the ``group_ids`` and ``organization_id``
+fields are the values as provided during the import of the other files listed above.
 
 Example ``teams.json``::
 
@@ -221,11 +228,8 @@ Each of the following lines must contain the following elements separated by tab
 - a country code in form of ISO 3166-1 alpha-3
 - an external institution ID, e.g. from the ICPC CMS, may be empty
 
-If the ``data_source`` setting of DOMjudge is set to external, the team ID field will be the
-ID used for the team and the category ID field is the value as provided during the import of
-the other files listed above. Otherwise, the team ID will be exposed as ``externalid``, a
-team ID will be generated by DOMjudge and you need to use the ID as generated by DOMjudge
-for the category ID.
+The team `id` field will be the ID used for the team and the category ID field is the value
+as provided during the import of the other files listed above.
 
 Example ``teams2.tsv``::
 
@@ -265,10 +269,8 @@ fields:
 - ``name``: (optional) the full name of the account
 - ``ip`` (optional): IP address to link to this account
 
-If the ``data_source`` setting of DOMjudge is set to external, the ``id`` field will be the ID
-used for the user and the ``team_id`` field is the value as provided during the team import.
-Otherwise, the ``id`` will be exposed as ``externalid``, a user ID will be generated by DOMjudge
-and you need to use the ID as generated by DOMjudge for ``team_id``.
+The ``id`` field will be the ID used for the user and the ``team_id`` field is the value as provided during
+the team import.
 
 Example ``accounts.yaml``::
 

doc/manual/shadow.rst

@@ -19,11 +19,9 @@ Configuring DOMjudge
 
 In the DOMjudge admin interface, go to *Configuration settings* page and modify
 the settings to mimic the system to shadow from. Also make sure to set
-*data_source* to ``configuration and live data external``. This tells DOMjudge
+*shadow_mode* to ``true``. This tells DOMjudge
 that it will be a shadow for an external system. This will:
 
-* Expose external ID's in the API for both configuration and live data, i.e.
-  problems, teams, etc. as well as submissions, judgings and runs.
 * Add a *Shadow Differences* and *External Contest Sources* item to the jury
   menu and homepage for admins.
 * Expose additional information in the submission overview and detail pages.

etc/db-config.yaml

@@ -355,17 +355,11 @@
             enum_class: App\Utils\EventFeedFormat
             public: false
             description: Format of the event feed to use. See [current draft](https://ccs-specs.icpc.io/draft/contest_api#event-feed) and [versions available](https://ccs-specs.icpc.io/).
-        -   name: data_source
-            type: int
-            default_value: 0
+        -   name: shadow_mode
+            type: bool
+            default_value: false
             public: false
-            description: "Source of data: used to indicate whether internal or external IDs are exposed in the API. `configuration data external` is typically used when loading configuration data from the ICPC CMS, and `configuration and live data external` when running DOMjudge as \"shadow system\"."
-            options:
-                0: all local
-                1: configuration data external
-                2: configuration and live data external
-            regex: /^\d+$/
-            error_message: A value between 0 and 2 is required.
+            description: Is this system running as a shadow system?
             docdescription: See :doc:`the chapter on running DOMjudge as a shadow system<shadow>` for more information.
         -   name: external_contest_sources_allow_untrusted_certificates
             type: bool

gitlab/ci/integration.yml

@@ -5,7 +5,7 @@
     - set -eux
     - if [ -z ${PHPVERSION+x} ]; then export PHPVERSION=8.3; fi
     - if [ "$TEST" = "E2E" ]; then exit 0; fi
-    - if [ "$CRAWL_DATASOURCES" != "0" ]; then exit 0; fi
+    - if [ "$CRAWL_SHADOW_MODE" != "0" ]; then exit 0; fi
     - timeout --signal=15 40m ./gitlab/integration.sh $PHPVERSION
   artifacts:
     when: always
@@ -29,7 +29,7 @@ integration_mysql:
     MYSQL_REQUIRE_PRIMARY_KEY: 1
     PIN_JUDGEDAEMON: 1
     TEST: "Unit"
-    CRAWL_DATASOURCES: "0"
+    CRAWL_SHADOW_MODE: "0"
 
 integration_mariadb_pr:
   except:
@@ -61,4 +61,4 @@ integration_unpinned_judgehost:
     MARIADB_PORT_3306_TCP_ADDR: sqlserver
     PIN_JUDGEDAEMON: 0
     TEST: "Unit"
-    CRAWL_DATASOURCES: "0"
+    CRAWL_SHADOW_MODE: "0"

gitlab/ci/template.yml

@@ -53,7 +53,7 @@
     matrix:
       - PHPVERSION: ["8.1","8.2","8.3"]
         TEST: ["E2E","Unit"]
-        CRAWL_DATASOURCES: ["0","1","2"]
+        CRAWL_SHADOW_MODE: ["0","1"]
 
 .phpsupported_job_pr:
   script:
@@ -62,4 +62,4 @@
     matrix:
       - PHPVERSION: ["8.3"]
         TEST: ["E2E","Unit"]
-        CRAWL_DATASOURCES: ["0"]
+        CRAWL_SHADOW_MODE: ["0"]

gitlab/ci/unit.yml

@@ -12,9 +12,9 @@
     - set -eux
     - if [ -z ${PHPVERSION+x} ]; then export PHPVERSION=8.1; fi
     - if [ -z ${TEST+x} ]; then export TEST="UNIT"; fi
-    - if [ "$TEST" = "UNIT" ] && [ "$CRAWL_DATASOURCES" != "0" ]; then exit 0; fi
-    - if [ "$TEST" = "E2E" ] && [ "$CRAWL_DATASOURCES" != "0" ] && [ "$CI_COMMIT_BRANCH" != "main" ]; then exit 0; fi
-    - export CRAWL_DATASOURCES
+    - if [ "$TEST" = "UNIT" ] && [ "$CRAWL_SHADOW_MODE" != "0" ]; then exit 0; fi
+    - if [ "$TEST" = "E2E" ] && [ "$CRAWL_SHADOW_MODE" != "0" ] && [ "$CI_COMMIT_BRANCH" != "main" ]; then exit 0; fi
+    - export CRAWL_SHADOW_MODE
     - ./gitlab/unit-tests.sh $PHPVERSION $TEST
   artifacts:
     when: always
@@ -48,4 +48,4 @@ run unit tests (MySQL):
   parallel:
     matrix:
       - TEST: ["E2E","Unit"]
-        CRAWL_DATASOURCES: ["0"]
+        CRAWL_SHADOW_MODE: ["0"]

gitlab/integration.sh

@@ -155,7 +155,7 @@ fi
 if [ $cgroupv1 -ne 0 ]; then
     section_start start_judging "Start judging"
     cd /opt/domjudge/judgehost/
-    
+
     sudo -u domjudge bin/judgedaemon $PINNING |& tee /tmp/judgedaemon.log &
     sleep 5
     section_end start_judging
@@ -178,7 +178,7 @@ for i in hello_kattis different guess; do
         cd "$i"
         zip -r "../${i}.zip" -- *
     )
-    curl --fail -X POST -n -N -F zip=@${i}.zip http://localhost/domjudge/api/contests/1/problems
+    curl --fail -X POST -n -N -F zip=@${i}.zip http://localhost/domjudge/api/contests/demo/problems
 done
 section_end submitting
 
@@ -195,21 +195,21 @@ curl $CURLOPTS -c $COOKIEJAR -F "_csrf_token=$CSRFTOKEN" -F "_username=admin" -F
 # Send a general clarification to later test if we see the event.
 curl $CURLOPTS -F "sendto=" -F "problem=1-" -F "bodytext=Testing" -F "submit=Send" \
      "http://localhost/domjudge/jury/clarifications/send" -o /dev/null
-    
+
 section_end curlcookie
 
 if [ $cgroupv1 -ne 0 ]; then
     section_start judging "Waiting until all submissions are judged"
     # wait for and check results
-    NUMSUBS=$(curl --fail http://admin:$ADMINPASS@localhost/domjudge/api/contests/1/submissions | python3 -mjson.tool | grep -c '"id":')
-    
+    NUMSUBS=$(curl --fail http://admin:$ADMINPASS@localhost/domjudge/api/contests/demo/submissions | python3 -mjson.tool | grep -c '"id":')
+
     # Don't spam the log.
     set +x
-    
+
     while /bin/true; do
         sleep 30s
         curl $CURLOPTS "http://localhost/domjudge/jury/judging-verifier?verify_multiple=1" -o /dev/null
-    
+
         # Check if we are done, i.e. everything is judged or something got disabled by internal error...
         if tail /tmp/judgedaemon.log | grep -q "No submissions in queue"; then
             break
@@ -219,12 +219,12 @@ if [ $cgroupv1 -ne 0 ]; then
             break
         fi
     done
-    
+
     NUMNOTVERIFIED=$(curl $CURLOPTS "http://localhost/domjudge/jury/judging-verifier" | grep "submissions checked"     | sed -r 's/^.* ([0-9]+) submissions checked.*$/\1/')
     NUMVERIFIED=$(   curl $CURLOPTS "http://localhost/domjudge/jury/judging-verifier" | grep "submissions not checked" | sed -r 's/^.* ([0-9]+) submissions not checked.*$/\1/')
     NUMNOMAGIC=$(    curl $CURLOPTS "http://localhost/domjudge/jury/judging-verifier" | grep "without magic string"    | sed -r 's/^.* ([0-9]+) without magic string.*$/\1/')
     section_end judging
-    
+
     # We expect
     # - two submissions with ambiguous outcome,
     # - one submissions submitted through the submit client, and thus the magic string ignored,
@@ -238,7 +238,7 @@ if [ $cgroupv1 -ne 0 ]; then
         echo "Of these $NUMNOMAGIC do not have the EXPECTED_RESULTS string (should be 1)."
         curl $CURLOPTS "http://localhost/domjudge/jury/judging-verifier?verify_multiple=1" | w3m -dump -T text/html
         section_end error
-    
+
         section_start logfiles "All the more or less useful logfiles"
         for i in /opt/domjudge/judgehost/judgings/*/*/*/*/*/compile.out; do
             echo $i;

misc-tools/import-contest.in

@@ -197,11 +197,6 @@ if os.path.exists('problems.yaml') or os.path.exists('problems.json') or os.path
                 problems_file = 'problemset.yaml'
             dj_utils.upload_file(f'contests/{cid}/problems/add-data', 'data', problems_file)
 
-        # We might need to translate the problem external ID's into an internal ID (when we are in data source = local mode)
-        # For this, we get the problems from the API and create a dict with the mapping
-        problem_mapping = {problem['externalid']: problem['id']
-                           for problem in dj_utils.do_api_request(f'contests/{cid}/problems')}
-
         if os.path.exists('problems.yaml'):
             with open('problems.yaml') as problemFile:
                 problemData = yaml.safe_load(problemFile)
@@ -226,11 +221,10 @@ if os.path.exists('problems.yaml') or os.path.exists('problems.json') or os.path
                 exit(3)
             os.system(f'cd {problem} && zip -r \'../{problem}\' -- .timelimit *')
 
-            problem_id = problem_mapping[problem]
-            if ((not confirmIndividually) or dj_utils.confirm(f'Ready to import problem \'{problem}\' to probid={problem_id}. Continue?', True)):
+            if ((not confirmIndividually) or dj_utils.confirm(f'Ready to import problem \'{problem}\' to problem={problem}. Continue?', True)):
                 print(f'Uploading problem \'{problem}\', please be patient, this may take a while.')
                 response = dj_utils.upload_file(
-                    f'contests/{cid}/problems', 'zip', f'{problem}.zip', {'problem': problem_id})
+                    f'contests/{cid}/problems', 'zip', f'{problem}.zip', {'problem': problem})
                 print(json.dumps(response, indent=4))
     else:
         print('Skipping contest import.')