Merge pull request #139 from EyeSeeTea/development

Ramon-Jimenez · web-flow · commit 2f0ec8edd28f · 2025-06-10T12:59:42.000+02:00
Release 1.15.0
diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml
@@ -0,0 +1,69 @@
+# For most projects, this workflow file will not need changing; you simply need
+# to commit it to your repository.
+#
+# You may wish to alter this file to override the set of languages analyzed,
+# or to provide custom queries or build logic.
+#
+# ******** NOTE ********
+# We have attempted to detect the languages in your repository. Please check
+# the `language` matrix defined below to confirm you have the correct set of
+# supported CodeQL languages.
+#
+name: "CodeQL Advanced"
+
+on:
+  push:
+    branches: [ "development", "master" ]
+  pull_request:
+    branches: [ "development", "master" ]
+
+jobs:
+  analyze:
+    name: Analyze (${{ matrix.language }})
+    # Runner size impacts CodeQL analysis time. To learn more, please see:
+    #   - https://gh.io/recommended-hardware-resources-for-running-codeql
+    #   - https://gh.io/supported-runners-and-hardware-resources
+    #   - https://gh.io/using-larger-runners (GitHub.com only)
+    # Consider using larger runners or machines with greater resources for possible analysis time improvements.
+    runs-on: self-hosted
+    permissions:
+      # required for all workflows
+      security-events: write
+
+      # required to fetch internal or private CodeQL packs
+      packages: read
+
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+        - language: python
+        # CodeQL supports the following values keywords for 'language': 'c-cpp', 'csharp', 'go', 'java-kotlin', 'javascript-typescript', 'python', 'ruby', 'swift'
+        # Use `c-cpp` to analyze code written in C, C++ or both
+        # Use 'java-kotlin' to analyze code written in Java, Kotlin or both
+        # Use 'javascript-typescript' to analyze code written in JavaScript, TypeScript or both
+        # To learn more about changing the languages that are analyzed or customizing the build mode for your analysis,
+        # see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning.
+        # If you are analyzing a compiled language, you can modify the 'build-mode' for that language to customize how
+        # your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v4
+
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v3
+      with:
+        languages: ${{ matrix.language }}
+        build-mode: ${{ matrix.build-mode }}
+        # If you wish to specify custom queries, you can do so here or in a config file.
+        # By default, queries listed here will override any specified in a config file.
+        # Prefix the list here with "+" to use these queries and those in the config file.
+
+        # For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
+        # queries: security-extended,security-and-quality
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v3
+      with:
+        category: "/language:${{matrix.language}}"
diff --git a/README.md b/README.md
@@ -9,7 +9,7 @@
 On Ubuntu 22.04:
 
 ```
-$ sudo apt install docker.io docker-compose python3 python3-setuptools
+$ sudo apt install docker.io docker-compose-v2 python3 python3-setuptools
 ```
 
 On Windows 10:
@@ -87,7 +87,7 @@ Some notes:
 - Use option `--deploy-path` to run the container with a deploy path namespace (i.e: `--deploy-path=dhis2` serves `http://localhost:8080/dhis2`)
 - Use option `-k`/`--keep-containers` to re-use existing docker containers, so data from the previous run will be kept.
 - Use option `-auth` to pass the instance authentication (`USER:PASS`). It will be used to call post-tomcat scripts.
-- Use option `--run-sql=DIRECTORY` to run SQL files (.sql, .sql.gz or .dump files) after the DB has been initialized.
+- Use option `--run-sql=DIRECTORY` to run SQL files (.sql, .sql.gz or .dump files) after the DB has been initialized. SQL files containing "strict" in their name will cause `d2-docker start` to stop if an error occurs.
 - Use option `--run-scripts=DIRECTORY` to run shell scripts (.sh) from a directory within the `dhis2-core` container. By default, a script is run **after** postgres starts (`host=db`, `port=5432`) but **before** Tomcat starts; if its filename starts with prefix "post", it will be run **after** Tomcat is available. `curl` and typical shell tools are available on that Alpine Linux environment. Note that the Dhis2 endpoint is always `http://localhost:8080/${deployPath}`, regardless of the public port that the instance is exposed to.
 - Use option `--java-opts="JAVA_OPTS"` to override the default JAVA_OPTS for the Tomcat process. That's tipically used to set the maximum/initial Heap Memory size (for example: `--java-opts="-Xmx3500m -Xms2500m"`)
 - Use option `--postgis-version=13-3.1-alpine` to specify the PostGIS version to use. By default, 10-2.5-alpine is used.
diff --git a/requirements.txt b/requirements.txt
@@ -1,6 +1,6 @@
-Flask==2.0.3
+Flask==3.1.0
 python-dotenv
-Flask_Cors==3.0.10
-Werkzeug==2.3.7
-requests==2.31.0
+Flask_Cors==6.0.0
+requests==2.32.3
 PyYAML
+setuptools
diff --git a/setup.py b/setup.py
@@ -4,7 +4,7 @@
 
 setuptools.setup(
     name="d2_docker",
-    version="1.14.0",
+    version="1.15.0.b2",
     description="Dockers for DHIS2 instances",
     long_description=open("README.md", encoding="utf-8").read(),
     keywords=["python"],
diff --git a/src/d2_docker/commands/commit.py b/src/d2_docker/commands/commit.py
@@ -10,11 +10,16 @@ def setup(parser):
         nargs="?",
         help="Destination Docker image (same if empty)",
     )
-
+    parser.add_argument(
+        "--folders",
+        nargs="+",
+        choices=utils.default_folders,
+        help="Folders to include in the image",
+    )
 
 def run(args):
     image_name = args.image or utils.get_running_image_name()
     docker_dir = utils.get_docker_directory("data", args)
     temp_dir = utils.get_temp_base_directory(args)
     utils.logger.info("Commit image: {}".format(image_name))
-    utils.build_image_from_source(docker_dir, image_name, image_name, temp_dir)
+    utils.build_image_from_source(docker_dir, image_name, image_name, temp_dir, args.folders)
diff --git a/src/d2_docker/commands/create.py b/src/d2_docker/commands/create.py
@@ -52,7 +52,9 @@ def get_core_build_dir(args):
     if not major_version:
         raise utils.D2DockerError("Cannot get version from --version or --war")
     else:
-        if major_version >= 41:
+        if major_version >= 42:
+            return os.path.join(base_dir, "java-17-tomcat-10")
+        elif major_version >= 41:
             return os.path.join(base_dir, "java-17")
         else:
             return os.path.join(base_dir, "java-11")
diff --git a/src/d2_docker/commands/run_sql.py b/src/d2_docker/commands/run_sql.py
@@ -1,5 +1,6 @@
 import subprocess
 from d2_docker import utils
+from d2_docker.utils import get_pg_dump_command
 
 DESCRIPTION = "Run SQL or open interactive session in a d2-docker container"
 NAME = "run-sql"
@@ -29,7 +30,7 @@ def run(args):
         utils.logger.debug("DB container: {}".format(db_container))
 
         if args.dump:
-            cmd = ["docker", "exec", db_container, "pg_dump", "-U", "dhis", "dhis2"]
+            cmd = ["docker", "exec", db_container, *get_pg_dump_command(compress=False)]
             utils.logger.info("Dump SQL for image {}".format(image_name))
             utils.run(cmd, raise_on_error=False)
         else:
@@ -54,7 +55,7 @@ def get_stream_db(image):
         raise utils.D2DockerError("Container must be running to dump database")
 
     db_container = status["containers"]["db"]
-    cmd_parts = ["docker", "exec", db_container, "pg_dump", "-U", "dhis", "dhis2", "|", "gzip"]
+    cmd_parts = ["docker", "exec", db_container, *get_pg_dump_command()]
     cmd = subprocess.list2cmdline(cmd_parts)
     utils.logger.info("Dump SQL for image: {}".format(cmd))
 
diff --git a/src/d2_docker/commands/version.py b/src/d2_docker/commands/version.py
@@ -1,4 +1,4 @@
-import pkg_resources  # part of setuptools
+from importlib.metadata import distribution
 import re
 from d2_docker import utils
 
@@ -11,7 +11,7 @@ def setup(_parser):
 
 def run(_args):
     utils.run(["docker", "-v"])
-    utils.run(["docker-compose", "-v"])
+    utils.run(["docker", "compose", "version"])
     d2_docker = get_d2_docker_version()
     if d2_docker:
         print("d2-docker version {}".format(d2_docker))
@@ -32,11 +32,11 @@ def get_field(command, field_index):
 
 def get_versions():
     docker = get_field(["docker", "-v"], 2)
-    docker_compose = get_field(["docker-compose", "-v"], 3)
+    docker_compose = get_field(["docker", "compose", "version"], 3)
     d2_docker = get_d2_docker_version()
     return dict({"docker": docker, "docker-compose": docker_compose, "d2-docker": d2_docker})
 
 
 def get_d2_docker_version():
-    resources = pkg_resources.require("d2-docker")
-    return resources[0].version if resources else None
+    dist = distribution("d2-docker")
+    return dist.version
diff --git a/src/d2_docker/config/dhis2-core-start.sh b/src/d2_docker/config/dhis2-core-start.sh
@@ -17,7 +17,9 @@ export PGPASSWORD="dhis"
 
 dhis2_url="http://localhost:8080/$DEPLOY_PATH"
 dhis2_url_with_auth="http://$DHIS2_AUTH@localhost:8080/$DEPLOY_PATH"
-psql_cmd="psql -v ON_ERROR_STOP=0 --quiet -h db -U dhis dhis2"
+psql_base_cmd="psql --quiet -h db -U dhis dhis2"
+psql_cmd="$psql_base_cmd -v ON_ERROR_STOP=0"
+psql_strict_cmd="$psql_base_cmd -v ON_ERROR_STOP=1"
 pgrestore_cmd="pg_restore -h db -U dhis -d dhis2"
 configdir="/config"
 homedir="/dhis2-home-files"
@@ -54,10 +56,26 @@ run_sql_files() {
     find "$base_db_path" -type f \( -name '*.sql' \) |
         sort | while read -r path; do
         echo "Load SQL: $path"
-        $psql_cmd <"$path" || true
+        exit_code=0
+        run_psql_cmd "$path" || exit_code=$?
+        if [ "$exit_code" -gt 0 ]; then
+            echo "Exit code: $exit_code"
+            exit "$exit_code"
+        fi
     done
 }
 
+run_psql_cmd() {
+    local path=$1
+    if [[ "$path" == *strict* ]]; then
+        echo "Strict mode: $path"
+        $psql_strict_cmd < "$path"
+    else
+        echo "Normal mode: $path"
+        $psql_cmd < "$path"
+    fi
+}
+
 run_pre_scripts() {
     find "$scripts_dir" -type f -name '*.sh' ! \( -name 'post*' \) | sort | while read -r path; do
         debug "Run pre-tomcat script: $path"
@@ -154,7 +172,7 @@ run() {
     else
         debug "Container: clean. Load DB"
         wait_for_postgres
-        run_sql_files || true
+        run_sql_files
         run_pre_scripts || true
         init_done
     fi
diff --git a/src/d2_docker/config/server.xml b/src/d2_docker/config/server.xml
@@ -25,6 +25,7 @@
             connectionTimeout="20000"
             URIEncoding="UTF-8"
             relaxedQueryChars='\ { } | [ ]'
+            maxHttpHeaderSize="32768"
         />
 
         <Connector
diff --git a/src/d2_docker/docker-compose.yml b/src/d2_docker/docker-compose.yml
@@ -1,4 +1,3 @@
-version: "3"
 services:
     core:
         image: "${DHIS2_CORE_IMAGE}"
@@ -22,7 +21,7 @@ services:
             DHIS2_AUTH: "${DHIS2_AUTH}"
         entrypoint: bash /config/dhis2-core-entrypoint.sh
         command: bash /config/dhis2-core-start.sh
-        restart: unless-stopped
+        restart: "no"
         depends_on:
             - "db"
             - "data"
diff --git a/src/d2_docker/images/dhis2-core/java-17-tomcat-10/Dockerfile b/src/d2_docker/images/dhis2-core/java-17-tomcat-10/Dockerfile
@@ -0,0 +1,31 @@
+FROM tomcat:10.1.36-jre21-temurin
+
+ENV DHIS2_HOME=/DHIS2_home
+ENV DHIS2_CERT=/DHIS2_home/who_pub_cert.cert
+ENV DATA_DIR=/data
+
+COPY docker-entrypoint.sh /usr/local/bin/
+RUN rm -rf /usr/local/tomcat/webapps/* && \
+    mkdir -p /usr/local/tomcat/webapps/ROOT && \
+    chmod +rx /usr/local/bin/docker-entrypoint.sh && \
+    mkdir $DHIS2_HOME && \
+    mkdir $DATA_DIR && \
+    addgroup --system tomcat && \
+    usermod -aG tomcat root && \
+    useradd --shell /bin/bash --uid 101 --gid tomcat tomcat
+
+COPY who_pub_cert.cert $DHIS2_CERT
+RUN  chmod +rx $DHIS2_CERT
+RUN keytool -importcert -alias who_mail_ichigoout -file $DHIS2_CERT -keystore /opt/java/openjdk/lib/security/cacerts -storepass changeit -noprompt
+
+
+RUN apt-get update
+RUN echo 'You can disregard the warning in noninteractive installations:' \
+         '"debconf: delaying package configuration, since apt-utils is not installed"'
+RUN apt-get install --no-install-recommends -y \
+        unzip curl postgresql-client fonts-dejavu fontconfig util-linux
+
+COPY dhis.war /usr/local/tomcat/webapps/ROOT.war
+COPY dhis2-home-files /dhis2-home-files
+
+CMD ["catalina.sh", "run"]
diff --git a/src/d2_docker/images/dhis2-core/java-17-tomcat-10/docker-entrypoint.sh b/src/d2_docker/images/dhis2-core/java-17-tomcat-10/docker-entrypoint.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+#
+# This file should be basically the same as config/dhis2-core-entrypoint.sh
+#
+set -e  # exit on errors
+
+WARFILE=/usr/local/tomcat/webapps/ROOT.war
+TOMCATDIR=/usr/local/tomcat
+DHIS2HOME=/DHIS2_home
+DATA_DIR=/data
+
+if [ "$(id -u)" = "0" ]; then
+    if [ -f $WARFILE ]; then
+        unzip -q $WARFILE -d $TOMCATDIR/webapps/ROOT
+        rm -v $WARFILE  # just to save space
+    fi
+
+    mkdir -p $DATA_DIR/apps
+    chown -R tomcat:tomcat $TOMCATDIR $DATA_DIR/apps $DHIS2HOME
+    chmod -R u=rwX,g=rX,o-rwx $TOMCATDIR $DATA_DIR/apps $DHIS2HOME
+
+    # Launch the given command as tomcat, in two ways for backwards compatibility:
+    if [ "$(grep '^ID=' /etc/os-release)" = "ID=alpine" ]; then
+        # The alpine linux way (for old images).
+        exec su-exec tomcat "$0" "$@"
+    else
+        # The ubuntu way (for new images).
+        exec setpriv --reuid=tomcat --regid=tomcat --init-groups "$0" "$@"
+    fi
+fi
+
+exec "$@"
diff --git a/src/d2_docker/images/dhis2-core/java-17-tomcat-10/who_pub_cert.cert b/src/d2_docker/images/dhis2-core/java-17-tomcat-10/who_pub_cert.cert
@@ -0,0 +1,35 @@
+-----BEGIN CERTIFICATE-----
+MIIGJDCCBQygAwIBAgIQJUJnlAcKVsfQ9MMGpFjCBTANBgkqhkiG9w0BAQsFADCB
+jzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G
+A1UEBxMHU2FsZm9yZDEYMBYGA1UEChMPU2VjdGlnbyBMaW1pdGVkMTcwNQYDVQQD
+Ey5TZWN0aWdvIFJTQSBEb21haW4gVmFsaWRhdGlvbiBTZWN1cmUgU2VydmVyIENB
+MB4XDTIyMDMxNjAwMDAwMFoXDTIzMDQxNjIzNTk1OVowFDESMBAGA1UEAwwJKi53
+aG8uaW50MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxSHGIzs9hI1G
+hoQpRqA6AJJHtV5UNU7HYGqewe/g/hO78YKa3GkCr2nXVz42fDrwtKI6qyTL8R4t
+JlqxK7NgorwQK7swZlQq77pj0jGo2NaxqI68oaoz8Itkg361jlk2LJQrrSJBGemc
+2vZC7o1hWZLwcf8PwSBkKTqjZjO7th0ESEUt8ic6YZxEWaWdJeZUc+xDGpM439L+
+mPeZKmwLzKUk41DPe1LBjeVCKOlTnFaGTX8/PescoifgAV+kZ9G+hmu5aZYVx2n+
+w8nVc4aIOiQow1LOtk8CDvBPoDnJMOpKEZobXcqcTDvbFnG4mzUnc6GXDWLaF/kJ
+JMvRKoTsQwIDAQABo4IC9DCCAvAwHwYDVR0jBBgwFoAUjYxexFStiuF36Zv5mwXh
+uAGNYeEwHQYDVR0OBBYEFHTQH+RF7rX7AXEqzVXnkXexoDIEMA4GA1UdDwEB/wQE
+AwIFoDAMBgNVHRMBAf8EAjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcD
+AjBJBgNVHSAEQjBAMDQGCysGAQQBsjEBAgIHMCUwIwYIKwYBBQUHAgEWF2h0dHBz
+Oi8vc2VjdGlnby5jb20vQ1BTMAgGBmeBDAECATCBhAYIKwYBBQUHAQEEeDB2ME8G
+CCsGAQUFBzAChkNodHRwOi8vY3J0LnNlY3RpZ28uY29tL1NlY3RpZ29SU0FEb21h
+aW5WYWxpZGF0aW9uU2VjdXJlU2VydmVyQ0EuY3J0MCMGCCsGAQUFBzABhhdodHRw
+Oi8vb2NzcC5zZWN0aWdvLmNvbTAdBgNVHREEFjAUggkqLndoby5pbnSCB3doby5p
+bnQwggF+BgorBgEEAdZ5AgQCBIIBbgSCAWoBaAB3AK33vvp8/xDIi509nB4+GGq0
+Zyldz7EMJMqFhjTr3IKKAAABf5MeKkoAAAQDAEgwRgIhAINPWOeOK9LR1KvwjXy3
+tq+VOtOl2pbxUSGpKb/on/5fAiEAl/zai8TthJdRZGYAQ1AJVY83+VPNlrAmYt4Y
+aATC6GoAdgB6MoxU2LcttiDqOOBSHumEFnAyE4VNO9IrwTpXo1LrUgAAAX+THioV
+AAAEAwBHMEUCIQC5LVSJjK/+axH2ygQBQvZBteFsrpfrjTwlxRUZIOVOegIgfe+9
+UxpopGPMoxAinRvmhV80DC2SjyB5tJ5gxdEwjy0AdQDoPtDaPvUGNTLnVyi8iWvJ
+A9PL0RFr7Otp4Xd9bQa9bgAAAX+THiniAAAEAwBGMEQCIGzDAwbnDDooUiBzsIgx
+qQTH9LWRw/sUP6oPqC/TCy5IAiBv5zoBNQRFF5xL747h+fonfzO29cOTOxII7OOE
+tB5PtDANBgkqhkiG9w0BAQsFAAOCAQEAZoC2S+nrxP2loGfjDvbjo0N9cc7KhCJv
+XPAG3qvc1T4RI4ZCfTKY5vpFYqu6E/TMdGlDTXD3NxheJyScm51x5tXGfvIsnKDc
+//zWyUXvZPnJWLEjP4JKsRb/Gi8hIPfLP0PST1xjk+JlxE+vfbHfH5BUegfpfBgv
+Fu+HiJEFS3ZDE6Im61Hq5BdMdLF84u5khs0smbWOyP4tP9fqD6gnq0RV+gcI2JzU
+LLc0mPIpLqQ87zvj/Re0rMGS/JhXp4cAyvIuwi1AuechGjQR1zeBNgVIkxd+ccnc
+OaI6t3HH7OQ2K8pCNLa4PyoAJL/XnIwq0FwvqSVKW8h3vTdZ4Z0+Jg==
+-----END CERTIFICATE-----
diff --git a/src/d2_docker/utils.py b/src/d2_docker/utils.py
