[_502] test harness and container-based tests

A new test harness is introduced in which we construct a new
container (using either Docker or podman) for each test program
we run. This allows full customization of the container
environment for the particular needs of each test.

Accordingly, included with the Github workflows (in a separate
commit) is a full run of the PRC test suite with the iRODS server
and catalog DB server running in the same container as the client.

In the process of putting old tests through new rigors, faults
were found and corrected in some of those tests.

Author: d-w-moore

Dockerfile.prc_test.centos

@@ -24,6 +24,5 @@ RUN  python${py_N} repo/docker_build/iinit.py \
         password rods
 SHELL ["/bin/bash","-c"]
 CMD  echo "Waiting on iRODS server... " ; \
-     python${PY_N} repo/docker_build/recv_oneshot -h irods-provider -p 8888 -t 360 && \
      sudo groupadd -o -g $(stat -c%g /irods_shared) irods && sudo usermod -aG irods user && \
      newgrp irods < repo/run_python_tests.sh

Dockerfile.prc_test.ubuntu

@@ -31,6 +31,5 @@ SHELL ["/bin/bash","-c"]
 #  3. run python tests as the new group
 
 CMD  echo "Waiting on iRODS server... " ; \
-     python${PY_N} repo/docker_build/recv_oneshot -h irods-provider -p 8888 -t 360 && \
      sudo groupadd -o -g $(stat -c%g /irods_shared) irods && sudo usermod -aG irods user && \
      newgrp irods < repo/run_python_tests.sh

irods/message/__init__.py

@@ -181,7 +181,7 @@ def ET(xml_type=(), server_version=None):
 
 logger = logging.getLogger(__name__)
 
-IRODS_VERSION = (5, 0, 1, "d")
+IRODS_VERSION = (5, 0, 2, "d")
 
 UNICODE = str
 

irods/test/access_test.py

@@ -533,11 +533,7 @@ def test_iRODSAccess_cannot_be_constructed_using_unsupported_type__issue_558(sel
         # Before the fix in #558, this would have been allowed and only later would the type discrepancy be revealed,
         # leading to opaque error messages. Now, the types are checked on the way in to ensure clarity and correctness.
         # TODO(#480): We cannot use the unittest.assertRaises context manager as this was introduced in python 3.1.
-        assertCall = getattr(self, "assertRaisesRegex", None)
-        if assertCall is None:
-            assertCall = self.assertRaisesRegexp
-
-        assertCall(
+        self.assertRaisesRegex(
             TypeError,
             "'path' parameter must be of type 'str', 'irods.collection.iRODSCollection', "
             "'irods.data_object.iRODSDataObject', or 'irods.path.iRODSPath'.",

irods/test/exception_test.py

@@ -41,8 +41,8 @@ def test_400(self):
             excep_repr = repr(exc)
             errno_object = irods.exception.Errno(errno.EACCES)
             errno_repr = repr(errno_object)
-            self.assertRegexpMatches(errno_repr, r"\bErrno\b")
-            self.assertRegexpMatches(
+            self.assertRegex(errno_repr, r"\bErrno\b")
+            self.assertRegex(
                 errno_repr, """['"]{msg}['"]""".format(msg=os.strerror(errno.EACCES))
             )
             self.assertIn(errno_repr, excep_repr)

irods/test/harness/000_install-irods.Dockerfile

@@ -0,0 +1,14 @@
+FROM ubuntu:22.04
+COPY install.sh /
+ARG irods_package_version
+ENV IRODS_PACKAGE_VERSION "$irods_package_version"
+RUN for phase in initialize install-essential-packages add-package-repo; do \
+        bash /install.sh --w=$phase 0; \
+    done
+RUN /install.sh 4
+COPY start_postgresql_and_irods.sh manage_irods5_procs /
+RUN apt install -y sudo
+RUN useradd -ms/bin/bash testuser
+RUN echo 'testuser ALL=(ALL) NOPASSWD: ALL' >>/etc/sudoers
+RUN apt install -y faketime
+CMD bash /start_postgresql_and_irods.sh

irods/test/harness/001_bats-python3.Dockerfile

@@ -0,0 +1,5 @@
+FROM install-irods
+RUN apt update; apt install -y python3-pip bats
+RUN python3 -m pip install --upgrade pip
+RUN python3 -m pip install virtualenv
+RUN python3 -m virtualenv /py3

irods/test/harness/002_ssl-and-pam.Dockerfile

@@ -0,0 +1 @@
+FROM bats-python3

irods/test/harness/003_compile-specific-python.Dockerfile

@@ -0,0 +1,17 @@
+FROM ssl-and-pam
+RUN apt update
+RUN apt install -y wget build-essential
+RUN apt install -y libssl-dev zlib1g-dev libffi-dev libncurses-dev wget build-essential
+ARG python_version
+RUN wget https://www.python.org/ftp/python/${python_version}/Python-${python_version}.tar.xz
+RUN touch /tmp/aaaaaaaaaa # dummmy change
+RUN tar xf Python-${python_version}.tar.xz
+WORKDIR /Python-${python_version}
+RUN ./configure --prefix /root/python --with-ensurepip=install
+RUN make -j
+RUN mkdir /root/python
+RUN make install
+WORKDIR /
+RUN /root/python/bin/python3 -m pip install virtualenv
+RUN chmod a+rx /root
+ENV PYTHON_VERSION=${python_version}

irods/test/harness/README.txt

@@ -0,0 +1,56 @@
+DOCKER POWERED TEST HARNESS
+
+a. DESCRIPTION
+
+A series of docker images which support running isolated test scripts (using BATS, bash, or Python).
+Once built, the images allow loading and customizing the Docker container environment for a given 
+test script.
+
+The general form for test invocation is: docker_container_driver.sh  <path_to_script_on_host>
+
+Within the container, a computed internal path to the same script is executed, whether directly or
+indirectly by a wrapper script.  The wrapper for many of the PRC authentication-via-PAM tests is
+irods/test/login_auth_test.sh.
+
+The test_script_parameters file, located in the irods/test/harness directory, contains customized
+settings for each test script run, including:
+
+   - Docker image name to be used.
+
+   - Wrapper to be invoked, if any.  Wrappers shall perform common setup tasks up to and including
+     invoking the test script itself.
+
+   - Which user is running the test.  (Unless otherwise specified, this is the passwordless-sudo-
+     enabled user 'testuser').
+
+When done with a test, the docker_container_driver.sh exit code mirrors the return code from the
+run of the test script. The container itself is removed unless the -L ("leak") option is given.
+
+b. SAMPLE RUNS
+
+To build required images
+------------------------
+For our convenience in this doc, set a shell variable REPOROOT to ~/python-irodsclient (or
+similar) to specify the path to the top level of the local repository.
+
+Sample command lines to build Docker images:
+
+    1) cd $REPO_ROOT/irods/test/harness
+        ./build_docker.sh
+
+       Builds docker images in proper sequence.
+
+    2) cd $REPO_ROOT/irods/test/harness;
+       IRODS_PACKAGE_VERSION=4.3.4 PYTHON_VERSION=3.11 NO_CACHE=1 ./build-docker.sh [ Dockerfiles... ]
+
+       Builds (ignoring docker cache) images based on specific iRODS package version and desired
+       Python Interpreter version, optionally with a restricted list of Docker files in need of rebulding.
+
+Sample command lines to run a test script.
+
+$REPO_ROOT/irods/test/harness/docker_container_driver.sh $REPO_ROOT/irods/test/scripts/run_local_suite
+
+For both builder and driver script, the environment variable DOCKER may be set to "podman" to run
+the alternative container engine.  It will otherwise default to a value of "docker".
+
+