Switch from deadsnakes to pyenv for base pythons (#38)

Several other changes included:

- update pre-commit hooks
- add vscode repo settings for auto format
- fix an issue with newer `lrzip` tools; it now spits out stderr noise
  unless the `-Q` option is set (previously just `-q` was enough)
- switch from a bash script to invoke for driving the various tests

_Intense_ hacks to support both running locally by mounting the cwd into
the container; need to match host UID+GUID to permit sane read/writing
to the mounted cwd... as well as running in CI, where we need to be root
to clone the repo (why!!!), but we MUST be non-root to pass one of the
tests (`can't write to directory`). Very sad workarounds to get
everything going, I'm not too happy about it but it does work.

Lastly, switch to running ci off a container hosted under dtrx-py
organization on the github container registry.

Invoke task for building + pushing the image. And updated documentation
in the README.

Author: noahp

.dockerignore

@@ -0,0 +1,3 @@
+*
+!/requirements.txt
+!/entrypoint.sh

.github/workflows/main.yml

@@ -10,52 +10,76 @@ on:
 
 jobs:
   build:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
+    container:
+      image: ghcr.io/dtrx-py/dtrx:2022-09-16
+      options: "--user=root"
 
     steps:
-      # checkout v2, with recursive submodule update
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         with:
           submodules: recursive
 
       - name: 🛠️ Build
         run: |
           pip install build==0.7.0
-          python -m build
+          python3 -m build
 
       - uses: actions/upload-artifact@v2
         with:
           name: release-packages
           path: dist/
   test:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
+    container:
+      image: ghcr.io/dtrx-py/dtrx:2022-09-16
+      # we need to be root for checkout to work without pain
+      options: "--entrypoint='' --user=root"
 
     strategy:
       matrix:
         python: [py27, py36, py37, py38, py39, py3.10]
 
     steps:
-      # checkout v2, with recursive submodule update
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         with:
           submodules: recursive
 
-      # build the Docker image we use to run the tests
       - name: 🧪 Run Tox Tests
-        run: TOX_ARGS='-e ${{ matrix.python }}' ./test.sh
+        run: |
+          # switch to a non-root user; this is required for one of the dtrx
+          # tests, which checks for an EACCESS error :/
+          bash /entrypoint.sh bash -c \
+            'sudo chown -R $(id -u):$(id -g) ${PWD} && TOX_ARGS="-e ${{ matrix.python }}" inv tox'
+
+  test-missing-file-command:
+    runs-on: ubuntu-22.04
+
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          submodules: recursive
+
+      # build the Docker image we use to run the tests
+      - name: ✅ Test When `file` Command Not Present
+        run: |
+          pip install -r requirements.txt
+          inv test-nonexistent-file-cmd
 
   manpage:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
+    container:
+      image: ghcr.io/dtrx-py/dtrx:2022-09-16
+      options: "--user=root"
 
     steps:
-      # checkout v2, with recursive submodule update
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         with:
           submodules: recursive
 
       # build the Docker image we use to run the tests
       - name: 📝 Generate Man Pages
-        run: RUN_JOB=rst2man ./test.sh
+        run: inv rst2man
 
       # upload-artifact to save the output wheels
       - uses: actions/upload-artifact@v2
@@ -64,13 +88,14 @@ jobs:
           path: dtrx.1
 
   windows:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
 
     steps:
-      # checkout v2, with recursive submodule update
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         with:
           submodules: recursive
 
-      - name: ❌ Verify Windows Install Fails
-        run: RUN_JOB=windows ./test.sh
+      - name: 🪟 Verify Windows Install Fails
+        run: |
+          pip install -r requirements.txt
+          inv windows

.pre-commit-config.yaml

@@ -3,7 +3,7 @@
 exclude: web/
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.1.0
+    rev: v4.3.0
     hooks:
       - id: check-added-large-files # prevents giant files from being committed.
       - id: check-case-conflict # checks for files that would conflict in case-insensitive filesystems.
@@ -21,11 +21,12 @@ repos:
       - id: fix-byte-order-marker # removes utf-8 byte order marker.
       - id: mixed-line-ending # replaces or checks mixed line ending.
       - id: pretty-format-json # This hook sets a standard for formatting JSON files.
+        exclude: .vscode
       - id: requirements-txt-fixer # sorts entries in requirements.txt.
       - id: trailing-whitespace # trims trailing whitespace.
 
   - repo: https://github.com/psf/black
-    rev: 22.3.0
+    rev: 22.8.0
     hooks:
       - id: black
 
@@ -36,7 +37,7 @@ repos:
         files: (dtrx)|(\.(py)$)
 
   - repo: https://github.com/sirosen/check-jsonschema
-    rev: 0.14.1
+    rev: 0.18.3
     hooks:
       - id: check-github-actions
       - id: check-github-workflows
@@ -52,11 +53,12 @@ repos:
         files: \./README\.md
 
   - repo: https://github.com/pre-commit/mirrors-prettier
-    rev: v2.6.1
+    rev: v3.0.0-alpha.0
     hooks:
       - id: prettier
         args: [--write]
         files: \.(js|ts|jsx|tsx|css|less|html|json|markdown|md|yaml|yml)$
+        exclude: .vscode
 
   - repo: https://github.com/pryorda/dockerfilelint-precommit-hooks
     rev: v0.1.0
@@ -75,6 +77,6 @@ repos:
       - id: shellcheck
 
   - repo: https://github.com/asottile/setup-cfg-fmt
-    rev: v1.20.0
+    rev: v2.0.0
     hooks:
       - id: setup-cfg-fmt

.python-version

@@ -0,0 +1,6 @@
+2.7.16
+3.6.15
+3.7.13
+3.8.13
+3.9.12
+3.10.4

.vscode/settings.json

@@ -1,3 +1,12 @@
 {
-  "markdown.extension.toc.levels": "2..6"
+    "editor.formatOnSave": true,
+    "python.formatting.provider": "black",
+    "python.sortImports.args": [
+        "--profile", "black"
+    ],
+    "[python]": {
+        "editor.codeActionsOnSave": {
+            "source.organizeImports": true
+        }
+    }
 }

Dockerfile

@@ -1,55 +1,84 @@
-FROM ubuntu:focal-20200319
+
+FROM ubuntu:jammy-20220815
 
 ARG DEBIAN_FRONTEND=noninteractive
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
     arj \
     binutils \
     brotli \
+    build-essential \
+    ca-certificates \
+    clang \
     cpio \
+    curl \
     file \
+    git \
     gzip \
     lhasa \
+    libbz2-dev \
     libffi-dev \
+    liblzma-dev \
+    libncursesw5-dev \
+    libreadline-dev \
+    libsqlite3-dev \
+    libssl-dev \
+    libxml2-dev \
+    libxmlsec1-dev \
+    llvm \
     lrzip \
     lzip \
+    make \
     p7zip-full \
-    python-pip-whl \
-    python2.7 \
+    python3 \
+    python3-dev \
     python3-pip \
-    python3.8 \
+    python3-venv \
+    sudo \
+    tk-dev \
     unrar \
     unzip \
     wget \
     xz-utils \
     zip \
+    zlib1g-dev \
     zstd \
-    software-properties-common && \
-    add-apt-repository ppa:deadsnakes/ppa && apt-get update && \
-    bash -c "\
-    apt-get install -y \
-    python2.7{,-dev} \
-    python3.6{,-dev} \
-    python3.7{,-dev,-distutils} \
-    python3.8{,-dev} \
-    python3.9{,-dev,-distutils}\
-    python3.10{,-dev,-distutils}\
-    python3-distutils" \
     && rm -rf /var/lib/apt/lists/*
 
-# create a user inside the container. if you specify your UID when building the
-# image, you can mount directories into the container with read-write access:
-# docker build -t "dtrx" -f Dockerfile --build-arg UID=$(id -u) .
-ARG UID=1010
-ARG UNAME=builder
-RUN useradd --uid ${UID} --create-home --user-group ${UNAME} && \
-    echo "${UNAME}:${UNAME}" | chpasswd
+# pyenv
+RUN git clone --branch v2.3.4 https://github.com/pyenv/pyenv.git /pyenv
+ENV PYENV_ROOT /pyenv
+RUN /pyenv/bin/pyenv install 2.7.16
+# openssl version on jammy (3) is too new for python 3.6, and breaks :/
+# workaround is to use clang to build it
+# https://github.com/pyenv/pyenv/issues/2239#issuecomment-1079275184
+RUN CC=clang /pyenv/bin/pyenv install 3.6.15
+RUN /pyenv/bin/pyenv install 3.7.13
+RUN /pyenv/bin/pyenv install 3.8.13
+RUN /pyenv/bin/pyenv install 3.9.12
+RUN /pyenv/bin/pyenv install 3.10.4
+
+ENV PATH=/pyenv/bin:${PATH}
 
-ENV PATH=${PATH}:/home/${UNAME}/.local/bin
+# Python requirements
+COPY requirements.txt /tmp/requirements.txt
+RUN pip3 install -r /tmp/requirements.txt
 
-USER ${UNAME}
+COPY entrypoint.sh /entrypoint.sh
+
+ARG UID=1010
+ARG USERNAME=builder
+RUN echo "root:root" | chpasswd \
+    && adduser --disabled-password --uid "${UID}" --gecos "" "${USERNAME}" \
+    && echo "${USERNAME}:${USERNAME}" | chpasswd \
+    && echo "%${USERNAME}    ALL=(ALL)   NOPASSWD:    ALL" >> /etc/sudoers.d/${USERNAME} \
+    && chmod 0440 /etc/sudoers.d/${USERNAME} \
+    && adduser ${USERNAME} sudo
 
-# Need tox to run the tests, docutils for rst2man.py
-RUN pip3 install tox==3.15.2 docutils==0.16 pyyaml==5.4.1
+# Ensure sudo group users are not
+# asked for a password when using
+# sudo command by ammending sudoers file
+RUN echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> \
+    /etc/sudoers
 
-WORKDIR /workspace
+ENTRYPOINT ["/bin/bash", "/entrypoint.sh"]

README.md

@@ -7,13 +7,15 @@ pyversions](https://img.shields.io/pypi/pyversions/dtrx.svg?style=for-the-badge&
 
 <!-- toc -->
 
-- [Changes in this repo](#changes-in-this-repo)
-- [Development](#development)
-  - [Contributions](#contributions)
-  - [Issues](#issues)
-  - [Releases](#releases)
-  - [Tests](#tests)
-  - [Linting](#linting)
+- [dtrx](#dtrx)
+  - [Changes in this repo](#changes-in-this-repo)
+  - [Development](#development)
+    - [Contributions](#contributions)
+    - [Issues](#issues)
+    - [Releases](#releases)
+    - [Tests](#tests)
+    - [Linting](#linting)
+    - [Docker](#docker)
 
 <!-- tocstop -->
 
@@ -123,3 +125,31 @@ pre-commit install
 
 pre-commit will run anytime `git commit` runs (disable with `--no-verify`). You
 can manually run it with `pre-commit run`.
+
+### Docker
+
+The tests in CI (and locally) can be run inside a Docker container, which
+provides all the tested python versions.
+
+This image is defined at [`Dockerfile`](Dockerfile). It's pushed to the GitHub
+Container Registry so it can be managed by the `dtrx-py` organization on GitHub-
+Docker Hub charges for Organizations.
+
+There are Invoke tasks for building + pushing the Docker image, which push both
+a `:latest` tag as well as a `:2022-09-16` ISO8601 numbered tag. The tag can
+then be updated in the GitHub actions runner.
+
+> Note: there's a bit of complexity around how the image is used, because the
+> dtrx tests need to run as a non-root user (there's one test that checks for
+> error handling when the output directory is not accessible by the current
+> user). To deal with this, there's an entrypoint script that switches user to a
+> non-root user, but that still has read/write access to the mounted host volume
+> (which is the cwd, intended for local developement work). This is required on
+> Linux, where it's nice to have the host+container UID+GUID matching, so any
+> changes to the mounted host volume have the same permissions set.
+>
+> In the GitHub actions runner, we need to run inside the same container (to
+> have access to the correct python versions for testing), and the github action
+> for checkout assumes it can write to somewhat arbitrary locations in the file
+> system (basically root access). So we switch to the non-root user _after_
+> checkout.