fix: certbot deploy hook uses Podman REST API via Python; fix EL9 cgroup/SELinux settings

Root cause: certbot container (Alpine-based) does not include the `podman`
binary, so the v1.0.0 deploy hook failed silently when trying to run
`podman restart`.  Separately, the compose files used `cgroupns: private`
which crashes the testpoint container's internal systemd on EL9 hosts.
Diagnosed and fixed on psum01.aglt2.org (Feb 2026).

certbot-deploy-hook.sh (v1.0.0 → v2.0.0):
- Replace `podman restart` CLI call with Python HTTP client talking to
  the Podman REST API over the mounted Unix socket
  (/run/podman/podman.sock). python3 is available in the Alpine certbot
  image; no additional packages required.
- Removes dependency on the `podman` binary inside the container.
- Add SHA256 checksum file (certbot-deploy-hook.sh.sha256).

docker-compose.testpoint-le{,-auto}.yml:
- testpoint: replace `privileged: true` + `cgroupns: private` with
  `cgroup: host` + `/sys/fs/cgroup` volume mount + `tty: true`.
  The old settings prevented systemd from running inside the container
  on EL9. Remove `CAP_SYS_ADMIN` and `CAP_SYS_PTRACE` (not needed).
  Increase healthcheck start_period to 60s (allows systemd more time).
- certbot: add `security_opt: label=disable` so SELinux does not block
  the container from accessing the host Podman socket.

install-perfsonar-testpoint.md:
- Correct deploy hook troubleshooting path from /opt/certbot/deploy-hook.sh
  to the correct /etc/letsencrypt/renewal-hooks/deploy/certbot-deploy-hook.sh.
- Expand deploy hook description: note Python REST API usage and the
  SELinux security_opt requirement for EL9 hosts.
- Add actionable verification commands to the troubleshooting section.

Author: root

docs/perfsonar/tools_scripts/certbot-deploy-hook.sh

@@ -1,27 +1,54 @@
 #!/bin/sh
 # certbot-deploy-hook.sh
 # This script is executed by Certbot's --deploy-hook after a successful renewal.
-# It gracefully restarts the perfsonar-testpoint container to load the new certificate.
+# It restarts the perfsonar-testpoint container to load the new certificate by
+# calling the Podman REST API over the mounted Unix socket.
 #
-# It requires the host's Podman socket to be mounted into the certbot container.
+# Requirements (inside the certbot container):
+#   - /run/podman/podman.sock mounted from the host (read-only)
+#   - python3 available (present in docker.io/certbot/certbot:latest on Alpine)
 #
-# Version: 1.0.0
+# The compose file must mount the socket and disable SELinux labeling:
+#   volumes:
+#     - /run/podman/podman.sock:/run/podman/podman.sock:ro
+#   security_opt:
+#     - label=disable
+#
+# Version: 2.0.0
 
 set -eu
 
-# The name of the container to restart
+SOCKET="/run/podman/podman.sock"
 TARGET_CONTAINER="perfsonar-testpoint"
+STOP_TIMEOUT=30
+
+echo "[INFO] Certbot deploy hook triggered for domains: ${RENEWED_DOMAINS:-unknown}"
+echo "[INFO] Restarting container '${TARGET_CONTAINER}' via Podman socket..."
 
-echo "[INFO] Certbot deploy hook triggered for domains: $RENEWED_DOMAINS"
-echo "[INFO] Attempting to gracefully restart container: $TARGET_CONTAINER"
+python3 - <<PYEOF
+import http.client, socket, sys
 
-# Use the mounted Podman socket to restart the container on the host
-# The --time=30 gives the container 30 seconds to shut down gracefully
-podman restart --time=30 "$TARGET_CONTAINER"
+class _UnixConn(http.client.HTTPConnection):
+    def __init__(self, path):
+        super().__init__("localhost")
+        self._path = path
+    def connect(self):
+        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+        self.sock.connect(self._path)
 
-if [ $? -eq 0 ]; then
-  echo "[SUCCESS] Container '$TARGET_CONTAINER' restarted successfully."
-else
-  echo "[ERROR] Failed to restart container '$TARGET_CONTAINER'." >&2
-  exit 1
-fi
+conn = _UnixConn("${SOCKET}")
+try:
+    conn.request("POST",
+                 "/v4.0.0/containers/${TARGET_CONTAINER}/restart?t=${STOP_TIMEOUT}")
+    resp = conn.getresponse()
+    body = resp.read().decode()
+    if resp.status == 204:
+        print("[SUCCESS] Container '${TARGET_CONTAINER}' restarted successfully.")
+        sys.exit(0)
+    else:
+        print(f"[ERROR] Podman API returned {resp.status}: {body}", file=sys.stderr)
+        sys.exit(1)
+except Exception as e:
+    print(f"[ERROR] Failed to contact Podman socket at ${SOCKET}: {e}", file=sys.stderr)
+    sys.exit(1)
+PYEOF

docs/perfsonar/tools_scripts/certbot-deploy-hook.sh.sha256

@@ -0,0 +1 @@
+6074f4dca208bcc556afa3155e8c295a238e59524a26c2dfe960fd85163c70bf

docs/perfsonar/tools_scripts/docker-compose.testpoint-le-auto.yml

@@ -7,8 +7,7 @@ services:
     labels:
       - io.containers.autoupdate=registry
     network_mode: "host"
-    privileged: true
-    cgroupns: private
+    cgroup: host
     environment:
       - TZ=UTC
       # Optional: Set SERVER_FQDN to explicitly specify which Let's Encrypt certificate to use.
@@ -23,23 +22,23 @@ services:
       - /run/lock
       - /tmp
     volumes:
+      - /sys/fs/cgroup:/sys/fs/cgroup:rw
       - /opt/perfsonar-tp/psconfig:/etc/perfsonar/psconfig:Z
       - /var/www/html:/var/www/html:z
       - /etc/apache2:/etc/apache2:z
       - /etc/letsencrypt:/etc/letsencrypt:z
       # Mount the tools_scripts directory so the entrypoint wrapper is available
       - /opt/perfsonar-tp/tools_scripts:/opt/perfsonar-tp/tools_scripts:ro
+    tty: true
     pids_limit: 8192
     cap_add:
       - CAP_NET_RAW
-      - CAP_SYS_ADMIN
-      - CAP_SYS_PTRACE
     healthcheck:
       test: ["CMD-SHELL", "curl -kSfS https://localhost/ || exit 1"]
       interval: 30s
       timeout: 10s
       retries: 3
-      start_period: 30s
+      start_period: 60s
 
   certbot:
     image: docker.io/certbot/certbot:latest
@@ -51,7 +50,12 @@ services:
     # on port 80, avoiding conflicts. Both containers share the host network
     # namespace without collision.
     network_mode: "host"
-    # Mount the host's Podman socket to allow the deploy hook to restart containers
+    # SELinux: disable label confinement so the container can access the host
+    # Podman socket. Required on EL9 hosts with SELinux in enforcing mode.
+    security_opt:
+      - label=disable
+    # Mount the host's Podman socket so the deploy hook can call the Podman REST
+    # API to restart the testpoint container after renewal.
     volumes:
       - /run/podman/podman.sock:/run/podman/podman.sock:ro
       - /var/www/html:/var/www/html:Z

docs/perfsonar/tools_scripts/docker-compose.testpoint-le.yml

@@ -7,8 +7,7 @@ services:
     labels:
       - io.containers.autoupdate=registry
     network_mode: "host"
-    privileged: true
-    cgroupns: private
+    cgroup: host
     environment:
       - TZ=UTC
     restart: unless-stopped
@@ -17,21 +16,21 @@ services:
       - /run/lock
       - /tmp
     volumes:
+      - /sys/fs/cgroup:/sys/fs/cgroup:rw
       - /opt/perfsonar-tp/psconfig:/etc/perfsonar/psconfig:Z
       - /var/www/html:/var/www/html:z
       - /etc/apache2:/etc/apache2:z
       - /etc/letsencrypt:/etc/letsencrypt:z
+    tty: true
     pids_limit: 8192
     cap_add:
       - CAP_NET_RAW
-      - CAP_SYS_ADMIN
-      - CAP_SYS_PTRACE
     healthcheck:
       test: ["CMD-SHELL", "curl -kSfS https://localhost/ || exit 1"]
       interval: 30s
       timeout: 10s
       retries: 3
-      start_period: 30s
+      start_period: 60s
 
   certbot:
     image: docker.io/certbot/certbot:latest
@@ -43,7 +42,12 @@ services:
     # on port 80, avoiding conflicts. Both containers share the host network
     # namespace without collision.
     network_mode: "host"
-    # Mount the host's Podman socket to allow the deploy hook to restart containers
+    # SELinux: disable label confinement so the container can access the host
+    # Podman socket. Required on EL9 hosts with SELinux in enforcing mode.
+    security_opt:
+      - label=disable
+    # Mount the host's Podman socket so the deploy hook can call the Podman REST
+    # API to restart the testpoint container after renewal.
     volumes:
       - /run/podman/podman.sock:/run/podman/podman.sock:ro
       - /var/www/html:/var/www/html:Z

docs/personas/quick-deploy/install-perfsonar-testpoint.md

@@ -645,7 +645,9 @@ The certbot container runs a renewal loop that checks for expiring certificates
 a deploy hook script (`certbot-deploy-hook.sh`) that gracefully restarts the `perfsonar-testpoint`
 container. This ensures the new certificates are loaded without manual intervention. The deploy hook
 uses the mounted Podman socket (`/run/podman/podman.sock`) to communicate with the host's container
-runtime.
+runtime via the Podman REST API (using Python, since the `podman` CLI is not present in the certbot
+image). On EL9 hosts with SELinux enforcing, the certbot service requires `security_opt: label=disable`
+to access the socket.
 
 **Note:** The certbot container in this setup uses **host networking mode** (via `network_mode: host` in the
 compose file) so it can bind directly to port 80 for HTTP-01 challenges during renewals. This works
@@ -1196,9 +1198,10 @@ Perform these checks before handing the host over to operations:
     
     **Solutions:**
     
-    - Ensure certbot container has deploy hook configured: `--deploy-hook /opt/certbot/deploy-hook.sh`
-    - Verify Podman socket is mounted in certbot container
-    - Manually restart testpoint after renewals if deploy hook fails
+    - Ensure certbot container has deploy hook configured: `--deploy-hook /etc/letsencrypt/renewal-hooks/deploy/certbot-deploy-hook.sh`
+    - Verify Podman socket is mounted in certbot container: `podman exec certbot ls /run/podman/podman.sock`
+    - Verify `security_opt: label=disable` is set on the certbot service in `docker-compose.yml` (required on EL9/SELinux hosts)
+    - Manually restart testpoint after renewals if deploy hook fails: `podman restart perfsonar-testpoint`
 
 ### perfSONAR Service Issues