Use mounted volumes for webserver htpasswd files

aequitas · aequitas · commit 75ca94c86cd7 · 2026-04-07T20:53:32.000+02:00
- also some small fixes to documentation
- and supression of whois warning
diff --git a/Changelog.md b/Changelog.md
@@ -1,5 +1,9 @@
 # Change Log
 
+## 1.11.x (upcoming)
+
+- MONITORING_AUTH_RAW, is no longer used to configure monitoring authentication. Instead password entries must be entered directly in `/opt/Internet.nl/volumes/webserver/htpasswd/monitoring.htpasswd`. See: [Docker-Metrics](https://github.com/internetstandards/Internet.nl/blob/main/documentation/Docker-metrics.md#monitoring-user/allowlist-management). If you had configured monitoring auth previously you need to move this into the new file.
+
 ## 1.11.0 (in progress)
 
 _Compared to the latest 1.10 release._
@@ -11,7 +15,7 @@ All tests were updated to match the
 [2025-05 version of the NCSC TLS guidelines](https://www.ncsc.nl/documenten/publicaties/2025/juni/01/ict-beveiligingsrichtlijnen-voor-transport-layer-security-2025-05).
 Most significant changes:
 
-- The list of good/sufficient/phase out/insufficient TLS versions, TLS authentication, curves, hashes, 
+- The list of good/sufficient/phase out/insufficient TLS versions, TLS authentication, curves, hashes,
   key exchange algorithms, FFDHE groups, RSA key lengths, and bulk encryption algorithms were updated
   to match the new guidelines.
 - A test for Extended Master Secret (RFC7627) was added.
@@ -26,17 +30,17 @@ Most significant changes:
   [are now detected as such](https://github.com/internetstandards/Internet.nl/issues/1641).
   Several issues with OCSP stapling reliability were also resolved.
 - Issues were fixed where the cipher order failed to detect some bad scenarios,
-  including some where servers preferred RSA over ECDHE, or CBC over POLY1305. 
+  including some where servers preferred RSA over ECDHE, or CBC over POLY1305.
 - CCM_8 ciphers are now detected when enabled on a server.
 - OLD ciphers are no longer detected.
-- The cipher order test no longer separates between "the server cipher order preference is wrong" 
+- The cipher order test no longer separates between "the server cipher order preference is wrong"
   and "the server has no preference".
-  
+
 
 ### Significant internal changes
 
 - ...
-- 
+-
 ### Possibly required changes to deployments
 
 ...
@@ -105,7 +109,7 @@ The API version is updated to 2.6.0 due to the new CAA fields.
 - Fixed handling for [CAA with non-ascii characters](https://github.com/internetstandards/Internet.nl/pull/1788).
 - Fixed possible exception in [mail test prechecks](https://github.com/internetstandards/Internet.nl/pull/1787).
 - Fixed an [issue with rate limiting](https://github.com/internetstandards/Internet.nl/pull/1792).
-- Update [Django to 4.2.22](https://github.com/internetstandards/Internet.nl/pull/1795) to fix 
+- Update [Django to 4.2.22](https://github.com/internetstandards/Internet.nl/pull/1795) to fix
   [CVE-2025-48432](https://www.djangoproject.com/weblog/2025/jun/04/security-releases/).
 
 ## 1.10.0
@@ -126,7 +130,7 @@ _Compared to the latest 1.9 release._
 
 ### Significant internal changes
 
-- The test code no longer interfaces with libunbound, but 
+- The test code no longer interfaces with libunbound, but
  [uses dnspython as a stub resolver](https://github.com/internetstandards/Internet.nl/pull/1578).
 - Periodic tests [are no longer enabled by default](https://github.com/internetstandards/Internet.nl/pull/1628).
 - UWSGI [cheaper](https://uwsgi-docs.readthedocs.io/en/latest/Cheaper.html) options are used to reduce idle processes and reduce memory consumption.
@@ -157,7 +161,7 @@ docker network rm internetnl-prod_public-internet
 ## 1.9.3
 
 - Updated the [expired PGP key](https://github.com/internetstandards/Internet.nl_content/pull/57).
-  
+
 ## 1.9.2
 
 - Fixed an issue where static files incorrectly required authentication (#1676)
@@ -212,7 +216,7 @@ jobs to generate the same report over and over.
 
 1.8.7 mainly contains various important fixes to support batch deployment.
 
-* Updated sectxt to use a patched version of PGPy with a fix for a 
+* Updated sectxt to use a patched version of PGPy with a fix for a
   [catastrophic regex backtracking issue](https://github.com/SecurityInnovation/PGPy/pull/467)
 * Updated nassl to fix memory leak in OCSP check.
 * Connection test zones are now re-signed every week instead of every month.
@@ -299,7 +303,7 @@ This release has API version 2.4.0:
 
 ## 1.7.1
 
-- Fixed the new [display of TLS versions](https://github.com/internetstandards/Internet.nl/issues/944) for mail tests. 
+- Fixed the new [display of TLS versions](https://github.com/internetstandards/Internet.nl/issues/944) for mail tests.
 - Fixed a [language mix-up](https://github.com/internetstandards/Internet.nl/issues/941) in the security.txt labels.
 - Fixed an [issue with the connection test and CSP form-action](https://github.com/internetstandards/Internet.nl/issues/945)
 
@@ -409,7 +413,7 @@ Bugfixes
 - Fix some minor typos and broken link [(#574)] [(#575)]
 - Add a missing ' in the frame-ancestors explanation [(#578)]
 - An empty part of Content Security Policy gives an error [(#583)]
-- Recursion error when stripping nonces in IPv4 and IPv6 comparison [(#587)] 
+- Recursion error when stripping nonces in IPv4 and IPv6 comparison [(#587)]
 - Remove certificate from the certificate chain in the shipped cert chain file [(#614)]
 
 Dependencies
@@ -716,19 +720,19 @@ Initial public release.
 --- Brief description for next version ---
 
 New
-- 
+-
 
 Changes
-- 
+-
 
 Bug Fixes
 -
 
 Dependencies
-- 
+-
 
 Migrations
-- 
+-
 
 Settings
-- 
+-
diff --git a/docker/batch-test.env b/docker/batch-test.env
@@ -38,9 +38,6 @@ IPV4_IP_TEST_TARGET_MAIL_PUBLIC=172.16.43.52
 IPV6_IP_TEST_TARGET_PUBLIC=fd00:43:1::51
 IPV6_IP_TEST_TARGET_MAIL_PUBLIC=fd00:43:1::52
 
-# use easy test/test user/passwords for authenticated endpoints
-MONITORING_AUTH_RAW='test:$apr1$PfpYZVWM$tLUKMXt91KJV6I.CF3TOt1,test_raw:$apr1$6YuDyduL$706z.FPTe5c09R767N3W90'
-
 LETSENCRYPT_STAGING=1
 LETSENCRYPT_EMAIL=letsencrypt@example.com
 
diff --git a/docker/compose.development.yaml b/docker/compose.development.yaml
@@ -26,6 +26,9 @@ services:
         # auto rebuild/reload when config files change
         - path: ./webserver/
           action: rebuild
+    volumes:
+      # mount monitoring credentials for testing/development
+      - ./webserver/dev.htpasswd:/etc/nginx/htpasswd/monitoring.htpasswd:ro
 
   app:
     develop:
diff --git a/docker/compose.integration-tests.yaml b/docker/compose.integration-tests.yaml
@@ -192,6 +192,9 @@ services:
       public-internet:
         ipv6_address: $IPV6_IP_PUBLIC
         ipv4_address: $IPV4_WEBSERVER_IP_PUBLIC
+    volumes:
+      # mount monitoring credentials for testing/development
+      - ./webserver/dev.htpasswd:/etc/nginx/htpasswd/monitoring.htpasswd:ro
 
   unbound:
     networks:
diff --git a/docker/compose.yaml b/docker/compose.yaml
@@ -31,7 +31,6 @@ services:
     environment:
       - INTERNETNL_DOMAINNAME
       - IPV6_TEST_ADDR
-      - MONITORING_AUTH_RAW
       - AUTH_ALL_URLS
       - ALLOW_LIST
       - ROUTINATOR_ALLOW_LIST
@@ -56,7 +55,11 @@ services:
     volumes:
       # persist certbot configuration between restarts
       - certbot-config:/etc/letsencrypt
-      - htpasswd-files:/etc/nginx/htpasswd/external
+      # include configured password for http basic auth (if enabled)
+      - $INTERNETNL_INSTALL_BASE/volumes/webserver/htpasswd:/etc/nginx/htpasswd
+      # mount old static password configuration for migration
+      - htpasswd-files:/etc/nginx/htpasswd-old
+      # share logs with logs exporter
       - nginx-logs-exporter:/var/log/nginx/prometheus-nginxlog-exporter/
 
     healthcheck:
diff --git a/docker/defaults.env b/docker/defaults.env
@@ -91,12 +91,6 @@ ALLOW_LIST=
 # comma separated of IP(v6) addresses/subnets that are allowed to access the /routinator endpoint (used for multi instance deployements
 ROUTINATOR_ALLOW_LIST=
 
-# comma separated user:htpasswd_encrypted pairs for /grafana and /prometheus, and side wide
-# password must already be encrypted
-# please not that the value needs to be enclosed by single quotes to prevent interpolation of the dollar signs
-# eg: MONITORING_AUTH_RAW='test1:$apr1$wGM8gxBe$DxGwifTGWZJ7nftK7LzFt/,user2:$apr1$BoZzsbb/$2NgfYCfF9lxmGrfSqsZKc/'
-MONITORING_AUTH_RAW=
-
 # Django debug mode, on test run without debug, same as production
 DEBUG=False
 
diff --git a/docker/deploy.sh b/docker/deploy.sh
@@ -11,6 +11,7 @@ cp -v /dist/docker/* docker
 # put $RELEASE into the compose.sh file
 envsubst '$RELEASE' < docker/compose-dist.sh > docker/compose.sh
 chmod a+x docker/compose.sh
+chmod a+x docker/user_manage.sh
 
 # set release version in local.env config
 echo "RELEASE='$RELEASE' # deploy $(date)" >> docker/local.env
diff --git a/docker/develop.env b/docker/develop.env
@@ -11,9 +11,6 @@ COMPOSE_PROJECT_NAME=internetnl-develop
 # enable for testing batch api
 ENABLE_BATCH=True
 
-# use easy test/test user/passwords for authenticated endpoints
-MONITORING_AUTH='test:$apr1$PfpYZVWM$tLUKMXt91KJV6I.CF3TOt1'
-
 LETSENCRYPT_STAGING=1
 LETSENCRYPT_EMAIL=letsencrypt@example.com
 
diff --git a/docker/test.env b/docker/test.env
@@ -37,9 +37,6 @@ IPV4_IP_TEST_TARGET_MAIL_PUBLIC=172.16.43.52
 IPV6_IP_TEST_TARGET_PUBLIC=fd00:43:1::51
 IPV6_IP_TEST_TARGET_MAIL_PUBLIC=fd00:43:1::52
 
-# use easy test/test user/passwords for authenticated endpoints
-MONITORING_AUTH_RAW='test:$apr1$PfpYZVWM$tLUKMXt91KJV6I.CF3TOt1,test_raw:$apr1$6YuDyduL$706z.FPTe5c09R767N3W90'
-
 LETSENCRYPT_STAGING=1
 LETSENCRYPT_EMAIL=letsencrypt@example.com
 
diff --git a/docker/user_manage.sh b/docker/user_manage.sh
@@ -1,4 +1,11 @@
 #!/usr/bin/env sh
+
 # Small wrapper around user mgmt script shipped in webserver image
 # For both convenience, and to have a suitable command to put in sudo
-/usr/bin/docker compose --env-file=docker/defaults.env --env-file=docker/host.env --env-file=docker/local.env exec -ti webserver /user_manage_inner.sh "$1" "$2"
+
+set -e # fail on error
+
+# determine install base (parent of directory containing this file)
+INTERNETNL_INSTALL_BASE=$(dirname "$(dirname "$(readlink -f "$0")")")
+
+"$INTERNETNL_INSTALL_BASE/docker/compose.sh" exec -ti webserver /user_manage_inner.sh "$1" "$2"
diff --git a/docker/webserver/authentication.sh b/docker/webserver/authentication.sh
@@ -1,10 +1,29 @@
 #!/bin/sh
-echo $MONITORING_AUTH_RAW|tr ',' '\n' >> /etc/nginx/htpasswd/monitoring.htpasswd
 
-# enable basic auth when user/password is configured
+# this script sets up nginx configuration files for user and IP authentication
+
+set -e # exit on error
+
+# migrate htpasswd file from old way to new storage location
+if test -f /etc/nginx/htpasswd/users.htpasswd; then
+  echo "Existing user password configuration found, not migrating old configuration."
+else
+  if ! test -f /etc/nginx/htpasswd-old/users.htpasswd; then
+    echo "No old user password configuration found, not migrating."
+  else
+    echo "Migrating old user password configuration"
+    cp /etc/nginx/htpasswd-old/users.htpasswd /etc/nginx/htpasswd/users.htpasswd
+  fi
+fi
+
+# create empty password files if they don't exist
 touch /etc/nginx/conf.d/basic_auth.include
+touch /etc/nginx/htpasswd/users.htpasswd
+touch /etc/nginx/htpasswd/monitoring.htpasswd
+
+# enable basic auth when user/password is configured
 if [ "$AUTH_ALL_URLS" != "False" ] || [ "$ENABLE_BATCH" != "False" ]; then
-  echo 'auth_basic "Please enter your access username and password";auth_basic_user_file /etc/nginx/htpasswd/external/users.htpasswd;' > /etc/nginx/conf.d/basic_auth.include
+  echo 'auth_basic "Please enter your access username and password";auth_basic_user_file /etc/nginx/htpasswd/users.htpasswd;' > /etc/nginx/conf.d/basic_auth.include
 fi
 
 # create IP allow list
diff --git a/docker/webserver/dev.htpasswd b/docker/webserver/dev.htpasswd
@@ -0,0 +1 @@
+test:$apr1$PfpYZVWM$tLUKMXt91KJV6I.CF3TOt1
diff --git a/docker/webserver/nginx_templates/default.conf.template b/docker/webserver/nginx_templates/default.conf.template
@@ -278,7 +278,7 @@ server {
     # batch API, requires authentication and passes basic auth user to Django App via headers
     location /api/batch/v2 {
         auth_basic "Please enter your batch username and password";
-        auth_basic_user_file /etc/nginx/htpasswd/external/users.htpasswd;
+        auth_basic_user_file /etc/nginx/htpasswd/users.htpasswd;
 
         # pass logged in user to Django
         proxy_set_header REMOTE-USER $remote_user;
diff --git a/docker/webserver/user_manage_inner.sh b/docker/webserver/user_manage_inner.sh
@@ -1,6 +1,6 @@
 #!/usr/bin/env sh
 
-HTPASSWD_FILE="/etc/nginx/htpasswd/external/users.htpasswd"
+HTPASSWD_FILE="/etc/nginx/htpasswd/users.htpasswd"
 
 if [ ! -f "$HTPASSWD_FILE" ]; then
   touch "$HTPASSWD_FILE"
diff --git a/documentation/Docker-deployment-batch.md b/documentation/Docker-deployment-batch.md
@@ -87,19 +87,13 @@ Batch installations require the following settings:
 
 - `ENABLE_BATCH`: Must be set to `True`, to enable batch API
 - `ENABLE_HOF`: Must be set to `False`, to disable Hall of Fame processing
-
-And optionally:
-
-- `MONITORING_AUTH_RAW`: May be a comma separated list of `user:hashed-password` pairs which are allowed to access the metrics at `https://example.com/grafana/`.
-- `AUTH_ALL_URLS` and `ALLOW_LIST`: Can be set to restrict access to the single scan webpage. See [Restricting Access](Docker-deployment.md#restricting-access) for more information.
+- `ALLOW_LIST` : Can be set to allow visitors to the 'frontpage' which allows single tests to be performed. This is normally not needed but can be useful for manually testing. Batch test result pages are always accessible.
 
 For example:
 
     cat >> docker/local.env <<EOF
     ENABLE_BATCH=True
     ENABLE_HOF=False
-    # user/password(s) for access to /grafana monitoring
-    MONITORING_AUTH_RAW=user:<htpasswd hash>
     # allowed IP's to visit web interface without password
     ALLOW_LIST=198.51.100.1,2001:db8:2::1
     EOF
diff --git a/documentation/Docker-deployment.md b/documentation/Docker-deployment.md
@@ -114,7 +114,7 @@ Spin up instance:
       ghcr.io/internetstandards/util:latest \
       /deploy.sh
 
-This command will take a long time (up to 30 minutes) due to RPKI data that needs to be synced initially. After that it should complete without an error, indicating the application stack is up and running healthy. You can already prepare continue with the DNS setup below in the meantime.
+This command will take a a few minutes. After that it should complete without an error, indicating the application stack is up and running healthy. You can already prepare continue with the DNS setup below in the meantime.
 
 ## DNS setup
 
@@ -130,6 +130,10 @@ After deployment is complete, all services are healthy and DNS is setup you can
 
 For more information see: [documentation/Docker-live-tests.md](Docker-live-tests.md)
 
+It might be that RPKI tests are not giving proper results yet. This might be because the RPKI database needs to be synced. This happens automatically and should be finished ~30 minutes after deployment.
+
+
+
 ## Compose command
 
 To reduce issues with different versions a Compose command is included in the installation and can be accessed using `/opt/Internet.nl/docker/compose.sh`. Use this command for everything where you would normaly use `docker compose` to manage the Compose project.
@@ -286,9 +290,7 @@ Besides the single scan webpage, the Internet.nl application also contains a Bat
 
 ## Metrics (grafana/prometheus)
 
-The default deployment includes a metrics collection system. It consists of a Prometheus metrics server with various exporters and a Grafana frontend. To view metrics and graphs visit: `https://example.com/grafana/`. Authentication is configured using the `MONITORING_AUTH_RAW` variable.
-
-Also see: [Metrics](Docker-metrics.md)
+The default deployment includes a metrics collection system. It consists of a Prometheus metrics server with various exporters and a Grafana frontend. To view metrics and graphs visit: `https://example.com/grafana/`. For authentication and other information see: [Metrics](Docker-metrics.md)
 
 ## Monitoring/alerting
 
@@ -376,8 +378,6 @@ To manage users, call the `/opt/Internet.nl/docker/user_manage.sh` script. This
 and a username. The operation can be `add_update` to add or update a user's password, `delete` to delete a user,
 and `verify` to verify a user's existence and password. Passwords are entered interactively.
 
-If you would like users on the host to manage batch users, set sudo access for this script.
-
 ### IP allow/deny lists
 
 Site wide IP(v6) allow lists can be configured by specifying the `ALLOW_LIST` variable. It should contain a comma separated list of IP(v6) addresses or subnets.
diff --git a/documentation/Docker-metrics.md b/documentation/Docker-metrics.md
@@ -2,9 +2,7 @@
 
 The Docker deployment includes a metrics collection system which is available on production as well as development/test environments. It consists of a Prometheus metrics server which scrapes metrics from various exporters. Grafana is provided as frontend to visualise metrics and create graphs/dashboards.
 
-To view metrics and graphs visit the `/grafana/` endpoint. Eg: `http://localhost:8080/grafana/` for development and `https://example.com/grafana/` for production. For development the user/password is set to `test/test`, for production users can be configured using the `MONITORING_AUTH_RAW` variable in `docker/local.env` (see `docker/defaults.env` for information).
-
-Metrics collection is defined in the `docker/compose.monitoring.yaml` file.
+To view metrics and graphs visit the `/grafana/` endpoint. Eg: `http://localhost:8080/grafana/` for development and `https://example.com/grafana/` for production. For development the user/password is set to `test/test`, for production users can be configured using the `/opt/Internet.nl/volumes/webserver/htpasswd/monitoring.htpasswd` file, see below for information.
 
 ## Overview
 
@@ -31,3 +29,17 @@ Dashboard can be manually created through the Grafana web interface and can be s
 ## Retention time
 
 By default metrics are retained for 5 years. This can be changed by settings the `PROMETHEUS_RETENTION_TIME` in the `docker/local.env` file. Optionally you can also set the `PROMETHEUS_RETENTION_SIZE` to limit the disk space used.
+
+## Monitoring user/allowlist management
+
+Monitoring endpoints are IP/password restricted. To allow users to visit these pages credentials or allowlists must be configured.
+
+Set `ALLOW_LIST` in `/opt/Internet.nl/docker/local.env` to allow certain IP's to access the monitoring without authentication, eg: `ALLOW_LIST=198.51.100.1,2001:db8:2::1`
+
+Add `user:passwordhash` entries to `/opt/Internet.nl/volumes/webserver/htpasswd/monitoring.htpasswd`. These should be Apache2 htpassword hashes. To add or update a user password run:
+
+    /opt/Internet.nl/docker/compose.sh exec webserver htpasswd /etc/nginx/htpasswd/monitoring.htpasswd username
+
+You will be prompted to enter the password.
+
+After updating `ALLOW_LIST` or changing users update the configuration by running: `/opt/Internet.nl/docker/compose.sh restart webserver`.
diff --git a/integration_tests/common/test_basic.py b/integration_tests/common/test_basic.py
@@ -114,27 +114,13 @@ def test_monitoring_auth(page, app_url, path):
     response = requests.get(app_url + path, verify=False)
     assert response.status_code == 401
 
-    # MONITORING_AUTH provided in docker/test.env
+    # credentials provided in docker/test.env
     auth = ("test", "test")
 
     response = requests.get(app_url + path, auth=auth, verify=False)
     response.raise_for_status()
 
 
-def test_monitoring_auth_raw(page, app_url):
-    """Monitoring endpoints must be behind basic auth."""
-    path = "/grafana"
-
-    response = requests.get(app_url + path, verify=False)
-    assert response.status_code == 401
-
-    # MONITORING_AUTH_RAW provided in docker/test.env
-    auth = ("test_raw", "test_raw")
-
-    response = requests.get(app_url + path, auth=auth, verify=False)
-    response.raise_for_status()
-
-
 def test_no_server_banner(page, app_url_subdomain):
     response = requests.get(app_url_subdomain, verify=False)
     assert response.headers["server"] == "nginx"
diff --git a/integration_tests/conftest.py b/integration_tests/conftest.py
diff --git a/pyproject.toml b/pyproject.toml
