Fix /etc/hosts/ modification

Author: JannisBush

README.md

@@ -24,10 +24,10 @@ Our modified version of the wptserve HTTP server implementation can be found in
 ## Quickstart
 In this section, we explain how to setup our custom WPT-HeaderTesting backend server and run some small tests with it.
 - Prequistes:  Working installation of [Docker](https://docs.docker.com/get-started/get-docker/) on Linux or macOS. Ports 80, 443, 5432, and 8888 have to be available on the host machine or these exposed ports in [docker-compose.yml](docker-compose.yml) have to be disabled or changed.
-- Installation: Run `docker compose up` this starts a database, configures the HTTP responses, and starts our modified WPT-HeaderTesting server (the initial startup takes a couple of minutes)
-- Verify setup: Run `docker compose exec header-testing-server bash -c "poetry run -C _hp pytest /app/_hp"`: All tests should pass and the output should look similar to: `11 passed in 23.73s`
+- Installation: Run `docker compose up` this starts a database, configures the HTTP responses, and starts our modified WPT-HeaderTesting server (the initial startup takes a couple of minutes; it is finished when the output/logs say `INFO - Starting https server on https://0.0.0.0:<PORT>`)
+- Verify setup: Run `docker compose exec header-testing-server bash -c "poetry run -C _hp pytest /app/_hp"`: All tests should pass and the output should look similar to: `11 passed in <time>s`
 - Manual checks:
-  - The server is now serving all the test pages and responses and should be available on the host (by default it binds to port 80 and 443). 
+  - The server is now serving all the test pages and responses and should be available on the host (by default it binds to port 80 and 443).
   - Visit: http://localhost/_hp/common/frame-script-csp.js in a browser
   - Run `curl -I http://localhost/_hp/common/empty.html` and `curl -I --insecure https://localhost/_hp/common/empty.html` (our dummy certificates are not valid, thus `--insecure` is required) on the host should return a response from `BaseHTTP/0.6 Python/3.11.5`
 - Demo run of the desktop linux test-runner in docker:
@@ -43,7 +43,7 @@ In the following, we explain how to use this software to collect security header
 ### WPT-HeaderTesting Server
 - Starting the server: Run `docker compose up -d` to start the server. Optionally modify the [docker-compose.yml](docker-compose.yml) for your needs, e.g., make the database available on the outside, disable the platform mode on non `linux/amd64` platforms for increased efficiency, or change ports.
 - (Optional) Adding responses:  modify [create_responses](_hp/hp/tools/create_responses.py) to add `basic` responses, modify [response_header_generation.py](_hp/hp/tools/response_header_generation.py) to add `parsing` responses
-- (Optional) Add new HTML tests: 
+- (Optional) Add new HTML tests:
   - If you want to add a test to an existing feature, you only need to open a test file in [_hp/tests/](_hp/tests/), add the testcode and add the test to `test_declarations`
   - If you want to add a new test feature: add a new test file in [_hp/tests/](_hp/tests/), add supporting files in [_hp/common/](_hp/common/), add the server logic in [_hp/server/responses.py](_hp/server/responses.py), add responses for this feature (see above `adding responses`)
 - (Optional) Change the `host` domain of the server:
@@ -102,15 +102,15 @@ The test runner on macOS is in general the same as the one on Ubuntu.
     - The `--ignore_certs` option is not working in Safari, thus either setup the WPT-HeaderTesting server with a valid certificate or trust the self-signed certificate in macOS Keychain.
     - The Safari version is bound to the OS version and the version has to be updated in [desktop_selenium.py](_hp/hp/tools/crawler/desktop_selenium.py), e.g., set it to `18.6` for macOS 17.6.
     - Selenium can only automate a single Safari instance at a time (this is a Safari restriction), thus one always needs to set `--num_browsers 1`
-    - Everytime a new test is loaded, the Safari instance is receiving focus on MacOS. We try to minimize the annoyance by moving the Safari window (almost)out of the visible screen. However, one might still accidentally click or send other events to the automated Safari window. If that happens, the automation pauses (this is another Safari restriction) and one has to click on `Continue automation`. We recommend to use testing devices that only run the tests and are not otherwise used. 
+    - Everytime a new test is loaded, the Safari instance is receiving focus on MacOS. We try to minimize the annoyance by moving the Safari window (almost)out of the visible screen. However, one might still accidentally click or send other events to the automated Safari window. If that happens, the automation pauses (this is another Safari restriction) and one has to click on `Continue automation`. We recommend to use testing devices that only run the tests and are not otherwise used.
   - Example run: `poetry run python desktop_selenium.py --resp_type debug --num_browsers 1`
 
 #### Android (Emulated)
 We provide a test runner that can be used with emulated Android devices to efficiently run the tests in parallel.
 - Prerequisites:
   - A powerful server to run the emulated android devices on, in particular hardware accelaration has to be available on the system. We tested it on Ubuntu 22.04, but other host operating systems should also work.
   - The WPT-HeaderTesting server has to be setup to be reachible from the emulators via public DNS and the server needs valid certificates. Note: it is recommended to configure your firewall such that it only allows local access.
-  - The machine where the emulators are started also needs to have the poetry project installed 
+  - The machine where the emulators are started also needs to have the poetry project installed.
   - Download the Android SDK Command-Line Tools (command line tools only) and set it up (see https://developer.android.com/tools/sdkmanager for full documentation):
     - Run `wget <link-from-downloads-page>` [Android Studio downloads page](https://developer.android.com/studio#command-line-tools-only)
     - Unzip it: `unzip cmdlinetools-linux-<version_latest>.zip`
@@ -153,7 +153,7 @@ In addition to the above browser runners that require Selenium, AndroidSDK, and
   - Popups needs to be allowed in the browser. For our iPad runs, we used the Chrome browser (currently still uses WebKit). On the iPad, open Chrome and go to `Open Settings` -> `Content-Settings` -> `Block Pop-Ups` and toggle it `off`.
 - Usage instructions:
   - On the WPT-HeaderTesting server:
-    - Adjust [create_generic_browser.py](_hp/hp/tools/crawler/create_generic_browser.py) to fit to the browser/os version info, you are using and then run `docker compose exec header-testing-server bash -c "cd _hp/hp/tools/crawler && poetry run python create_generic_browser.py"` and note down the `browser_id` printed. 
+    - Adjust [create_generic_browser.py](_hp/hp/tools/crawler/create_generic_browser.py) to fit to the browser/os version info, you are using and then run `docker compose exec header-testing-server bash -c "cd _hp/hp/tools/crawler && poetry run python create_generic_browser.py"` and note down the `browser_id` printed.
     - Run: `docker compose exec header-testing-server bash -c "cd _hp/hp/tools/crawler && poetry run python desktop_selenium.py --gen_page_runner --resp_type debug --max_urls_until_restart 10000 --gen_multiplier 1"` and note the returned URLs (they are also saved ina file in the `crawler` folder)
       - `--resp_type` specifies the response type that should be tested `debug, basic, parsing`
       - `--max_urls_until_restart` specifies how many URLs are opened on maximum on one top-level test page. If this is high you only need to visit a single URL in the browser, but there could be issues with state accumulation. If this is low, many URLs that you have to visit somehow are generated. Note that there are always at least two URLs to visit, as we do not mix tests on HTTP and HTTPS.
@@ -166,7 +166,7 @@ We note that a full reproduction of this work is a significant effort and refer
 
 The files [analysis_may_2024.ipynb](_hp/hp/tools/analysis/analysis_may_2024.ipynb) (original analysis) and [analysis_december_2024.ipynb](_hp/hp/tools/analysis/analysis_december_2024.ipynb) (updated with additional browser versions) contain the full analysis used in our paper, including the output of the analysis. They can be viewed directly on GitHub or a jupyter server can be started to view them in Jupyter Lab. Note that the clustering output uses Jupyter Widgets that cannot be saved fully.
 
-We also provide instructions to rerun the analysis scripts such that the clustering output can be seen and to verify that the output is correct. Note that re-executing the analysis scripts require a large amount of RAM available for the docker container (~60GB per script; they can be run indepedently) and take around 30m to execute. 
+We also provide instructions to rerun the analysis scripts such that the clustering output can be seen and to verify that the output is correct. Note that re-executing the analysis scripts require a large amount of RAM available for the docker container (~60GB per script; they can be run indepedently) and take around 30m to execute.
 - Download the database: `curl https://zenodo.org/records/16996059/files/http_header_original.dump\?download\=1 --output data/http_header_original.dump` [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.16996058.svg)](https://doi.org/10.5281/zenodo.16996058)
 - Import the database into your local postgres: `docker compose exec postgres psql -U header_user -d http_header_demo -c "CREATE DATABASE http_header_original;"` and `docker compose exec -T postgres pg_restore -U header_user -d http_header_original -v /tmp/data/http_header_original.dump`
 - Start the jupyter-lab: `docker compose exec header-testing-server bash -c "cd /app/_hp/hp/tools/analysis && poetry run jupyter-lab --allow-root --ip 0.0.0.0"` and access the URL printed on your local browser
@@ -202,7 +202,7 @@ If there are questions about our tools or paper, please either file an issue or
 
 ## Research Paper
 
-The paper is available at the [ACM Digital Library](https://doi.org/10.1145/3719027.3765119). 
+The paper is available at the [ACM Digital Library](https://doi.org/10.1145/3719027.3765119).
 You can cite our work with the following BibTeX entry:
 ```latex
 @inproceedings{rautenstrauch2025header,

entrypoint.sh

@@ -1,6 +1,9 @@
 #!/bin/bash
 set -e
 
+# Update /etc/hosts (include custom hosts)
+cat _hp/host-config.txt | sudo tee -a /etc/hosts
+
 echo "Waiting for Postgres at $DB_HOST:$DB_PORT..."
 until pg_isready -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME"; do
   sleep 2