Update README and similar

Author: JannisBush

README.md

@@ -12,21 +12,21 @@ Our modified version of the wptserve HTTP server implementation can be found in
   - Clone this repository: `git clone [email protected]:header-testing/header-testing.git`
   - Run the setup file: `cd header-testing/_hp`, `./setup.bash` (reopen all terminals or run `source ~/.bashrc` afterwards)
   - Configure DB settings in [config.json](_hp/tools/config.json); Make sure to create a database with the correct name
-  - Setup the database: `cd _hp/tools && poetry run python models.py`
-  - Setup certs: either remove `.demo` from the files in `_hp/tools/certs/` to use self-signed certs or add your own certs here
+  - Setup the database: `cd _hp/hp/tools && poetry run python models.py`
+  - Setup certs: either remove `.demo` from the files in `_hp/hp/tools/certs/` to use self-signed certs or add your own certs here
 
 ## Run Instructions
 - Always start the WPT server first (from the top-most folder): `poetry run -C _hp python wpt serve --config _hp/wpt-config.json`
-- Create the basic and parsing responses: Run `cd _hp/tools && poetry run python create_responses.py` (basic), run `cd analysis` and execute `response_header_generation.ipynb` to generate the parsing responses.
+- Create the basic and parsing responses: Run `cd _hp/hp/tools && poetry run python create_responses.py` (basic), run `cd analysis` and execute `response_header_generation.ipynb` to generate the parsing responses.
 - Manually check if the server and the tests are working: Visit http://sub.headers.websec.saarland:80/_hp/tests/framing.sub.html
 - Automatic testrunners:
-  - `cd _hp/tools/crawler`
+  - `cd _hp/hp/tools/crawler`
   - Android: `poetry run python android_intent.py` (Additional config required)
   - MacOS/Ubuntu: `poetry run python desktop_selenium.py` (For a quick test run: `poetry run python desktop_selenium.py --debug_browsers --resp_type debug --ignore_certs`)
   - iPadOS/iOS: `poetry run python desktop_selenium.py ----gen_page_runner --page_runner_json urls.json --max_urls_until_restart 10000"`, then visit the URLs in that file manually
 - Analysis:
-  - Run `cd _hp/tools/analysis && poetry run jupyter-lab`
-  - Open `_hp/tools/analysis/main_analysis_desktop_basic+parsing.ipynb` (Also contains the mobile analysis)
+  - Run `cd _hp/hp/tools/analysis && poetry run jupyter-lab`
+  - Open `_hp/hp/tools/analysis/main_analysis_desktop_basic+parsing.ipynb` (Also contains the mobile analysis)
 
 ## Inventory
 - `_hp/`: All test and analysis code for the paper:
@@ -36,7 +36,7 @@ Our modified version of the wptserve HTTP server implementation can be found in
     - `testharness.sub.js`: Modified testharness.js to store the recorded output additonally to the test status
   - `server/`: Custom server endpoints to save data in our database and serve the correct responses from the DB
   - `tests/`: The template pages containing the test functions for the 12 tested features
-  - `tools/`: All other code
+  - `hp/tools/`: All other code
     - `analysis/`: Analysis code (.ipynb files) + utils
     - `certs/`: Put your certs here to enable testing of HTTPS
     - `crawler/`:  Intent (Android), Selenium (Mac + Ubuntu), and Browser Page Runner (iOS) test runners + utils

_hp/hp/tools/analysis/main_analysis_desktop_basic+parsing-new_chrome_data.ipynb

@@ -42,7 +42,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 11,
    "id": "a5639063-07f2-4283-ba32-098aa5b57671",
    "metadata": {},
    "outputs": [
@@ -74,7 +74,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 12,
    "id": "d2e1f6ac-ccd6-4115-a9a1-83b245225bac",
    "metadata": {},
    "outputs": [],
@@ -91,7 +91,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 13,
    "id": "e2646a25-3184-459f-854b-0a3c5c2378ee",
    "metadata": {},
    "outputs": [
@@ -122,7 +122,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 14,
    "id": "589725d1-b403-4c0c-9fb1-219560b382c3",
    "metadata": {},
    "outputs": [],
@@ -141,7 +141,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 15,
    "id": "16c48765-0460-427a-b2cf-7e084bbd4788",
    "metadata": {},
    "outputs": [],
@@ -152,7 +152,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 16,
    "id": "c921e555-b71b-41e6-89c9-73d6df846c37",
    "metadata": {},
    "outputs": [
@@ -195,18 +195,18 @@
        "      <td>128</td>\n",
        "      <td>Ubuntu 22.04</td>\n",
        "      <td>headless-new</td>\n",
-       "      <td>3152</td>\n",
+       "      <td>4728</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "text/plain": [
        "  automation_mode browser_id    name version            os headless_mode    id\n",
-       "0        selenium          3  chrome     128  Ubuntu 22.04  headless-new  3152"
+       "0        selenium          3  chrome     128  Ubuntu 22.04  headless-new  4728"
       ]
      },
-     "execution_count": 9,
+     "execution_count": 16,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -219,17 +219,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 9,
    "id": "9038e1fe-d3f5-4c48-8b36-5a85b8495968",
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "3152"
+       "4728"
       ]
      },
-     "execution_count": 10,
+     "execution_count": 9,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -240,7 +240,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 17,
    "id": "cd18c0fe-fa04-4348-af05-34755e07b4df",
    "metadata": {},
    "outputs": [
@@ -277,7 +277,7 @@
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>chrome Ubuntu 22.04 128 selenium headless-new</th>\n",
-       "      <td>3152</td>\n",
+       "      <td>4728</td>\n",
        "      <td>54</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
@@ -287,10 +287,10 @@
       "text/plain": [
        "                                               count  nunique\n",
        "browser                                                      \n",
-       "chrome Ubuntu 22.04 128 selenium headless-new   3152       54"
+       "chrome Ubuntu 22.04 128 selenium headless-new   4728       54"
       ]
      },
-     "execution_count": 11,
+     "execution_count": 17,
      "metadata": {},
      "output_type": "execute_result"
     }

_hp/hp/tools/crawler/desktop_selenium.py

@@ -295,7 +295,7 @@ def unraisable_hook(unraisable):
         config = [
             # Released 2024-01-22 (17.3 (19617.2.4.11.8))
             # ("safari", "17.3.1", None, None, get_or_create_browser("safari", "17.3.1", "macOS 14.3.1", "real", "selenium", "")),
-            ("safari", "17.5", None, None, get_or_create_browser("safari", "17.5", "macOS 14.5", "real", "selenium", "")),
+            ("safari", "17.5", None, [], get_or_create_browser("safari", "17.5", "macOS 14.5", "real", "selenium", "")),
 
             # Brave without updates on MacOS
             # Download .dmg from https://github.com/brave/brave-browser/releases and install