Update demo instructions

Author: JannisBush

README.md

@@ -9,22 +9,24 @@ Our modified version of the wptserve HTTP server implementation can be found in
 ## Setup
 - Create a fresh Ubuntu22 container/VM: `lxc launch ubuntu:22.04 <name>` and connect to it `lxc exec <name> bash`
   - Switch to the ubuntu user: `su - ubuntu`
-  - Clone this repository: `[email protected]:header-testing/header-testing.git`
+  - Clone this repository: `git clone git@github.com:header-testing/header-testing.git`
   - Run the setup file: `cd header-testing/_hp`, `./setup.bash` (reopen all terminals or run `source ~/.bashrc` afterwards)
-  - Configure DB settings in [config.json](_hp/config.json)
+  - Configure DB settings in [config.json](_hp/tools/config.json); Make sure to create a database with the correct name
   - Setup the database: `cd _hp/tools && poetry run python models.py`
   - Setup certs: either remove `.demo` from the files in `_hp/tools/certs/` to use self-signed certs or add your own certs here
 
 ## Run Instructions
 - Always start the WPT server first (from the top-most folder): `poetry run -C _hp python wpt serve --config _hp/wpt-config.json`
-- Create the basic and parsing responses: Run `cd _hp/tools && poetry run create_responses.py` (basic), run `cd analysis` and execute `response_header_generation.ipynb` to generate the parsing responses.
+- Create the basic and parsing responses: Run `cd _hp/tools && poetry run python create_responses.py` (basic), run `cd analysis` and execute `response_header_generation.ipynb` to generate the parsing responses.
 - Manually check if the server and the tests are working: Visit http://sub.headers.websec.saarland:80/_hp/tests/framing.sub.html
 - Automatic testrunners:
   - `cd _hp/tools/crawler`
-  - Android: `poetry run android_intent.py` (Additional config required)
-  - MacOS/Ubuntu: `poetry run desktop_selenium.py`
-  - iPadOS/iOS: `poetry run desktop_selenium.py ----gen_page_runner --page_runner_json urls.json --max_urls_until_restart 10000"`, then visit the URLs in that file manually
-- Analysis: Open `_hp/tools/analysis/main_analysis_desktop_basic+parsing.ipynb` (Also contains the mobile analysis)
+  - Android: `poetry run python android_intent.py` (Additional config required)
+  - MacOS/Ubuntu: `poetry run python desktop_selenium.py` (For a quick test run: `poetry run python desktop_selenium.py --debug_browsers --resp_type debug --ignore_certs`)
+  - iPadOS/iOS: `poetry run python desktop_selenium.py ----gen_page_runner --page_runner_json urls.json --max_urls_until_restart 10000"`, then visit the URLs in that file manually
+- Analysis:
+  - Run `cd _hp/tools/analysis && poetry run jupyter-lab`
+  - Open `_hp/tools/analysis/main_analysis_desktop_basic+parsing.ipynb` (Also contains the mobile analysis)
 
 ## Inventory
 - `_hp/`: All test and analysis code for the paper:

_hp/tools/analysis/utils.py

@@ -8,10 +8,10 @@
 
 @dataclass
 class Config:
-    DB_HOST: str="134.96.225.205"
-    DB_NAME: str="http_header"
-    DB_USER: str="http_header"
-    DB_PASSWORD: str="bde50f42d5851a9eb6bf4166e737cb1c"
+    DB_HOST: str="localhost"
+    DB_NAME: str="http_header_demo"
+    DB_USER: str=""
+    DB_PASSWORD: str=""
     DB_PORT: str="5432"
 
 def get_data(config: Config, select_query: str, quiet=False) -> pd.DataFrame:

_hp/tools/config.json

@@ -1,3 +1,3 @@
 {
-    "DB_URL": "postgresql+psycopg2://http_header:[email protected]:5432/http_header"
+    "DB_URL": "postgresql+psycopg2://localhost:5432/http_header_demo"
 }

_hp/tools/crawler/desktop_selenium.py

@@ -274,6 +274,8 @@ def unraisable_hook(unraisable):
                         help="Toggle on debugging for browser selection")
     parser.add_argument("--debug_input", action="store_true",
                         help="Toggle on debugging for input(Next) during the run.")
+    parser.add_argument("--ignore_certs", action="store_true",
+                        help="Ignore certificate errors (necessary for local/debug run)")
     parser.add_argument("--run_mode", choices=["run_all", "repeat"], default="run_all",
                         help="Specify the mode (default: run_all)")
     parser.add_argument("--num_browsers", default=60, type=int, help="How many browsers to start in parallel (max).")
@@ -371,13 +373,18 @@ def unraisable_hook(unraisable):
         all_args = []
         assert(len(config) == 1)
         browser_name, browser_version, binary_location, arguments, browser_id = config[0]
+        if args.ignore_certs:
+            arguments = arguments + ["--ignore-certificate-errors"]
         for url in urls:
             assert(int(re.findall("runner-(\d+)", url)[0]) == 1)
             url = url + f"?browser_id={browser_id}"
             all_args.append((log_path, browser_name, browser_version, binary_location, arguments, args.debug_input, [url], args.timeout_task, args.timeout_task-60))
     else:
         for scheme in ["http", "https"]:
             for browser_name, browser_version, binary_location, arguments, browser_id in config:
+                if args.ignore_certs:
+                    arguments = arguments + ["--ignore-certificate-errors"]
+
                 if args.run_mode == "run_all":
                     test_urls = get_tests(resp_type=args.resp_type, browser_id=browser_id, scheme=scheme, max_resps=args.max_resps, max_popups=args.max_popups)
                     page_timeout = TIMEOUT