|
| 1 | +# Reproduction Instructions |
| 2 | + |
| 3 | +The following presents the commands that we used to collect the data presented in the paper and some notes for the browser runs. |
| 4 | +Please first read the full [README](README.md), before using this file. |
| 5 | +Note that for using the commands in this file, all browser runners on all devices need to be able to reach the WPT-HeaderTesting server under the hostnames specified in [wpt-config.json](_hp/wpt-config.json) and [host-config.txt](_hp/host-config.txt). Additionally, the server needs to have valid certificates for these hosts or the clients have to be configured to ignore certificate errors or trust the self-signed certificate: `--ignore_certs` for Linux Ubuntu, for the other operating systems the certificates have to be trusted on the OS level. |
| 6 | + |
| 7 | +## Desktop Browsers (Linux Ubuntu) |
| 8 | +- Originally run on an `lxc` container with Ubuntu 22.04 |
| 9 | +- All commands are run from the `_hp/hp/tools/crawler` directory |
| 10 | +- The [pyproject.toml](_hp/pyproject.toml) has to be installed with poetry |
| 11 | +- The server requires a (virtual) display: we used `Xvfb :99 -screen 0 1920x1080x24 &`, `x11vnc -display :99 -bg -shared -forever -passwd abc -xkb -rfbport 5900`, `export DISPLAY=:99 && fluxbox -log fluxbox.log &` |
| 12 | +- We used a powerful container (100 vCPUs, 256GB RAM) to execute 50 browsers in parallel. If the container is less powerful, decrease the `--num_browsers` parameter. The timing estimates assume `50` browsers are used in parallel and that the setup is complete. |
| 13 | +- The Brave browser is not automatically installed by Selenium, follow the installation instructions in [desktop_selenium.py](_hp/hp/tools/crawler/desktop_selenium.py). |
| 14 | +- Commands to reproduce the `basic` responses data (20 person-minutes, 30 compute-minutes): |
| 15 | + - Run `for i in {1..5}; do poetry run python desktop_selenium.py --num_browsers 50 --resp_type basic; done`, should take around 13m for all 5 iterations |
| 16 | + - It can happen that some tests do not have 5 results after the above commands due to timeouts and similar, to ensure that all tests have at least 5 results run the below commands: |
| 17 | + - Create a `repeat.json` file with all tests that have to be rerun: `poetry run python create_repeat.py --selection_str "\"Response\".resp_type = 'basic' and \"Browser\".os = 'Ubuntu 22.04'"` |
| 18 | + - Run `poetry run python desktop_selenium.py --num_browsers 50 --run_mode repeat --max_urls_until_restart 50` to reexecute these tests |
| 19 | + - It can happen that some tests are still missing results. Run the `create_repeat.py` command again to verify and potentially run the `--run_mode repeat` command until all tests have 5 responses |
| 20 | + - The above commands collect data for the original set of browsers. To collect the data for the browsers added in December, run the above `desktop_selenium.py` commands again with the additional flag `--new_browsers` (`create_repeat.py` does not require this flag) |
| 21 | +- Commands to reproduce the `parsing` responses data (45 person-minutes, 6 computer-hours) |
| 22 | + - Run `for i in {1..5}; do poetry run python desktop_selenium.py --num_browsers 50 --resp_type parsing; done`, should take around 5h for all 5 iterations |
| 23 | + - It can happen that some tests do not have 5 results after the above commands due to timeouts and similar, to ensure that all tests have at least 5 results run the below commands: |
| 24 | + - Create a `repeat.json` file with all tests that have to be rerun: `poetry run python create_repeat.py --selection_str "\"Response\".resp_type = 'parsing' and \"Browser\".os = 'Ubuntu 22.04'"` |
| 25 | + - Run `poetry run python desktop_selenium.py --num_browsers 50 --run_mode repeat --max_urls_until_restart 50` to reexecute these tests |
| 26 | + - It can happen that some tests are still missing results. Run the `create_repeat.py` command again to verify and potentially run the `--run_mode repeat` command until all tests have 5 responses |
| 27 | + - The above commands collect data for the original set of browsers. To collect the data for the browsers added in December, run the above `desktop_selenium.py` commands again with the additional flag `--new_browsers` (`create_repeat.py` does not require this flag) |
| 28 | + |
| 29 | +## Desktop Browsers (macOS) |
| 30 | +- For the original run, a macOS device with macOS 14.5 is required, for the updated browser run a macOS device with macOS 15.2 is required and that device requires a display. |
| 31 | +- All commands are run from the `_hp/hp/tools/crawler` directory |
| 32 | +- The [pyproject.toml](_hp/pyproject.toml) has to be installed with poetry |
| 33 | +- Make sure that the macOS device can reach the WPT-HeaderTesting server. |
| 34 | +- Make sure to configure you device, to not go to sleep/power-safe mode or similar. |
| 35 | +- To be able to use Selenium with Safari, one needs to activate remote automation. In Safari: develop -> developer settings -> developer settings -> allow remote automation. |
| 36 | +- The macOS device is not usable while running the tests as each time a new popup is opened the Safari window receives auto-focus from macOS. |
| 37 | +- For the experiment in the paper, we used a mixture of the generic browser runner (`--gen_page_runner`) and the desktop runner. |
| 38 | +- Preparation, has to be executed on the WPT-HeaderTesting server (e.g., prepend the commands with `docker compose exec header-testing-server bash -c "<command>"`): |
| 39 | + - Create generic test-page-runner pages for basic tests: `poetry run python desktop_selenium.py --resp_type basic --gen_page_runner --max_urls_until_restart 100` |
| 40 | + - Create generic test-page-runner pages for parsing tests: `poetry run python desktop_selenium.py --resp_type parsing --gen_page_runner --max_urls_until_restart 1000` |
| 41 | + - The above two commands output a path similar to `basic-MaxURLs100-MaxResps10-MaxPopups100-53332b.json`, make sure to copy these json files to the macOS device and replace the file name in the following commands. |
| 42 | +- Execute the runs: |
| 43 | + - Run the basic tests: `for i in {1..5}; do poetry run python desktop_selenium.py --num_browsers 1 --page_runner_json <basic-test-json> --timeout_task 1000; done`, this takes around 1 hour |
| 44 | + - Run the parsing tests: `for i in {1..5}; do poetry run python desktop_selenium.py --num_browsers 1 --page_runner_json <parsing-test-json> --timeout_task 10000; done`, this takes around 1 week |
| 45 | + - Add `--new_browsers` if running on macOS 15.2 |
| 46 | +- Ensure that all tests have 5 results: |
| 47 | + - For the basic tests: |
| 48 | + - Excute on the WPT-HeaderTesting server: `poetry run python create_repeat.py --selection_str "\"Response\".resp_type = 'basic' and \"Browser\".os != 'Android 11'"`, the output is a `repeat.json` file that has to be copied to the macOS device. (This command requires that the Desktop Linux browsers data collection is finished.) |
| 49 | + - Execute on the macOS device: `poetry run python desktop_selenium.py --num_browsers 1 --run_mode repeat --timeout_task 10000` |
| 50 | + - For the parsing tests: |
| 51 | + - Excute on the WPT-HeaderTesting server: `poetry run python create_repeat.py --selection_str "\"Response\".resp_type = 'parsing' and \"Browser\".os != 'Android 11'"`, the output is a `repeat.json` file that has to be copied to the macOS device. (This command requires that the Desktop Linux browsers data collection is finished.) |
| 52 | + - Execute on the macOS device: `poetry run python desktop_selenium.py --num_browsers 1 --run_mode repeat --timeout_task 10000` |
| 53 | + |
| 54 | +## Mobile Browsers (Android) |
| 55 | +- Originally run on a Debian 12.7 server. The machine has to be x86_64 and hardware acceleration has to be available. |
| 56 | +- We created 30 Android emulators and installed and configured the browsers as explained in the [README](README.md#android-emulated). |
| 57 | +- All commands are run from the `_hp/hp/tools/crawler` directory |
| 58 | +- The [pyproject.toml](_hp/pyproject.toml) has to be installed with poetry |
| 59 | +- Note that the timeouts are 2x for Android (due to speed issues in the emulator) and thus the tests execute slower |
| 60 | +- Commands used for collection the data: |
| 61 | + - Run the basic tests: `for i in {1..5}; do timeout 15m poetry run python android_intent.py -browsers chrome -repeat 1 -num_devices 30 -type basic -auto_restart; done`, should take around 30m |
| 62 | + - Run the parsing tests: `for i in {1..5}; do timeout 6h poetry run python android_intent.py -browsers chrome -repeat 1 -num_devices 30 -type parsing -auto_restart; done`, should take around 20h |
| 63 | + - Similarly to the other tests, it could happen that not all tests collected 5 results, thus run the following to rerun these tests: |
| 64 | + - Create the repeat file for the basic tests: `poetry run python create_repeat.py --selection_str "\"Response\".resp_type = 'basic' and \"Browser\".os = 'Android 11'"` |
| 65 | + - Run them: `poetry run python android_intent.py -browsers all -repeat 1 -num_devices 30 -url_json repeat.json -auto_restart` |
| 66 | + - Create the repeat file for the parsing tests: `poetry run python create_repeat.py --selection_str "\"Response\".resp_type = 'parsing' and \"Browser\".os = 'Android 11'"` |
| 67 | + - Run them: `poetry run python android_intent.py -browsers all -repeat 1 -num_devices 30 -url_json repeat.json -auto_restart` |
| 68 | + - Repeat these steps if some tests still do not have 5 results in a browser |
| 69 | + |
| 70 | + |
| 71 | +## Mobile Browsers (iPadOS) |
| 72 | +- The original run used two iPad Air(4th generation) devices with iPadOS 17.3.1 |
| 73 | +- On the iPad we installed Chrome 122 (uses WebKit) and allowed popups (Open Settings -> Content-Settings -> Block Pop-Ups -> Off) |
| 74 | +- Preparation on the WTP-HeaderTesting server (prepend the command with `sudo docker compose exec header-testing-server bash -c "cd _hp/hp/tools/crawler && <command>"`): |
| 75 | + - Run `poetry run python create_generic_browser.py` and note the returned browser_id |
| 76 | + - Generate URLs to visit: |
| 77 | + - Basic: `poetry run python desktop_selenium.py --resp_type basic --gen_page_runner --max_urls_until_restart 10000 --gen_multiplier 5`, the output is a list of URLs to visit (there should be two) |
| 78 | + - Parsing: `poetry run python desktop_selenium.py --resp_type parsing --gen_page_runner --max_urls_until_restart 100000 --gen_multiplier 5`, the output is a list of URLs to visit (there should be two) |
| 79 | +- On the iPad: |
| 80 | + - Manually visit the two URLs generated by the above commands and add `?browser_id=<browser_id>` to the URL, example: `https://sub.headers.websec.saarland/_hp/tests/test-page-runner-1_ed4f3b-0.html?browser_id=16` |
| 81 | + - If only one iPad is use, visit the URLs one after the other. The basic tests should take around 1h per URL. The parsing tests take several days. |
| 82 | + - Note: it could be that the browser get stuck, in that case reload the browser or use the below commands to generate only the tests that are still missing, depending on when it got stuck. |
| 83 | +- To ensure that all tests have at least 5 results run the following: |
| 84 | + - On the WPT-HeaderTesting: |
| 85 | + - First generate the usual `repeat.json` file: `poetry run python create_repeat.py --selection_str "\"Response\".resp_type = 'parsing' and \"Browser\".os != 'Android 11'"` (Note: this assumes that either the Desktop Linux or Desktop macOS browsers already collected data for all tests) |
| 86 | + - Then create a single page-runner URL containing all URLs from the `repeat.json` file: `poetry run python create_page_runner_repeats.py --browser_id <browser_id>` |
| 87 | + - On the iPad: |
| 88 | + - Visit the page-runner URL |
0 commit comments