WebArena Verified (#377)

* init commit for webarena verified

* upd Makefile

* adding the basic files

* update dependencies

* start adding integration with wa_verified

* upd readme

* use custom backend for webarena_verified

* pass the wa instance to the evaluator

* pass the wa instance to the evaluator

* cleanup evaluator

* remove custom webarena verified instance

* update requirements to latest wav code

* use simpler and cleaner wav eval

* enable tracing

* fix wav

* update to new webarena verified version

* update task name template to webarena_verified.templateID.taskID

* fix config

* fix csv file

* add webarena_verified backend

* fix wav tasks

* do not check reachable if url is todo

* fix tmp trace creation, update goal to prompt model to satisfy wav return format/

* create webarena_verified action space with special submit function to match the benchmark expected agent response format

* look for extra header file path in environment variable

* undo special action set for webarena_verified

* remove wav actions

* load extra context headers for webarena(+lite)

* update README

* update requirements

* update makefile and readme

* update readme

* update requirements

* update readme

* update test

* black formater

* upd makefile

* update to new webarena_verified dataset version

* small debug

* add massage of shopping_admin tasks

* assume all endpoints are running

* update to latest version before the public release

* update instructions to fetch latest version before the public release

* exponential backoff

* update README

* compare json with the one in the library

* update install instructions

* update makefile

* update pypi deployment with webarena-verified

* fix assets directory

* fix task id template

* remove task json file, use the one from the webarena-verified library. Update task template to include revision number

* remove metadata and create it dynamically

* do not hardcode revision number

* fix

* run black formater

* fix format?

* always create the metadata file

* version-bump-dev

* Remove git dependency and add ins to install from source

* version-bump-dev 0.14.3.dev3

* add webarena-verified package as a dependency

* version-bump-dev 0.14.3.dev4

* add webarena-verified in the dev requirements.txt

* update gitignore

---------

Co-authored-by: Nicolas Gontier <nicolas.gontier@servicenow.com>
Co-authored-by: Aman Jaiswal <amanjaiswal73892@gmail.com>

Author: NicolasAG

.github/workflows/pypi.yml

@@ -28,8 +28,13 @@ jobs:
 
       - name: Build a binary wheel and a source tarball (browsergym-webarena)
         run: python3 -m build browsergym/webarena/ --outdir dist/
+        
       - name: Build a binary wheel and a source tarball (browsergym-webarenalite)
-        run: python3 -m build browsergym/webarenalite/ --outdir dist/
+        run: python3 -m build browsergym/webarenalite/ --outdir dist/   
+        
+      - name: Build a binary wheel and a source tarball (browsergym-webarena-verified)
+        run: python3 -m build browsergym/webarena_verified/ --outdir dist
+        
       - name: Build a binary wheel and a source tarball (browsergym-visualwebarena)
         run: python3 -m build browsergym/visualwebarena/ --outdir dist/
 

.gitignore

@@ -155,3 +155,6 @@ bg_wl_data/
 miniwob-plusplus/
 
 uv.lock
+
+# webarena verified metadata (constructed automatically)
+browsergym/experiments/src/browsergym/experiments/benchmark/metadata/webarena_verified.csv

Makefile

@@ -38,12 +38,12 @@ clean-miniwob:
 
 help:
 	@echo "Available targets:"
-	@echo "  install          - Install project dependencies"
-	@echo "  setup-miniwob    - Setup MiniWoB++ dependencies"
-	@echo "  install-demo     - Install demo dependencies"
-	@echo "  demo             - Run demo agent"
-	@echo "  test-core        - Run core tests"
-	@echo "  clean-miniwob    - Remove MiniWoB++ directory"
-	@echo "  help             - Show this help message"
+	@echo "  install                 - Install project dependencies"
+	@echo "  setup-miniwob           - Setup MiniWoB++ dependencies"
+	@echo "  install-demo            - Install demo dependencies"
+	@echo "  demo                    - Run demo agent"
+	@echo "  test-core               - Run core tests"
+	@echo "  clean-miniwob           - Remove MiniWoB++ directory"
+	@echo "  help                    - Show this help message"
 
 .PHONY: install setup-miniwob install-demo demo test-core clean-miniwob help

README.md

@@ -39,6 +39,7 @@ _Example of a GPT4-V agent executing openended tasks (top row, chat interactive)
 BrowserGym includes the following benchmarks by default:
  - [MiniWoB](https://miniwob.farama.org/)
  - [WebArena](https://webarena.dev/)
+ - [WebArenaVerified](https://github.com/ServiceNow/platform-labs-webarena-verified)
  - [VisualWebArena](https://jykoh.com/vwa)
  - [WorkArena](https://github.com/ServiceNow/WorkArena)
  - [AssistantBench](https://github.com/oriyor/assistantbench)
@@ -55,6 +56,7 @@ pip install browsergym-experiments  # experiment utilities (agent, loop, benchma
 pip install browsergym-core  # core functionalities only (no benchmark, just the openended task)
 pip install browsergym-miniwob  # core + miniwob
 pip install browsergym-webarena  # core + webarena
+pip install browsergym-webarena-verified  # core + webarena_verified
 pip install browsergym-visualwebarena  # core + visualwebarena
 pip install browsergym-workarena  # core + workarena
 pip install browsergym-assistantbench  # core + assistantbench
@@ -69,6 +71,7 @@ playwright install chromium
 Finally, each benchmark comes with its own specific setup that requires to follow additional steps.
  - for MiniWoB++, see [miniwob/README.md](browsergym/miniwob/README.md)
  - for WebArena, see [webarena/README.md](browsergym/webarena/README.md)
+ - for WebArenaVerified, see [webarena_verified/README.md](browsergym/webarena_verified/README.md)
  - for VisualWebArena, see [visualwebarena/README.md](browsergym/visualwebarena/README.md)
  - for WorkArena, see [WorkArena](https://github.com/ServiceNow/WorkArena)
  - for AssistantBench, see [assistantbench/README.md](browsergym/assistantbench/README.md)

browsergym/assistantbench/requirements.txt

@@ -1,4 +1,4 @@
-browsergym-core==0.14.3.dev1
+browsergym-core==0.14.3.dev4
 datasets
 scipy
 numpy

browsergym/assistantbench/src/browsergym/assistantbench/evaluation/evaluate_utils/evaluate_strings.py

@@ -69,7 +69,7 @@ def _normalize_number(text: str) -> str:
 
 
 def _answer_to_bags(
-    answer: Union[str, List[str], Tuple[str, ...]]
+    answer: Union[str, List[str], Tuple[str, ...]],
 ) -> Tuple[List[str], List[Set[str]]]:
     if isinstance(answer, (list, tuple)):
         raw_spans = answer

browsergym/core/src/browsergym/core/__init__.py

@@ -1,4 +1,4 @@
-__version__ = "0.14.3.dev1"
+__version__ = "0.14.3.dev4"
 
 import playwright.sync_api
 

browsergym/experiments/requirements.txt

@@ -1,3 +1,3 @@
-browsergym-core==0.14.3.dev1
+browsergym-core==0.14.3.dev4
 tiktoken>=0.4
 dataclasses-json

browsergym/experiments/src/browsergym/experiments/benchmark/base.py

@@ -53,7 +53,13 @@ def make_action_set(self):
 
 
 BenchmarkBackend = Literal[
-    "miniwob", "webarena", "visualwebarena", "workarena", "assistantbench", "weblinx"
+    "miniwob",
+    "webarena",
+    "webarena_verified",
+    "visualwebarena",
+    "workarena",
+    "assistantbench",
+    "weblinx",
 ]
 
 

browsergym/experiments/src/browsergym/experiments/benchmark/configs.py

@@ -1,4 +1,5 @@
 import numpy as np
+
 from browsergym.experiments.benchmark.metadata.utils import (
     task_list_from_metadata,
     task_metadata,
@@ -132,6 +133,21 @@
         ),
         task_metadata=task_metadata("webarena"),
     ),
+    "webarena_verified": lambda n_repeats=1: Benchmark(
+        name="webarena_verified",
+        high_level_action_set_args=DEFAULT_HIGHLEVEL_ACTION_SET_ARGS["webarena"],
+        is_multi_tab=True,
+        supports_parallel_seeds=False,
+        backends=["webarena_verified"],
+        env_args_list=make_env_args_list_from_repeat_tasks(
+            task_list=task_list_from_metadata(metadata=task_metadata("webarena_verified")),
+            max_steps=30,
+            n_repeats=n_repeats,
+            seeds_rng=np.random.RandomState(42),
+        ),
+        task_metadata=task_metadata("webarena_verified"),
+    ),  # TODO: Add webarena-verified hard subsets by filtering tasks in
+    # https://github.com/ServiceNow/webarena-verified/blob/main/assets/dataset/subsets/webarena-verified-hard.json
     "webarena_lite": lambda n_repeats=1: Benchmark(
         name="webarena_lite",
         high_level_action_set_args=DEFAULT_HIGHLEVEL_ACTION_SET_ARGS["webarena"],
@@ -252,7 +268,8 @@
         backends=["assistantbench"],
         env_args_list=make_env_args_list_from_repeat_tasks(
             task_list=task_list_from_metadata(
-                metadata=task_metadata("assistantbench"), filter={"browsergym_split": "valid|test"}
+                metadata=task_metadata("assistantbench"),
+                filter={"browsergym_split": "valid|test"},
             ),
             max_steps=30,
             n_repeats=n_repeats,