Evaluation in WA differs from the official implementation

[hysyyds]

Why call the evaluator at every step? In the official implementation, it only evaluates at the final step.

https://github.com/ServiceNow/BrowserGym/blob/ec6b802cd655f2c6a84ebd66a22a4435d8147272/browsergym/webarena/src/browsergym/webarena/task.py#L185C9-L185C11

https://github.com/web-arena-x/webarena/blob/df352854eef255b007110948f6d4f539af039717/run.py#L330

[gasse]

Browsergym was originally designed without a stop action, which is why we call the evaluator at every step to detect if the task is done or not, from the environment side. It could be nice to have this (optional) stop action so that agents can run faster on webarena.

[hysyyds]

Browsergym was originally designed without a stop action, which is why we call the evaluator at every step to detect if the task is done or not, from the environment side. It could be nice to have this (optional) stop action so that agents can run faster on webarena.

I fully understand the idea of separating the environment implementation from specific task logic. However, there are two potential issues with this approach:

1. Evaluating at every step might lead to the model stopping prematurely, such as when it simply lands on the answer page, which could inadvertently leak labels.

2. Some evaluations in WebArena involve navigating away from the current page, which could result in losing unfinished operations on that page. The current implementation of page preservation in WebArena is ineffective (and I suspect it might be ineffective for most environments, as pages cannot be easily copied).

A possible solution would be to modify the evaluation implementation in the WebArena environment, ensuring that the actual evaluator is only invoked after the termination conditions are met.

[recursix]

We agree with this, in recent future we will provide variants where the agent is responsible for stopping.