Questions about the scores_all_data.pkl file and the evaluation process

[JizhanFang]

Hello, thank you very much for your work. I have two questions that I would like to kindly ask for clarification:

1. Regarding the scores_all_data.pkl file
   May I ask whether this file is directly taken from the original Mind2Web release? As far as I remember, this file was produced by Mind2Web using a tuned DeBERTa-v3-base model for candidate generation.
   If so, what impact might directly using the outputs of this model have on the results reported under add scores?
   In particular, in memory.py there is the following operation:

   pos_candidates = [
       c for c in s["pos_candidates"] if c["rank"] < args.top_k_elements
   ]

If the predicted ranks from this model are inaccurate, this filtering step could potentially result in an empty pos_candidates list.

2. Regarding the evaluation setup
   In memory.py, the following code is used:

   if len(pos_candidates) == 0:
       element_acc.append(0)
       action_f1.append(0)
       step_success.append(0)
       prev_obs.append("Observation: `" + target_obs + "`")
       prev_actions.append("Action: `" + target_act + "` (" + act_repr + ")")
       conversation.append("The ground truth element is not in cleaned html")
       continue

   Here, pos_candidates can be empty either because it is already empty in the original dataset, or because all candidates are filtered out due to c["rank"] >= args.top_k_elements.
   Since neither of these cases is caused by the model or method being evaluated, would counting such steps directly as failures potentially lead to an underestimation of the actual performance?

Thank you very much for your time and help. I would greatly appreciate any clarification on these points.

[zorazrw]

Hi! Yes, the scores_all_data.pkl file is directly taken from the Mind2Web release. For top-5 recall, I vaguely remember the chance of covering positive elements to be around 50%, so there's a pretty good chance that pos_candidates could be empty.
We decided to skip these examples because we found adding more negative elements would mostly degrade model performance, so there's barely any chance that the model can solve this example correctly.
We haven't systematically tested if changing the threshold (args.top_k_elements) would change the overall performances significantly, but my intuition is that because element prediction accuracy is so low they can hardly drive up the later performance. But you can definitely try it if interested!

[JizhanFang]

Thank you very much for your reply. I have a further question: from the README, it appears that only test_task is currently supported for testing. I was wondering whether there is a recommended way to conduct evaluations on test_website or test_domain.