WebArena Shopping evaluator issues

[imenelydiaker]

There is an issue with some WebArena shopping tasks:

• On task 275: it's a search task where the agent is asked to search for "xbox". So the reference URL is __SHOPPING__/catalogsearch/result/?q=xbox. The agent (GenericAgent) gets to that URL correctly but is rewarded 0.
• Same thing for task 274 and probably other tasks.

@xhluca do you have other task IDs with the same failure?

[xhluca]

those are the two i observed but i imagine it's worth looking at various cases with shoping

[ZLZLGe]

There is an issue with some WebArena shopping tasks:

• On task 275: it's a search task where the agent is asked to search for "xbox". So the reference URL is __SHOPPING__/catalogsearch/result/?q=xbox. The agent (GenericAgent) gets to that URL correctly but is rewarded 0.
• Same thing for task 274 and probably other tasks.

@xhluca do you have other task IDs with the same failure?

I also encountered this issue, and it needs to be fixed by modifying the environment variable.
Change
export WA_SHOPPING="$BASE_URL:8082/"
to
export WA_SHOPPING="$BASE_URL:8082"

In task 274, the failure occurred because if WA_SHOPPING is not modified, the reference URL will be set to
http://localhost:8082//catalogsearch/result/?q=usb+wifi,
while the page.url obtained from the website is
http://localhost:8082/catalogsearch/result/?q=usb+wifi.
As you can see, there is an extra “/”, so we need to update WA_SHOPPING to fix this mismatch.
@xhluca