question about the evaluation of alfworld

[sunnychenxiwang]

When I evaluated Alfworld using the vanilla React method, I found that the success rate of all models was 1. Moreover, even when the final trajectory did not actually complete the task, the environment feedback returned:

<tool_call_result>Nothing happens.

Congratulations! You have completed the task!
</tool_call_result>

This caused the final state to be marked as success. How can I resolve this issue?

[sunnynexus]

Since each task in ALFWorld independently stores its state, this issue is likely caused by state conflicts between different tasks—i.e., when one task completes successfully, it incorrectly marks other tasks as completed as well.

You can check your ReAct implementation to ensure that, during batch generation, each task is correctly assigned its own unique env_id.