Reproducing results in simulation

[edwardjjj]

Dear authors of the LFG paper,

We would like to include your work as a key baseline in our research and tried to reproduce the results you stated in the paper. However, after running the code you provided on openreview, we only got a success rate of 37%. Can you help us get a comparable result? We used the following launch script:

#!/bin/bash

# Scene IDs
scenes=(
    "4ok3usBNeis" "5cdEh9F2hJL" "6s7QHgap2fW" "7MXmsvcQjpJ" "a8BtkwhxdRV" "BAbdmeyTvMZ"
    "bCPU9suPUw9" "bxsVRursffK" "CrMo8WxCyVb" "cvZr5TUy5C5" "Dd4bFSTQ8gi" "DYehNKdT76V"
    "eF36g7L6Z9M" "GLAQ4DNUx5U" "h1zeeAwLh9Z" "HY1NcmCgn3n" "k1cupFYWXJ6" "LT9Jq6dN3Ea"
    "MHPLjHsuG27" "mL8ThkuaVTM" "mv2HUxq3B53" "Nfvxx8J5NCo" "p53SfW6mjZe" "q3zU7Yy5E5s"
    "q5QZSEeHe5g" "QaLdnwvtxbs" "qyAac8rV8Zk" "svBbv1Pavdk" "TEEsavR23oF" "VBzV5z6i1WS"
    "wcojb4TFT35" "XB4GS9ShBRE" "y9hTuugGdiq" "yr17PDCnDDW" "ziup5kvtCCR" "zt1RVoi7PcG"
)

# API keys and org IDs
# Please replace with your API keys and org ids, use multiple to avoid hitting rate limits
openai_keys=( "sk-***" "sk-***" "sk-***" )
openai_orgs=( "org-***" "org-***" "org-***" )

# Divide scenes among the runs
scenes_per_run=$(( ${#scenes[@]} / 5 ))

# Start runs
for i in {0..4}; do
    start_index=$(( i * scenes_per_run ))
    end_index=$(( (i+1) * scenes_per_run - 1 ))

    # Slice the array to get the scenes for this run
    run_scenes=(${scenes[@]:$start_index:$scenes_per_run})

    # Convert the array to a comma-separated string
    run_scenes=$(printf ",%s" "${run_scenes[@]}")
    run_scenes=${run_scenes:1} # remove the leading comma
    
    experiment_name="Eval_LFG_${i}"
    
    echo "Starting run $i with scenes: $run_scenes"

    # Set the GPU for this run
    export CUDA_VISIBLE_DEVICES=$(($i))

    # Select OpenAI key and org id for this run
    key_index=$(($i % 3)) # Alternates between 0, 1, 2
    openai_key=${openai_keys[$key_index]}
    openai_org=${openai_orgs[$key_index]}
    
    # Run the script with the appropriate scenes and experiment name
    nohup python3 projects/habitat_objectnav/eval_dataset_distributed.py --scene_ids="$run_scenes" --experiment_name="$experiment_name" --openai_key="$openai_key" --openai_org="$openai_org" --max_episodes_per_scene=10 --use_gt_semantics --use_language > "${experiment_name}.out" 2>&1 &
done