Clarification on LLM Output Preprocessing in Planetarium Evaluation

[jsbyun121]

I am working on reproducing your Planetarium results and have discovered a potential issue in the publicly released code.

The evaluation pipeline (evaluate.py) passes LLM outputs directly to equivalence() without preprocessing. However, when I run fine-tuned Gemma 2 models via SGLang, the raw outputs cause Lark parsing failures and prevent me from reproducing the reported Gemma 2-2B and Gemma 2-9B results.

1. Does the reported evaluation apply any preprocessing before calling equivalence()?
2. Could the result differences be due to different prompting strategies or model configurations between your setup and the public repository?
3. Is there a specific commit/branch that should be used to reproduce the reported results?

Any clarification would help ensure reproducibility.

[maxzuo]

Hi,

Thanks for your interest in our work! What are the specific errors that you get?

1. In evaluate.py, we try to extract a PDDL code block if there exists one. We do so in a try/except block, so it shouldn't crash the code, so I'm wondering where the error is.
2. Different prompting strategies can definitely play a role, but I used what is in the public repository to run the results shown in the final paper.
3. The main branch should be enough.

[jsbyun121]

Thank you for responding!
Btw I used
...
instruction = "Provide me with the complete, valid problem PDDL file that describes the following planning problem directly without further explanations, texts.\n"
return f"{instruction}{natural_language}\nThe domain for the planning problem is:\n{domain_pddl}\n
...
which is same prompt as the opened code and did test for 2 cases, temperature as 0.0 and 1.0. Each result of the base model (Gemma2 2b, Gemma2 9b) was all far lower than the paper. I used same evaluate function... So I'm curious where is the point I missed or if there is non-existing thing in the open code.

[jsbyun121]

• I used hugging face test dataset

[maxzuo]

Hi,

Are you using the instruct models?
And can you try running the evaluate.py file yourself to see if you get a different result?

Without a reproducible code example or the exact issue, it's hard for me to pinpoint the problem. Thanks.

[jsbyun121]

I tried to reproduce the full process exactly as described.

1. After running the data generation script and starting the evaluation, I noticed that the generated PDDL questions number 27,949, whereas the Hugging Face test set only contains around 15.9k examples. Could you please clarify why there is such a difference in the number of PDDL questions between the generated data and the released test set?

2. I’d also like to run the evaluation using the Gemma2 model. However, I’m running into several version conflicts that make it difficult to reproduce on the same codebase. Could you please share the exact dependency versions (e.g., from your requirements.txt, environment.yml, or installed packages list) that you used when running with Gemma2?

That would be very helpful for ensuring full reproducibility.
Thank you so much for your time and support!

[maxzuo]

Hi, I'm not sure how you got a 27k test dataset. I've just re-run the dataset generation myself from a clean install, and it is producing 15,943 examples. Here is what I ran, exactly:

git clone https://github.com/BatsResearch/planetarium
cd planetarium
unzip dataset-config.zip
poetry install --with all
python dataset_generator.py -c dataset_config.yaml

This will create a datasets_v2.db and a splits.yaml. In datasets_v2.db, you can check and verify that there are 146110 rows in the database by running:

sqlite3 dataset_v2.db
> SELECT count(*) from problems;

In splits.yaml, it is a little messy, but the heldout split type has the problem IDs associated with the test dataset that is on Hugging Face.

To check the number of problems, a quick python script:

import yaml

with open('splits.yaml', 'r') as f:
  splits = yaml.safe_load(f)

total = 0
for domain_splits in splits.values():
  total += len(domain_splits['heldout']['test'])

print('Number of test samples:', total)

This can also be corroborated in the report.md file that should have been generated.

For the dependency versions that I used when running with Gemma 2, they are listed in the pyproject.toml. Running poetry install --with all should install everything you need, although there might be an issue with certain packages. Running pip install "vllm==0.5.0.post1" should hopefully get you the correct transformers and torch versions we used.

Make sure you are using a clean install on a fresh environment. By the way, did you run Gemma 2 or Gemma 2 IT?