Discrepancy in Inference Cost for LLaMA2-7B on GSM8K Dataset

[sofan110]

Problem Description

While reading the paper, I noticed that the average inference cost for 32 rollouts of the LLaMA2-7B model on the GSM8K dataset is reported as 166. However, in our actual tests, even with 8 rollouts, the average inference cost has already reached 600. This is a significant discrepancy compared to the results reported in the paper. Could you kindly provide guidance on whether there are any specific settings, configurations, or considerations that we might have overlooked?