Perplexity implementation

[elements72]

Hi! I was looking to the implementation of the perplexity and I had a couple of doubts.
The first is that the actual implementation of the window ppl is slightly different from the one on HF. On HF code they took out the average from the loss and made the average at the end over all the tokens while in Axolotl it is an average of losses.
The second one is about performance, from what I have understood in Axolotl you compute the ppl over all the samples of the eval_dataset at one time. Would not lead to OOM if the eval set is composed of several samples?

[NanoCode012]

Thanks for catching those issues. Do you think you would be open to a PR to fix those two issues?