How to get or generate a file of log of probabilities of base model to measure KL divergence for perplexity?

[beebopkim]

I read some threads here mentioning KL divergence is more appropriate than perplexity, so I decided to do it by my hand. But when I tried to run perplexity with --kl-divergence option, I faced below error message.

kl_divergence: you must provide a name of a file containing the log probabilities of the base model

I have no knowledge of KL divergence. Please teach me how to get or generate the file with probabilities. What I tried to do is to measure KL divergence for various quant types of llama.cpp on meta-llama/Llama-2-7b-chat-hf model.

[beebopkim]

Self answer: #5076