tools: update llama-bench to include TTFT, E2E, ITL metrics

[taronaeo]

Introduces the Time to First Token (TTFT), End-to-End Latency (E2E), and Inter-token Latency (ITL) metrics. Updates the README.md to explain the calculation as well.

[taronaeo]

Hi @ggerganov @slaren, any interest in having these metrics in llama-bench? :)

[slaren]

I am not convinced that this is necessary. It doesn't really fit all that well into the llama-bench model and will only produce meaningful results with some types of tests.

OTOH, you can already calculate all of these values if you formulate the tests properly, e.g. TTFT can be estimated with -pg <n_prompt>,1, E2E with any -pg test, and ITL with -n.

[slaren]

It may be more appropriate to add a python or shell script that runsllama-bench with the right tests, and calculates these values.

[taronaeo]

It may be more appropriate to add a python or shell script that runs llama-bench with the right tests, and calculates these values.

Hmm yeah thats more reasonable. I can move the metrics into a Python script but I would prefer that we at least still log the TTFT within llama-bench so that we don't have to do 2 benchmark runs (i.e., -pg 512,1 and -pg 512,128) to generate the metrics.

This is because at least on IBM Z & LinuxONE, most of our users are running Type-2 virtualisation and the benchmark runs can have data varying far apart from one another, invalidating the results from the first run for TTFT as now they're inaccurate.

[taronaeo]

Let me know if its okay to keep the samples_ttft_ns/samples_ttft_ms within llama-bench to avoid the data inaccuracy I mentioned above

[slaren]

Let me know if its okay to keep the samples_ttft_ns/samples_ttft_ms within llama-bench to avoid the data inaccuracy I mentioned above

This still seems too specific, however, having an option to export in the json formats the timing of every token generated could be useful. It would also allow other use cases, such as generating very detailed graphs of performance vs context depth.