server: Add "tokens per second" information in the backend

[lhpqaq]

Implement #10502

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[ngxson]

The idea is good but I'm not confident about the UI/UX part:

1. Not all users want this, so it must be hidden by default (for a clean UI) and user can activate it via Settings menu

2. The text takes up quite a lot of space, I would prefer to make it more subtle. Take jan.ai app as an example:
   

3. For the code, we can calculate these numbers in real-time, on the frontend. This provides a better UX and allow to show t/s speed in real time.

[lhpqaq]

@ngxson Thank you for your suggestion. I’m also not very confident about the UI/UX part.
I have restored the frontend section and added a real-time speed field in the backend. I hope it proves useful.

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":" assist"}}],"created":1732762062,"id":"chatcmpl-evuFEOOBA3wfqsmOq1Yvxz9S6X37BdLu","model":"gpt-3.5-turbo-0613","object":"chat.completion.chunk","gen_second":29.860551225775627}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":" you"}}],"created":1732762062,"id":"chatcmpl-evuFEOOBA3wfqsmOq1Yvxz9S6X37BdLu","model":"gpt-3.5-turbo-0613","object":"chat.completion.chunk","gen_second":29.295321955588292}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":" today"}}],"created":1732762062,"id":"chatcmpl-evuFEOOBA3wfqsmOq1Yvxz9S6X37BdLu","model":"gpt-3.5-turbo-0613","object":"chat.completion.chunk","gen_second":28.80692518481443}

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"?"}}],"created":1732762062,"id":"chatcmpl-evuFEOOBA3wfqsmOq1Yvxz9S6X37BdLu","model":"gpt-3.5-turbo-0613","object":"chat.completion.chunk","gen_second":28.234142607517185}

data: {"choices":[{"finish_reason":"stop","index":0,"delta":{}}],"created":1732762062,"id":"chatcmpl-evuFEOOBA3wfqsmOq1Yvxz9S6X37BdLu","model":"gpt-3.5-turbo-0613","object":"chat.completion.chunk","usage":{"completion_tokens":10,"prompt_tokens":1561,"total_tokens":1571,"gen_second":28.003281984648602,"prompt_second":479.472377660826}}

data: [DONE]

[ngxson]

I haven't had time to look deeper into this, but seems like what you're doing is already handled by get_formated_timings(). Can you have a look if it's a duplication?

[lhpqaq]

I haven't had time to look deeper into this, but seems like what you're doing is already handled by get_formated_timings(). Can you have a look if it's a duplication?

Yes, but get_formated_timings() only calculates the final result and lacks real-time speed during the prediction process.

[ngxson]

It doesn't get the correct value because slot.t_token_generation is not set during generation. You can simply set it.

What I'm thinking is:

• This feature should not enabled by default because it can potentially impact overall performance and network bandwidth. You should only return per-token timing if user ask for it (i.e. if user set "timing_per_token": true in the request)
• It's better and more intuitive to reuse "timings" object, which is provided by get_formated_timings(), so developers don't have to re-write their code or to memorize the difference between n_gen_second vs timings
• Also remember to update the documentation

[ngxson]

This code can be simplified further.

To pass the CI, you need to merge with latest upstream master branch

[lhpqaq]

@ngxson Thanks~