Feature Request: Add "tokens per second" information in the Web UI

[ggerganov]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

The client should display prompt processing and text generations speeds.

Motivation

I helps to investigate how different parameters affect the performance

Possible Implementation

No response

[david68cu]

Hi Maintainers,

I want to take ownership of this issue and work on resolving it. I’ve reviewed the CONTRIBUTING.md file and familiarized myself with the project structure and guidelines.

If there are any specific details or considerations I should keep in mind while working on this, please let me know. Otherwise, I’ll take the necessary steps to address this issue effectively.

Could you please assign this issue to me?

Thanks,
David

[slaren]

I believe this is already being worked on in #10548

[david68cu]

Thanks @slaren

[lhpqaq]

@david68cu I only added the backend API in #10548, maybe you can continue to implement the frontend.

[ngxson]

This can ideally be added after #10599 and #10548 to avoid conflicts