[RFC]: Optimize the metric.

[amy-why-3459]

Motivation.

vllm-project#533

Proposed Change.

vllm-project#533

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[LJH-LBJ]

正在梳理metrics结构

[LJH-LBJ]

1.12 开始方案设计

[zengchuang-hw]

1.19 SEG评审，开发同步做，1.25前交付

[LJH-LBJ]

1.13 设计进度20%

[LJH-LBJ]

1.14 方案设计延续目前实现，相当于是另外一套logger记录，在讨论过后觉得尽量与vllm保持一致，需要探索能不能自定义logger去记录omni特有数据

[LJH-LBJ]

1.15 方案设计基本完成，同步开发

[LJH-LBJ]

1.16 代码50%

[LJH-LBJ]

1.20 针对SEG会议上提出的意见进行修改
1、数据类型要与名称对应，ms留三位，bytes -> kb，整理字段语义，字段排布，打印成表格，复用打印开关
2、增加其他模态的统计量
3、每条请求都打印

[LJH-LBJ]

预期输出
StageRequestStats by_stage
+----------+---------------+---------------+----------+------------+-------------------+-------------------+-------------------+----------------------+------------------------+----------------------------+
| stage_id | num_tokens_in | num_tokens_out| batch_id | batch_size | stage_gen_time_ms | rx_transfer_bytes | rx_decode_time_ms | rx_in_flight_time_ms | stage_stats_total_token | stage_stats_total_gen_time |
+----------+---------------+---------------+----------+------------+-------------------+-------------------+-------------------+----------------------+------------------------+----------------------------+
| 0 | 4826.0 | 267.0 | 236.0 | 1.0 | 156.47 | 112104749.0 | 147.55 | 2.17 | 32767.0 | 888833.61 |
| 1 | 4826.0 | 267.0 | 236.0 | 1.0 | 156.47 | 112104749.0 | 147.55 | 2.17 | 32767.0 | 888833.61 |
| 2 | 4256.0 | 0.0 | 0.0 | 1.0 | 158.98 | 29503685.0 | 36.25 | 6.45 | 0.0 | 1038944.77 |
+----------+---------------+---------------+----------+------------+-------------------+-------------------+-------------------+----------------------+------------------------+----------------------------+

TransferEdgeStats by_edge
+--------+------------+----------+-------------+------------+-------------------+-------------------+
| edge | from_stage | to_stage | size_bytes | tx_time_ms | rx_decode_time_ms | in_flight_time_ms |
+--------+------------+----------+-------------+------------+-------------------+-------------------+
| 0->1 | 0.0 | 1.0 | 112104749.0 | 104.32 | 147.55 | 2.17 |
| 1->2 | 1.0 | 2.0 | 29503685.0 | 34.85 | 36.25 | 6.45 |
+--------+------------+----------+-------------+------------+-------------------+-------------------+

RequestE2EStats
+--------------+------------------+------------------------+----------------------+
| e2e_total_ms | e2e_total_tokens | transfers_total_time_ms| transfers_total_bytes|
+--------------+------------------+------------------------+----------------------+
| 29,047.01 | 5199.0 | 331.59 | 141,608,434.0 |
+--------------+------------------+------------------------+----------------------+

[LJH-LBJ]

按照意见修改中
预期输出
StageRequestStats

Field	0	1	2
num_tokens_in	4,826	4,826	4,256
num_tokens_out	267	267	0
batch_id	236	236	236
batch_size	1	1	1
stage_gen_time_ms	156.468	156.468	158.984
rx_transfer_kbytes	109,475.636	109,475.636	28,815.799
rx_decode_time_ms	147.546	147.546	36.251
rx_in_flight_time_ms	2.167	2.167	6.450
stage_stats_total_token	32,767	32,767	0
stage_stats_total_gen_time_ms	888,833.606	888,833.606	1,038,944.769

TransferEdgeStats

Field	0->1	1->2
from_stage	0	1
to_stage	1	2
size_kbytes	109,475.636	28,815.799
tx_time_ms	104.324	34.849
rx_decode_time_ms	147.546	36.251
in_flight_time_ms	2.167	6.450

RequestE2EStats

Field	Value
e2e_total_ms	29,047.010
e2e_total_tokens	5,199
transfers_total_time_ms	331.586
transfers_total_kbytes	138,338.696