feat: Report more vllm metrics #92
Conversation
|
@yinggeh |
@Pavloveuge Thanks for contributing to Triton. I'll keep you posted once there's an update. |
|
@yinggeh any update? These metrics are important for production environments |
README.md
Outdated
| # Number of requests currently running on GPU. | ||
| gauge_scheduler_running | ||
| # Number of requests waiting to be processed. | ||
| gauge_scheduler_waiting | ||
| # Number of requests swapped to CPU. | ||
| gauge_scheduler_swapped | ||
| # GPU KV-cache usage. 1 means 100 percent usage. | ||
| gauge_gpu_cache_usage | ||
| # CPU KV-cache usage. 1 means 100 percent usage. | ||
| gauge_cpu_cache_usage | ||
| # CPU prefix cache block hit rate. | ||
| gauge_cpu_prefix_cache_hit_rate | ||
| # GPU prefix cache block hit rate. | ||
| gauge_gpu_prefix_cache_hit_rate |
There was a problem hiding this comment.
Can you move gauges before counters?
src/utils/metrics.py
Outdated
| (self.metrics.gauge_gpu_cache_usage, stats.gpu_cache_usage_sys), | ||
| (self.metrics.gauge_cpu_cache_usage, stats.cpu_cache_usage_sys), |
src/utils/metrics.py
Outdated
| (self.metrics.gauge_num_requests_running, stats.num_running_sys), | ||
| (self.metrics.gauge_num_requests_waiting, stats.num_waiting_sys), | ||
| (self.metrics.gauge_num_requests_swapped, stats.num_swapped_sys), | ||
| (self.metrics.gauge_gpu_cache_usage, stats.gpu_cache_usage_sys), |
There was a problem hiding this comment.
Please update namings to be consistent with https://github.com/vllm-project/vllm/blob/main/vllm/engine/metrics.py
src/utils/metrics.py
Outdated
| gauge_metrics = [ | ||
| (self.metrics.gauge_num_requests_running, stats.num_running_sys), | ||
| (self.metrics.gauge_num_requests_waiting, stats.num_waiting_sys), | ||
| (self.metrics.gauge_num_requests_swapped, stats.num_swapped_sys), |
There was a problem hiding this comment.
# Deprecated in 0.8 - KV cache offloading is not used in V1
# Hidden in 0.9, due to be removed in 0.10
There was a problem hiding this comment.
Remove deprecated metrics. thanks
|
@Pavloveuge Left comments |
Co-authored-by: Yingge He <[email protected]>
Co-authored-by: Yingge He <[email protected]>
Co-authored-by: Yingge He <[email protected]>
Co-authored-by: Yingge He <[email protected]>
Co-authored-by: Yingge He <[email protected]>
Co-authored-by: Yingge He <[email protected]>
|
@yinggeh Done |
Pavloveuge
force-pushed
the
report_more_vllm_metric
branch
2 times, most recently
from
1cb26b7 to
9ad54c2
Compare
Pavloveuge
force-pushed
the
report_more_vllm_metric
branch
from
9ad54c2 to
9b96279
Compare
|
@Pavloveuge Thanks. There is a concern over the performance degradation with more metrics being reported. While I am getting the data, can you continue your work by applying the change to |
|
Yes, I've already applied the change to |
What does the PR do?
Report more counter, histogram, gauge metrics from vLLM to Triton metrics server.
Checklist:
<commit_type>: <Title>Commit Type:
Check the conventional commit type
box here and add the label to the github PR.
Related PRs:
n/a
Where should the reviewer start?
n/a
Test plan:
ci/L0_backend_vllm/metrics_test
CI Pipeline ID:
n/a
Caveats:
n/a
Background
Customers requested additional histogram metrics from vLLM.
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
n/a