Use channels for metrics updates, added metrics tests #171
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ira <[email protected]>
mayabar
requested changes
Aug 26, 2025
pkg/llm-d-inference-sim/metrics.go
Outdated
|
|
||
| // reportWaitingRequests sets information about waiting completion requests | ||
| func (s *VllmSimulator) reportWaitingRequests() { | ||
| func (s *VllmSimulator) reportWaitingRequests(nWaitingReqs int64) { |
There was a problem hiding this comment.
no need to pass the nWaitingReqs, the s.nWaitingReqs has an updated value
pkg/llm-d-inference-sim/metrics.go
Outdated
|
|
||
| // reportRunningRequests sets information about running completion requests | ||
| func (s *VllmSimulator) reportRunningRequests() { | ||
| func (s *VllmSimulator) reportRunningRequests(nRunningReqs int64) { |
There was a problem hiding this comment.
no need to pass the nRunningReqs, the s. nRunningReqs has an updated value
| // waitingReqChan is a channel to update nWaitingReqs | ||
| waitingReqChan chan int64 | ||
| // loraInfo is prometheus gauge | ||
| loraInfo *prometheus.GaugeVec |
There was a problem hiding this comment.
maybe we need a channel for loraInfos? please create an issue for this
pkg/llm-d-inference-sim/simulator.go
Outdated
| kvcacheHelper: nil, // kvcache helper will be created only if required after reading configuration | ||
| namespace: os.Getenv(podNsEnv), | ||
| pod: os.Getenv(podNameEnv), | ||
| runReqChan: make(chan int64, 1000), |
There was a problem hiding this comment.
maybe we can define a constant for metrics channels size or even expose it in the configuration?
pkg/llm-d-inference-sim/simulator.go
Outdated
| case inc := <-s.waitingReqChan: | ||
| s.nWaitingReqs += inc | ||
| s.reportWaitingRequests(s.nWaitingReqs) | ||
| case inc := <-s.runReqChan: |
There was a problem hiding this comment.
please create a separate loop for each metric
Signed-off-by: Ira <[email protected]>
|
/lgtm /approve |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
mayabar
approved these changes
Aug 26, 2025
|
/lgtm |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #145
This PRs changes the way the metrics of number of running and waiting requests are updated. The current way of atomic increments and a separate Prometheus set is causing the metrics to be inconsistent. Therefore, a use of channels is added.
Also, this PR adds metrics tests, and reorganizes simulator tests a bit.