(Fix) Prometheus time-to-first-token metric to use dynamic labels #17416
+101
−17
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Updated the `_set_latency_metrics` method to use the `prometheus_label_factory` for the time-to-first-token metric, ensuring it respects the configured metric labels instead of using hardcoded parameters. This change: - Removes unused parameters (model, user_api_key, user_api_key_alias, user_api_team, user_api_team_alias) from `_set_latency_metrics` - Uses `prometheus_label_factory` to generate labels dynamically based on the metric's supported labels - Ensures consistency with other metrics in the PrometheusLogger class 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Added test to verify that the time-to-first-token metric uses prometheus_label_factory for dynamic label generation instead of hardcoded parameters. The test ensures that: - The metric is properly registered - Metric samples are recorded during streaming requests - Labels are configured dynamically via get_labels_for_metric 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Added test to verify that the time-to-first-token metric properly uses dynamic label filtering via prometheus_label_factory. The test: - Configures the metric to only include specific labels (model, hashed_api_key, api_key_alias) - Excludes team and team_alias labels - Verifies that excluded labels do not appear in the recorded metrics - Ensures that included labels are present This validates that the recent fix to use prometheus_label_factory correctly respects the configured metric labels. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Title
(Fix) Prometheus time-to-first-token metric to use dynamic labels
Relevant issues
N/A
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unitType
🐛 Bug Fix
Changes
Updated the
_set_latency_metricsmethod to use theprometheus_label_factoryfor the time-to-first-token metric, ensuring it respects the configured metric labels instead of using hardcoded parameters.Without this recording this metric results in an error
This change:
_set_latency_metricsprometheus_label_factoryto generate labels dynamically based on the metric's supported labels