Skip to content

chore(ai): Add metric for tracking ai cost calculation#5560

Merged
shellmayr merged 15 commits intomasterfrom
shellmayr/chore/add-metric-for-cost-calculation-ai
Feb 2, 2026
Merged

chore(ai): Add metric for tracking ai cost calculation#5560
shellmayr merged 15 commits intomasterfrom
shellmayr/chore/add-metric-for-cost-calculation-ai

Conversation

@shellmayr
Copy link
Member

@shellmayr shellmayr commented Jan 20, 2026

  • Add a new metric in relay-event-normalization genai.cost_calculation.result to track positive, negative and zero costs using a metric
  • Add finite tag matchers for the origin integration and platform, to prevent our cardinality from becoming too large
  • In the cost calculation function normalize_ai_costs, emit that metric depending on the outcome

Closes TET-1675

@linear
Copy link

linear bot commented Jan 22, 2026

@shellmayr shellmayr force-pushed the shellmayr/chore/add-metric-for-cost-calculation-ai branch from bb51b0b to 807f933 Compare January 26, 2026 08:36
@shellmayr shellmayr marked this pull request as ready for review January 28, 2026 14:09
@shellmayr shellmayr requested a review from a team as a code owner January 28, 2026 14:09
Copy link
Member

@Dav1dde Dav1dde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, just a tiny nit and what we discussed that it'll only cover a single pipeline not transaction and span streaming.

let integration = map_origin_to_integration(origin);
let platform = platform_tag(platform);

let Some(costs) = ai::calculate_costs(model_cost, tokens) else {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed in Slack, the change here would be only for the span streaming pipeline, not the transaction one (the only one in active use atm)

Maybe you can move the metric emission into ai::calculate_costs, then it covers both pipelines at once.

}
}

fn platform_tag(platform: Option<&str>) -> &'static str {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit awkward because we already have such a function in relay-server, but no good way of sharing it :(

Fine to keep here, would consider just starting utils.rs module in the crate to hide it away, but also not a big deal, just would move it at the end of the file, generally we like to keep public interfaces at the top.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would move this to statsd.rs module because it is a concern of metrics reporting (same for map_origin_to_integration).

Comment on lines 202 to 211
let metric_label = if (costs.input > 0.0 && costs.output > 0.0)
|| ((costs.input > 0.0 && costs.output == 0.0)
|| (costs.input == 0.0 && costs.output > 0.0))
{
"calculation_positive"
} else if costs.input < 0.0 || costs.output < 0.0 {
"calculation_negative"
} else {
"calculation_zero"
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like this might be more concise (untested):

Suggested change
let metric_label = if (costs.input > 0.0 && costs.output > 0.0)
|| ((costs.input > 0.0 && costs.output == 0.0)
|| (costs.input == 0.0 && costs.output > 0.0))
{
"calculation_positive"
} else if costs.input < 0.0 || costs.output < 0.0 {
"calculation_negative"
} else {
"calculation_zero"
};
let metric_label = match (costs.input.signum(), costs.output.signum()) {
(-1, _) | (_, -1) => "calculation_negative",
(0, 0) => "calculation_zero",
_ => "calculation_positive",
};

}
}

fn platform_tag(platform: Option<&str>) -> &'static str {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would move this to statsd.rs module because it is a concern of metrics reporting (same for map_origin_to_integration).

@shellmayr shellmayr requested a review from Dav1dde January 30, 2026 08:07
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

@shellmayr shellmayr changed the title chore(ai): add metric for tracking ai cost calculation chore(ai): Add metric for tracking ai cost calculation Jan 30, 2026
@shellmayr shellmayr requested a review from Dav1dde January 30, 2026 14:30
return;
};

// Overwrite all values, the attributes should reflect the values we used to calculate the total.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume removing that comment was a mistake?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, adding it back in

@shellmayr shellmayr added this pull request to the merge queue Feb 2, 2026
Merged via the queue into master with commit 6a0f237 Feb 2, 2026
28 checks passed
@shellmayr shellmayr deleted the shellmayr/chore/add-metric-for-cost-calculation-ai branch February 2, 2026 08:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants