Is there a metric that measures how often a call to a LLM failed with errors like 502?

[cmasantos]

Hello,

I'm trying to find a way to measure how often a call to a LLM fails with erros like 502. I'm aware of the gen_ai_* metrics, but I think there isn't one to measure errors in the call.
I'm also aware of the @Observes AiServiceErrorEvent event where I could add a @Counted annotation, but then to test it, how can I make the AI service fail in a way that the counter is increased?

I'm looking for any tips that could make this effective and testable.

Thank you

[geoand]

👋🏽

gen_ai.request.model has an error.type type that should give information about the error. Does that not that not cover what you are after?

[cmasantos]

Maybe i'm doing something wrong. Do I need to enable some config or add a specific dependency in order to enable that metric? The only gen ai metrics I can see being generated are gen_ai_client_operation_* and gen_ai_client_token_usage_total.

[geoand]

You should not need to do anything to get these. Are you using the latest version of Quarkus LangChain4?

[cmasantos]

I've running 1.3.0 since it was released. I'll update to 1.4.0.CR2 and check if it changes something. But in theory after using an AI Service the metrics endpoint should return a gen_ai_request_model metric, right?

[geoand]

Correct