What is the best way of measuring the performance of each step of a conversation?

[rikkiprince]

I'm looking at understanding the performance of our agent that uses RubyLLM and Rails integration.

Is the time between message.created_at and message.updated_at a good measure of how long that step took? As in, does it give a good indication of how long the LLM took to respond and how long the tool took to respond?

I know I could add some telemetry into our code to measure some of these steps, but I'm also looking at understanding the conversations users have already had in the past.

Thanks in advance!

[crmne]

What are your requirements? Do you need absolute precision of how long the LLM took to respond from the moment it received the API call, or you're OK with corroborating that with some Ruby call time? Do you need the time from API call to first token or from API call to response completed?

If you're OK with not being absolutely precise (why would you in such a non-deterministic system anyway, but I digress) you can simply clock the time between chat.ask and on_end_message for time to complete response, chat.ask and on_new_message if you want time to first token, and on_tool_call and on_tool_result for tool execution time.