Can the cost of each request be sent back in the response synchronously?

[portkey-support]

On Oct 17: Yokesh GK posted:

Actually I see the cost for the each prompts I made in my dashboard and I want the same in the final result of the chat/completions response along with the usage details

But I see that is not available in that endpoint as well as in other endpoints too. So I kindly request you guys to help me with this

[portkey-support]

Portkey team

Hey Yokesh GK the cost is computed after the request is completed as a post processing step, this is so that the response is not blocked
Is there any specific reason you'd want the cost in the response and not get it later by using the API or the dashboard?
you can export logs later for post processing using the API https://portkey.ai/docs/api-reference/admin-api/data-plane/logs/log-exports-beta/create-a-log-export
or the dashboard

[portkey-support]

Yogesh GK

Actually our team needs to generate a cost report for the prompts we use and to achieve this I need to save them in our DB
Can you guys provide the cost data too at the time of prompt response?
Or is there any other way to calculate the cost of each prompts from my side

[portkey-support]

Portkey-Team

It's not computed synchronously Yokesh GK , for computing the cost, there is a service which tokenizes the text and then calculates the cost, it needs to be non-blocking,
If this is the use case, you could consider exporting logs using the API above
If you know the model you want to calculate cost for, you could tokenize the text, different providers use different tokenizers (eg: tiktoken for OpenAI) then cost would be:
input_tokens * cost per input token + output_tokens * cost per output token
You’d need to get the rate from the provider’s pricing page

[portkey-support]

Discussion extracted from:
https://discord.com/channels/1143393887742861333/1143402158394454016/1296102890326200351