Log token usage for Azure OpenAI Embedding with get_openai_callback

[henrymcl]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Azure OpenAI embeddings returns token usage by default

{
    "object": "list",
    "data": [
        {
            "object": "embedding",
            "index": 0,
            "embedding": [...]
        }
    ],
    "model": "text-embedding-ada-002",
    "usage": {
        "prompt_tokens": 1,
        "total_tokens": 1
    }
}

It is however not recorded by get_openai_callback:

Tokens Used: 0
        Prompt Tokens: 0
        Completion Tokens: 0
Successful Requests: 0
Total Cost (USD): $0.0
Verify embedding vector of length: 1024

Implementing this should be simple.

Motivation

Record accurate usage for cost tracking.

Proposal (If applicable)

No response

Versions:

langchain==0.1.17
langchain-community==0.0.37
langchain-core==0.1.52
langchain-openai==0.1.6
langchain-text-splitters==0.0.1

[cf-sewe]

I am looking too for an easy way how to find out the cost of my index() run with SQLRecordManager - considering only the processed chunks.