Transcription Errors When Using Nova-3

[afurkankjf]

Hi,

We have a voice agent service built in Python, where we use Deepgram’s Speech-to-Text WebSocket API for real-time speech transcription. During live sessions, the user’s speech is transcribed via Flux. After the session ends, we make an HTTP request with the session’s recording to get the transcription of the whole session where we utilize the Nova-3 model.

Also note that in all our recordings, we have 2 channels: the user and the AI agent. The agent channel includes a synthetic voice generated by a Text-to-Speech model, and the user channel is organic.

Our question is related to the second step where we try to get the transcription of the whole recording. In the sample whose request ID is given below, around 1:20, the user says “Okay. I would like that.”, but it is transcribed as “Okay, I won't let you do it.” by Nova-3.

Request ID: a6697888-b156-4b55-a2e1-c7bfd17d3b94

We would appreciate your help and insights into why this might be happening and how we can improve such cases.

Thanks in advance.

[deepgram-community[bot]]

Thanks for asking your question. Please be sure to reply with as much detail as possible so the community can assist you efficiently.
_{Consider joining our Discord community for more opportunity to engage with your fellow Deepgram users. You can earn points which can be redeemed for cool stuff by being active in our communities!}

[deepgram-community[bot]]

Hey there! It looks like you haven't connected your GitHub account to your Deepgram account. You can do this at https://community.deepgram.com - being verified through this process will allow our team to help you in a much more streamlined fashion.

[Jacob-Lasky]

I have a similar recommendation here as I did with #1499 , specifically around confidence scores. What I see is:

{
  "start": 80.32001,
  "end": 81.68,
  "confidence": 0.722198,
  "channel": 0,
  "transcript": "Okay, I won't let you do it.",
  "words": [
    {
      "word": "okay",
      "start": 80.32001,
      "end": 80.8,
      "confidence": 0.5939101,
      "punctuated_word": "Okay,"
    },
    {
      "word": "i",
      "start": 80.8,
      "end": 80.880005,
      "confidence": 0.8162796,
      "punctuated_word": "I"
    },
    {
      "word": "won't",
      "start": 80.880005,
      "end": 81.04,
      "confidence": 0.6561969,
      "punctuated_word": "won't"
    },
    {
      "word": "let",
      "start": 81.04,
      "end": 81.200005,
      "confidence": 0.38693348,
      "punctuated_word": "let"
    },
    {
      "word": "you",
      "start": 81.200005,
      "end": 81.36,
      "confidence": 0.89967084,
      "punctuated_word": "you"
    },
    {
      "word": "do",
      "start": 81.36,
      "end": 81.520004,
      "confidence": 0.99133813,
      "punctuated_word": "do"
    },
    {
      "word": "it",
      "start": 81.520004,
      "end": 81.68,
      "confidence": 0.7110572,
      "punctuated_word": "it."
    }
  ],
  "id": "38296353-6788-4c9b-aa37-a34bca921818"
}

This has a relatively low confidence score. It is higher than my initial recommendation of 0.65 but, just as an example, we usually provide scores > 0.90:

{
  "start": 117.405,
  "end": 117.805,
  "confidence": 0.9915129,
  "channel": 0,
  "transcript": "Yes.",
  "words": [
    {
      "word": "yes",
      "start": 117.405,
      "end": 117.805,
      "confidence": 0.9915129,
      "punctuated_word": "Yes."
    }
  ],
  "id": "1d256df1-7bca-466a-98e9-1d65ab6dbe9d"
}

My suggestion on improving this is to verify what the user said when low confidence transcripts come through.