Seeing high word error rate with nova2-general, what settings can we use to minimize errors?

[egoldschmidt]

We are building a conversational AI companion experience and leveraging Deepgram nova2-general (in streaming mode, interim results disabled), yet we are still seeing very high rates of transcription errors. Can you advise on the set of parameters to use that can offer the best performance for our use case (inclusive of whether we are using the best model for this use case, which is largely people using speakerphone to communicate with an AI agent). Thanks!

[team-deepgram]

Thanks for asking your question about Deepgram! If you didn't already include it in your post, please be sure to add as much detail as possible so we can assist you efficiently, such as:

• The request_id if you have a question about your requests or transcription responses.
• The features you used or the full api.deepgram.com URL you sent your request to, including parameters.
• Any code snippets you can share.

[jkroll-deepgram]

Hi @egoldschmidt, can you characterize the main types of errors you're seeing?

Given that you've tested Nova-2 General, a few other options to test are:

• Nova-2 Phonecall (for lower-bandwidth audio, and more background noise - your use case of being on speakerphone may be a good fit here) - model=nova-2-phonecall
• Nova General (slightly different architecture which has a lower occurrence of occasional missing words) model=nova
• Enhanced (higher-fidelity timestamps) model=enhanced

[egoldschmidt]

Mostly just incorrect transcriptions, e.g, errors like below (edited & anonymized of course):

Assistant: I appreciate that. It's nice to be here and chat with you. Remember, I'm just a call away whenever you need to talk.
User: Well, I need the dog right now.
Assistant: I get it. Dogs are such great comfort and companions. Enjoy your time with your furry friend!
User: What? That's not what I said.

(I presume the user said "I need to talk right now"?)

or

Assistant: I understand, how are you feeling about it today?
User: Good. How are you?
Assistant: I'm good, thanks! Always happy to chat with you.
User: Not you. Not you. I'm saying I didn't say how are you. I said not you.

(I presume the user said "not good"?)

Also I'm curious if nova-2-conversationalai is appropriate for our use case, or no?

[jkroll-deepgram]

@egoldschmidt If you do have Deepgram request IDs, I'd be happy to look at some of these failures further. These do seem to be pretty egregious errors, so I'd be interested to examine the audio and request parameters to look for root causes.

The Nova-2 conversational AI model is meant to transcribe TTS/synthetic voices, and it looks like you're just transcribing human voices, so a General or Phonecall model sounds like a better fit for your use case.

[egoldschmidt]

Hi @jkroll-deepgram here's one request ID to look into: 93f5ab40-0cdb-43ce-a761-02a0d5ebb7a0

Deepgram came back with "To Atlanta" when the user said "I like to garden". The raw recording is uploaded here (it was an internal test call, this is not user data)

Recording - go to about 0:46 in

[jkroll-deepgram]

Hi @egoldschmidt, thanks for sharing that example. Reproducing on your pre-recorded audio file (note that outputs can differ slightly compared to streaming), I was able to reproduce a transcription of "To Atlanta Garden" instead of "To plant a garden", using our Nova-2 model.

Here's an excerpt of the transcript from my test. Interestingly, it does indeed have low confidence on "Atlanta" (0.30).

...
              {
                "word": "to",
                "start": 48.165,
                "end": 48.665,
                "confidence": 0.974518,
                "punctuated_word": "To"
              },
              {
                "word": "atlanta",
                "start": 49.845,
                "end": 50.325,
                "confidence": 0.30575344,
                "punctuated_word": "Atlanta"
              },
              {
                "word": "garden",
                "start": 50.325,
                "end": 50.825,
                "confidence": 0.7098837,
                "punctuated_word": "Garden."
              },
...

Listening to the audio, I heard that the human speaker is pretty low-bandwidth and grainy, which made me think of Nova-2 Phonecall, which I then saw in your request parameters that you're already using. Retranscribing the file with that model corrected that particular error.

              {
                "word": "to",
                "start": 48.165,
                "end": 48.665,
                "confidence": 0.5535042,
                "punctuated_word": "To"
              },
              {
                "word": "plant",
                "start": 49.845,
                "end": 50.165,
                "confidence": 0.27928555,
                "punctuated_word": "plant"
              },
              {
                "word": "a",
                "start": 50.165,
                "end": 50.245003,
                "confidence": 0.92422426,
                "punctuated_word": "a"
              },
              {
                "word": "garden",
                "start": 50.245003,
                "end": 50.745003,
                "confidence": 0.9948418,
                "punctuated_word": "garden."
              },

Overall, this case is a pretty typical error of mistaking two similar words/phrases, "Atlanta" / "plant a". I see how it was semantically very different and provided a jarring user experience! Unfortunately I don't have a quick fix for this case, but it does confirm that Nova-2 Phonecall continues to be the model best suited for your use case.

[egoldschmidt]

Thank you Julia this is helpful, we will look more carefully at confidence scores going forward