Problems getting good speech to text for auctions and fundraisers

[johnnydanger]

The issue is that Deepgram consistently incorrectly transcribes an auctioneer calling dollar amounts.

For example, the audio "seven thousand, seven, seventy five hundred, seven five, seventy five hundred, seven thousand, do I have seventy five hundred? At seven thousand, do I have seventy five hundred?" transcribes as follows:

7 1000, 700,75100, 75, 7500, 10000, 7000.
7 1000.
Do I have 75100? At 7000, do I have 75100?

I find it particularly odd that Deepgram will transcribe "seventy five hundred" as "75100". It does this often! no space, no comma, not properly transcribed without say "thousand" after "seven". Any ideas? The current settings are:

const deepgram = deepgramClient.listen.live({
model: "enhanced",
language: "en-US",
smart_format: true,
interim_results: true
});

[jpvajda]

@johnnydanger could you try this with our Nova-2 model? it might handle it better. Though numbers can always be tricky for STT, and the speed at which the auctioneer is saying the numbers could also make this somewhat challenging to get right.

Depending on if there are breaks between you might want to look at endpointing with interim results. Also check out our docs on utterance_end and speech_started

Playing with these different features of our Streaming STT API might get you what you need, but it might take some experimentation.

[jpvajda]

@johnnydanger you might get some value out of using keywords you could try to pass in numbers as keywords and try boost those numerical values you are trying to get the model to handle better.

Setting the keyword intensifier higher might help, so just play around with that

curl \
  --request POST \
  --header 'Authorization: Token YOUR_DEEPGRAM_API_KEY' \
  --header 'Content-Type: audio/wav' \
  --data-binary @youraudio.wav \
  --url 'https://api.deepgram.com/v1/listen?keywords=6500:1'

[johnnydanger]

Interesting suggestion. I gave it a try and it does indeed solve things... kind of. I put in the keywords: "4500", "5500", "6500", "7500", and now those transcribe properly. EXCEPT! Now when I say "eighty five hundred" is transcribes as "4500" every single time. Bah. Then I experimented more and found that by adding numbers as keywords gives a lot of random unexpected results. Now when I say "fifteen thousand" for instance it returns "6500", also every time. So alas, using numbers as keywords isn't a functional solution.

I wonder if there's any possibility of Deepgram altering how numbers are transcribed. It seems like a serious weak point, particularly if it was to be used as an automated solution for banks, auctions, or anything relating to saying a number.

[johnnydanger]

Just did a bit more experimenting, and found that "smart_format" is the issue when it comes to numbers. The transcription is great with smart format turned off—however doing so loses all the punctuation, line breaks, etc. so I would have to add more processing in my app to get that stuff back. Not the end of the world, though unfortunate. I wonder if there are any secret parameters that turn off number formatting when using smart_format? Because that would solve things tout suite!

[jpvajda]

@johnnydanger there are actually. try using numerals or punctuate and set to true

https://developers.deepgram.com/docs/numerals
https://developers.deepgram.com/docs/punctuation

These are the older formatting features that smart_format attempted to replace, but I know sometimes they work better independently

[johnnydanger]

Awesome, thanks John! I’ll dig into it and let you know the solution I land on. Thanks for all the help.

…

On Oct 19, 2024 at 9:24 AM -0700, John Vajda (JV) ***@***.***>, wrote: @johnnydanger there are actually. try using numerals or punctuate and set to true https://developers.deepgram.com/docs/numerals https://developers.deepgram.com/docs/punctuation These are the older formatting features that smart_format attempted to replace, but I know sometimes they work better independently — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>