Phoneme-level timestamps and confidence

[TasseDeCafe]

Hi @MahmoudAshraf97,

I'm looking for a tool that could provide phoneme-level timestamps as well as confidence data (about the recognized phoneme, not the timestamp). Is your tool capable of returning this data?

I'm trying to build a tool that can provide feedback on pronunciation at the phoneme level. I've looked into MFA and WhisperX but they don't seem to be able to provide this data.

Thank you!

[MahmoudAshraf97]

This tool provides letter level timestamps and confidence, it's later aggregated to provide word or segment timestamps and confidence, you can use custom splitting and aggregation to adjust it to phoneme level, so it's doable but you'll need to write custom code

[TasseDeCafe]

Hi, thank you for the answer!

Are the confidence scores provided by the tool normalized posterior probabilities that can be interpreted as an actual measure of confidence (0-1), or are they likelihood scores that would need additional processing to be meaningful for pronunciation feedback?

I'm asking this question because if they're just likelihood scores, the values would be hard to interpret meaningfully. However, if they're proper posterior probabilities, they would be much more useful as a confidence score to evaluate pronunciation.

Thank you.

[MahmoudAshraf97]

They are not normalized, just the aggregated raw scores of each single letter, so the maximum will be 0 and the minimum will be -inf

[TasseDeCafe]

Thank you.

Are you aware of any tool that could provide normalized confidence scores?

[MahmoudAshraf97]

normalized scores imply some sort of known limits which I think only you can define, so I don't think such a tool exist, but I'm not completely sure

[TasseDeCafe]

Makes sense, I might try to have a go at it to see if it's actually possibly to have some meaningful feedback here. Thanks!