How hard would it be to integrate openai whisper to this part of the stack? #72

[lahwran]

https://github.com/openai/whisper

would this be an appropriate place to integrate openai whisper, for handling of dictation? note that as of right now, whisper doesn't support streaming, and it looks to me like it may require transfer learning to be able to add streaming support to it; it might work as an alternate kaldi backend, or it might work best as an alternative to

(I've asked this question in other repos as well (kaldi-active-grammar), and I'm crosslinking between the questions)

[LexiconCode]

The best place to ask this question is actually within the dragonfly repository. (I could transfer this there if you like)
The dragonfly repository implements integrating speech recognition backends.

if you would like to play around with it the following gives you a chance to experiment in real time.
The demo below is pretty rudimentary therefore it's a bit slow in the latency department but I think that could be improved with modern VAD.

So getting real-time dictation would be pretty easy to do however plumbing it up to plumbing it into dragonflys grammar recognition and contexts is something more to learn about.

Take a look at the engines that already exist https://github.com/dictation-toolbox/dragonfly/tree/master/dragonfly/engines

https://github.com/tekacs/openai-whisper-realtime/blob/main/openai-whisper-realtime.py
py -m pip install poetry
py -m poetry install
py -m poetry shell
python openai-whisper-realtime.py

Other resources
https://github.com/shirayu/whisper_streaming
openai/whisper#2
https://github.com/fcakyon/pywhisper

[drmfinlay]

I will defer to David Zurow on whether openai whisper would be better than KaldiAG overall. Even if it is, I would prefer not to add another engine to Dragonfly. What we have is enough, I think.

It is possible to write custom engines outside Dragonfly. One need only register an instance of the custom engine class before loading grammars:

from dragonfly.engines import register_engine_init

engine = CustomEngine()
register_engine_init(engine)

Custom engine classes may be developed with minimal changes to Dragonfly's test suite code (TDD).

I'll add the above to the Dragonfly documentation.

[lahwran]

okay cool. I'll look at KAG primarily as a reference, then. thank you!

[drmfinlay]

No problem. I wish you good luck.

[LexiconCode]

@lahwran here is a basic implementation for an alternate dictation source for kaldi daanzu/kaldi-active-grammar#73