Does this package work alongside Vivox?

[Vivraan]

I've set up Vivox alongside a multiplayer setup in my project, which is being deployed to Quest 2 hardware. From what I understand, Vivox uses its own libraries to access the device microphone, which can then be tested in echo and group voice channels.

My question is: does this interfere in any way with how whisper.unity handles microphone usage requests (through Unity's Microphone API?)

Additionally, I haven't gotten it to work in the Quest 2 headsets.

[Macoron]

whisper.unity uses custom MicrophoneRecord script to make circular recording and run VAD. If Vivox uses their own custom logic for working with a microphone - you would need to rewrite that to make them work together with whisper.

Alternatively, whisper can work with an AudioClip. If Vivox can provide you final AudioClip - you can transcribe it and doesn't interact with a microphone directly. Check how to transcribe audio clips in samples.

As for running on Quest 2, it should work, but very slow. Check #47

[Vivraan]

Yes, I only just saw zhouwg/kantv#64, which shows somewhat fast inference on Android using the original whisper.cpp repo.

As for the Vivox integration, Vivox supports passing its audio outputs through Unity, so it's possible to use OnAudioFilterRead to directly pass in float arrays of the audio data to whisper.unity. I get terrible performance, though.