Willow with on device wake word

[janstadt]

Just curious if there has been any thought in to doing on device wake word detection via microwakeword? I find that it is more effective compared to openWakeWord currently (via native HA Voice). I havent tested it out via willow but was curious if anyone has or if this is even an option?

[nikito]

So the wake word processing is done on device (on the ESP Box) in willow, using the ESP-ADF libraries. We have a few different configurable wakewords, with more on the way :)

[janstadt]

Ah ok thanks. Just curious, how is it doing the wake word detection?

[nikito]

Not really sure how to answer, it's just part of the ESP-ADF libraries. It loads the model and the device listens for it. Once it hears it, that triggers it to start streaming to the inference server and wait for VAD to end listening. I imagine microWakeWord is using the same libraries, but I am not certain as I haven't looked at it in great detail.

[StuartIanNaylor]

I think you lose much of the ADF with the microWakeWord as the esspressif uses BSS (Blind source seperation) which turns the stereo mix into 2x source mixes. What it does then I think if you look at https://docs.espressif.com/projects/esp-sr/en/latest/esp32/benchmark/README.html#wakenet, seems to be is run a KWS for each channel as a mimo KWS and the KWS is used to find which channel contains the KW and maybe VAD.
BSS and advanced types such as target voice extraction have become the SotA methods of cleaning noise and RIR (Room Impulse Response) problem is with basic BSS you never know the channel order so you need some form of check and from the params of 1 to 3 channel it seems to double and they have 1 to 3 channel models.

Even though its named microwakeword its not the lightest KWS you could have and also to get best accuracy it supports layers that are not part of TF4Micro https://arxiv.org/pdf/2005.06720 as its orig focus was mobile. If the Espressif KWS is MIMO (multiple input multiple output) its not going to squeeze onto a esp32-s3.
Its hard to know for sure as much of the API is supplied as blobs but looks like it is around https://github.com/espressif/esp-box/blob/master/examples/factory_demo/main/app/app_sr.c

The worst problem is using synthetic data that has not been captured by the device as that can reduce accuracy considerabilly 10% or more. If you look at the arvix there is very little perfomance difference with a smaller model and the right dataset which is sadly something we are lacking (ondevice datasets).