ES8388: Slow Audio Output and Microphone not running #2
Comments
|
Thank you for bringing this to my attention. Currently, I don't have access to external DACs to perform tests. It's unclear whether the adf_pipeline component is compatible with 'source: github://pr#5230'; it's likely incompatible. You may consider removing it since adf_pipeline already incorporates the idf-adf SDK. Additionally, could you experiment with a different TTS voice? The one you're currently using generates samples at a 24000 rate, which appears to be unsupported or at least not set correctly. You could also try testing if it is possible to stream radio stations using the radio browser component. And I just noticed, you should remove the speaker component from you configuration when you use voice assistant with media_player ... |
|
Can you try using the branch: 3-make-supported-i2s-sampling-rates-configurable - source:
type: git
url: https://github.com/nighi/esphome_audio
ref: 3-make-supported-i2s-sampling-rates-configurable
components: [ adf_pipeline, i2s_audio ]
refresh: 0sI removed any restriction on setting the i2s sampling rate and number of channels in this branch Regarding the microphone: Unfortunately I can't see any helpful information in you log. Could you set your use_wake_word switch to RESTORE_DEFAULT_OFF, wait until all components are set up and then switch the voice assistant component on manually? Hopefully the log will show more relevant information then... |
|
Oh, good hint with the idf-adf, thought I'd removed it. I removed the external component and the speaker, but it seems to make no difference. I've tried different audio sources, but nothing really works. Some tracks are even to fast or to slow. I even With your new branch Deutschlandfunk and BBC World from the radiobrowser works correctly. The tts engines and the 16K-sample doesn't work and sound is additionally distorted. Regarding the microphone I also realized that I only pasted the tail of my log. Here is the full: log |
|
Thanks, that's very helpful information to localise the bug! I will look into it... |
|
Ok, I committed a fix for switching the number of channels into the 3-make-supported-i2s-sampling-rates-configurable branch. Can you check if it helps with your problem? I will look into the microphone problem later... |
|
Okay, it seems to be better. Switching from 2 to 1 channel still causes some problems, though. output |
|
Do the problems only occur when you switch audio streams or also when you start an audio stream the first time? Regarding the microphone, in your configuration you have set channel to left, but in the external component you are loading, the channel register on the ES8388 ist set hard coded to right: /@nightdav
// set to Mono Right
this->write_byte(0x0B, 0x10); When the microphone component only receives zeros for a specific time, it sets its state to 'stopped'. This could be the reason why you receive the 'microphone is not running' error. |
|
Yes, exactly this occurs only if I plqy first a 2 channel audio and than a 1 channel audio. I'll looking into it to only configure a mediaplayer. Regarding the microphone: I changed the config to |
|
Thanks for testing, I just committed a new version to the 3-make-supported-i2s-sampling-rates-configurable branch. |
|
The playback works like a charm 🎉. Unfortunately, it isn't possible to change the volume anymore. But re-enabling ALC does the trick |
|
Perfect, yes that was exactly the reason for disabling it. I will make it configurable though, so people that use the volume control of their sound chip can disable alc. |
|
Regarding the microphone, I guess the problem is within using the same i2s-controller for input and output. I will look into it, but I need to prepare some hardware for testing i2s in duplex-mode first. Is it possibility for you to test your microphone configuration alone? Without configuring any speaker or media_player component? |
|
I tested the microphone with a silent voice assistant (without media_player or speaker). It seems to successfully register the microphone, but the wake word is not detected. I think the state change from the voice-assistant from After toggling log after enabling use wake wordI could also do a i2s capture with a cheap logic analyzer if this helps. |
|
the log file looks quite normal to me so far, maybe the sampling rate or channel settings or something like that is not correctly set. Could you check if you notice something wrong when listening to the audio that home assistant receives? You can configure that the assist-pipeline of HA stores the received audio file for debugging purposes: assist_pipeline:
debug_recording_dir: /config/debug |
|
It seems like to create one empty .wav file. If I toggle use wake word, it repeatedly creates exactly this file. Every time the same 44 Bytes. ➜ ~ cd media/assist-pipeline/3f14b00d61113444bb926dbf819e24f5/Home\ Assistant/591206133912532
➜ 591206133912532 hexdump -C 00_wake-wake_word.openwakeword_2.wav
00000000 52 49 46 46 24 00 00 00 57 41 56 45 66 6d 74 20 |RIFF$...WAVEfmt |
00000010 10 00 00 00 01 00 01 00 80 3e 00 00 00 7d 00 00 |.........>...}..|
00000020 02 00 10 00 64 61 74 61 00 00 00 00 |....data....|
0000002c
➜ 591206133912532 cd ..
➜ Home Assistant cd 590593051580302
➜ 590593051580302 hexdump -C 00_wake-wake_word.openwakeword_2.wav
00000000 52 49 46 46 24 00 00 00 57 41 56 45 66 6d 74 20 |RIFF$...WAVEfmt |
00000010 10 00 00 00 01 00 01 00 80 3e 00 00 00 7d 00 00 |.........>...}..|
00000020 02 00 10 00 64 61 74 61 00 00 00 00 |....data....|
0000002c
➜ 590593051580302 cd ../591206252426423
➜ 591206252426423 hexdump -C 00_wake-wake_word.openwakeword_2.wav
00000000 52 49 46 46 24 00 00 00 57 41 56 45 66 6d 74 20 |RIFF$...WAVEfmt |
00000010 10 00 00 00 01 00 01 00 80 3e 00 00 00 7d 00 00 |.........>...}..|
00000020 02 00 10 00 64 61 74 61 00 00 00 00 |....data....|
0000002c |
|
Hey, |
|
Hey, voice_assistant:
volume_multiplier: 6
noise_suppression_level: 4
auto_gain: 20dBFSThese were my settings which I've tuned with my old yaml config and I think they should still apply anyway. Without the high noise_suppression I got very noisy audio. A higher gain and volume multiplier seems to don't change anything. If I scream directly into the microphone I am able to trigger the wake word detection though. The audio is attached: I think (and correct me if I'm wrong) for the settings for For the parallel use with the media player your fix doesn't help, which is expected I guess. |
|
The wake work detection expects a mono signal with 16k sampling rate and 16bit bits_per_sample but these settings seem to be correct in your configuration and also in the es8388 register settings. Ok, you have the old configuration which uses the components from pr#5230 and which is working, right? |
|
I have added a version where the i2s controllers are released after stopping the pipeline. This doesn't allow full duplex yet but could you try if this helps with you hardware? |
|
Ok, we are one step further. I think you mean by not full duplex yet that the voice-assistant and the media-player may not In general I think both works although I weren't able to trigger the wakeword due to the very quiet recording. Could it be that the settings from the voiceassistant component aren't applied for the microphone pipeline? It seems like changing the settings doesn't change anything. I already tried to enable ALC for the input but no difference. Log testing first media player second voice-assistantBut there are still some reconnects/exceptions? And I was sometimes able to trigger the microphone not running errror loop. unexpected reconnect and microphone not running after stopping the mediaplayer pipeline.Another thing I noticed: After stopping the mediaplayer by Let me know if I can test any special scenario. |
|
I would like to test if any of your fixes of the other issues has fix any of mine. But the duplex branch is not up to date and only partially merged to main. Is that correct or should main already contain all changes of the duplex branch? I tried to merge it myself but couldn't figure out which changes I need. Anyhow as I have seen there is also a custom voice assistant component. Should I ve tested with this? I'm pretty shure I didn't. Also is the esphome version important? Since there is a target version in the readme. |
|
Hey, you could either try the main branch or the vad-test branch, they should both work in half duplex mode. The custom voice assistant component, ensures that the va pipeline gets only started when the microphone is ready and in the vad-test branch it allows using vad with adf pipeline.Am 31.03.2024 um 18:06 schrieb nighi ***@***.***>:
I would like to test if any of your fixes of the other issues has fix any of mine. But the duplex branch is not up to date and only partially merged to main. Is that correct or should main already contain all changes of the duplex branch?
I tried to merge it myself but couldn't figure out which changes I need.
Anyhow as I have seen there is also a custom voice assistant component. Should I ve tested with this? I'm pretty shure I didn't. Also is the esphome version important? Since there is a target version in the readme.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>
|
|
I just found my es8388 a1s in a box and started to look if the support for them is more stable now than when I last tried. I would love to give them another shot with the help of your amazing work. Could you give me some hints on where to start testing as you provide several routes of integration and some of the problems discussed in this thread seem to be solved in the meantime… And I noticed your integration of the init-code for the es8388-chips into this repo – so the external component by mulcmu shouldn't be needed anymore? Correct? Any advice would be much appreciated and I would be a happy tester ;) Thanks in advance! |
|
Hey, sorry I couldn't find much time to test lately. I could provide my config with the new syntax. Maybe @gnumpi could doublecheck, if this is the intended use. Unfortunately there are still problems left. esphome:
name: voice-sat
name_add_mac_suffix: false
on_boot:
- priority: -100
then:
- wait_until: api.connected
- delay: 1s
- if:
condition:
switch.is_on: use_wake_word
then:
- voice_assistant.start_continuous:
api:
logger:
ota:
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
captive_portal:
esp32:
board: esp-wrover-kit
framework:
type: esp-idf
external_components:
- source: github://mulcmu/esphome-es8388-a1s@nightdav-patch
components: [es8388]
refresh: 0s
- source:
type: git
url: https://github.com/gnumpi/esphome_audio
ref: main
components: [ adf_pipeline, i2s_audio]
refresh: 0s
i2c:
sda: GPIO33
scl: GPIO32
es8388:
i2s_audio:
id: i2s_in_out
i2s_lrclk_pin: GPIO25
i2s_bclk_pin: GPIO27
i2s_mclk_pin: GPIO0
access_mode: duplex
adf_pipeline:
# create an I2SWriter pipeline element
# using the i2s_out configuration
- platform: i2s_audio
type: audio_out
id: adf_i2s_out
i2s_audio_id: i2s_in_out
i2s_dout_pin: GPIO26
dac:
model: es8388
sample_rate: 16000
bits_per_sample: 16bit
fixed_settings: true
# create an I2SReader pipeline element
# using the i2s_in configuration
- platform: i2s_audio
type: audio_in
id: adf_i2s_in
i2s_audio_id: i2s_in_out
i2s_din_pin: GPIO35
channel: right
pdm: false
adc:
model: generic
bits_per_sample: 16bit
fixed_settings: true
microphone:
- platform: adf_pipeline
id: adf_microphone
pipeline:
- adf_i2s_in
- resampler
- self
media_player:
- platform: adf_pipeline
id: adf_media_player
name: s3-dev_media_player
internal: false
pipeline:
- self
- resampler
- adf_i2s_out
voice_assistant:
microphone: adf_microphone
media_player: adf_media_player
id: va
volume_multiplier: 8.0
noise_suppression_level: 4
auto_gain: 31dBFS
#vad_threshold: 3
on_end:
- delay: 100ms
- wait_until:
not:
media_player.is_playing:
- script.execute: reset_led
on_listening:
- light.turn_on: led
switch:
- platform: gpio
pin: GPIO21
name: "AMP Switch"
restore_mode: ALWAYS_OFF
- platform: template
name: Use wake word
id: use_wake_word
optimistic: true
restore_mode: ALWAYS_OFF #RESTORE_DEFAULT_ON
entity_category: config
on_turn_on:
- lambda: id(va).set_use_wake_word(true);
- if:
condition:
not:
- voice_assistant.is_running
then:
- voice_assistant.start_continuous
- script.execute: reset_led
on_turn_off:
- voice_assistant.stop
- lambda: id(va).set_use_wake_word(false);
- script.execute: reset_led
script:
- id: reset_led
then:
- if:
condition:
switch.is_on: use_wake_word
then:
- light.turn_on: led
else:
- light.turn_off: led
light:
- platform: binary
id: led
name: "Test LED 1"
output: light_output_1
- platform: binary
name: "Test LED 2"
output: light_output_2
output:
- id: light_output_1
platform: gpio
pin: GPIO22
inverted: true
- id: light_output_2
platform: gpio
pin: GPIO19
inverted: true
I tried to play different files and get different kind of errors and exceptions with reconnects. The only stream that worked is the "Deutschlandfunk" from the radio browser (Media URL: http://st01.dlf.de/dlf/01/128/mp3/stream.mp3). LogSorry for just pasting the whole log. If I should test something specific please tell me. |
|
Hey, thanks for the config! Just to be sure: did you capture the log on latest main? From looking around the code, my assumptions are:
Based on those assumptions you should remove the inclusion of the nightdev-custom-component as you are basically initializing the esp8388 two times. Not sure if that's a problem though… Basically I wanted to get some baseline for my expectations before starting to test, to reduce friction and be sure to tackle real problems and not just repeating other peoples findings… |
|
Yes, you were right. I was not sure whether I need this However, I could get other media sources to work by adding I could also get the wakeword detection working, as I removed the Also, I could trigger exceptions with reconnect, if I play something on the mediaplayer when Log 2 channel mediaMy updated config with a small automation of the amp switch (was done in HA before). :esphome:
name: voice-sat
name_add_mac_suffix: false
on_boot:
- priority: -100
then:
- wait_until: api.connected
- delay: 1s
- if:
condition:
switch.is_on: use_wake_word
then:
- voice_assistant.start_continuous:
api:
logger:
ota:
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
captive_portal:
esp32:
board: esp-wrover-kit
framework:
type: esp-idf
external_components:
- source:
type: git
url: https://github.com/gnumpi/esphome_audio
ref: main
components: [ adf_pipeline, i2s_audio]
refresh: 0s
i2c:
sda: GPIO33
scl: GPIO32
i2s_audio:
id: i2s_in_out
i2s_lrclk_pin: GPIO25
i2s_bclk_pin: GPIO27
i2s_mclk_pin: GPIO0
access_mode: duplex
adf_pipeline:
# create an I2SWriter pipeline element
# using the i2s_out configuration
- platform: i2s_audio
type: audio_out
id: adf_i2s_out
i2s_audio_id: i2s_in_out
i2s_dout_pin: GPIO26
adf_alc: true
dac:
model: es8388
sample_rate: 16000
bits_per_sample: 16bit
fixed_settings: true
channel: right
# create an I2SReader pipeline element
# using the i2s_in configuration
- platform: i2s_audio
type: audio_in
id: adf_i2s_in
i2s_audio_id: i2s_in_out
i2s_din_pin: GPIO35
channel: right
pdm: false
adc:
model: generic
bits_per_sample: 16bit
fixed_settings: true
microphone:
- platform: adf_pipeline
id: adf_microphone
pipeline:
- adf_i2s_in
- self
media_player:
- platform: adf_pipeline
id: adf_media_player
name: s3-dev_media_player
internal: false
pipeline:
- self
- resampler
- adf_i2s_out
on_play:
then:
- switch.turn_on: amp_switch
on_idle:
then:
- switch.turn_off: amp_switch
on_pause:
then:
- switch.turn_off: amp_switch
voice_assistant:
microphone: adf_microphone
media_player: adf_media_player
id: va
volume_multiplier: 4.0
noise_suppression_level: 2
auto_gain: 20dBFS
#vad_threshold: 3
on_end:
- delay: 100ms
- wait_until:
not:
media_player.is_playing:
- script.execute: reset_led
on_listening:
- light.turn_on: led
switch:
- platform: gpio
id: amp_switch
pin: GPIO21
name: "AMP Switch"
restore_mode: ALWAYS_OFF
- platform: template
name: Use wake word
id: use_wake_word
optimistic: true
restore_mode: RESTORE_DEFAULT_ON #ALWAYS_OFF
entity_category: config
on_turn_on:
- lambda: id(va).set_use_wake_word(true);
- if:
condition:
not:
- voice_assistant.is_running
then:
- voice_assistant.start_continuous
- script.execute: reset_led
on_turn_off:
- voice_assistant.stop
- lambda: id(va).set_use_wake_word(false);
- script.execute: reset_led
script:
- id: reset_led
then:
- if:
condition:
switch.is_on: use_wake_word
then:
- light.turn_on: led
else:
- light.turn_off: led
light:
- platform: binary
id: led
name: "Test LED 1"
output: light_output_1
- platform: binary
name: "Test LED 2"
output: light_output_2
output:
- id: light_output_1
platform: gpio
pin: GPIO22
inverted: true
- id: light_output_2
platform: gpio
pin: GPIO19
inverted: true |
|
Hey, yes you are right with your config you don’t need the es8388 component anymore but right now the implementation does only do a fixed initialization. The code is prepared to set the es8388 settings according to the the I2S settings but it is not fully implemented yet. As I don’t own an es8388 myself it’s not too easy to do the testing. Another idea is to use the work of #27 and implement a generic ADF DAC device which uses the implementations already available as part of the adf sdk. It‘s on my list, but I still want to fix other issues first. Regarding your exceptions and reboots you can try the next-dev branch which includes some code rewrites addressing this issue… |
|
I just wanted to try some testing and noticed that one of my two boards seem to be completely broken (no serial at all) and one has a broken headphone jack. I'll try to see if I can fix them or if I order some new ones in cn. But for the time being I might not be that helpful… ;) That said, I still think that it would be worth to have a working implementation of ES8388 as those chips are the cheapest and easiest option to build your own ha/esphome audio-device out there. Regarding the basic implementation of the code: From my notes for reference:
@nighi try to set the sample_rate to 44100 and enable apll-timers (see readme) and see if that improves quality. EDIT: @gnumpi I ordered 2 boards from aliexpress. Lead time ~2 weeks. If you are interested I would sent one over when they arrive. |
|
@gnumpi got my new boards and started testing. Mp3-playback with a http-stream seems to be working fine, but ssl seems to be crashing: Any ideas? I'm on the |
|
@nighi what device are you working on here? I'm trying to make a 'esp32-audio-kit' v2.2 work and I was wondering if you were working on the same board or something different. |
|
Hi @blitzaxt, yes I have the same hardware. But I was lately not working on it and haven't kept up on the latest development. |
|
That makes sense, I really like the idea of the audio kit though for my purposes 🤷. If it's of any use, I was able to find out that using EDIT: EDIT 2: final for real now Working media player on esp32-audio-kit v2.2esphome:
name: esp32-audio-kit-mww-adf
name_add_mac_suffix: false
api:
encryption:
key: !secret api_key
logger:
ota:
- platform: esphome
id: ota_esphome
password: !secret ota_password
wifi:
ssid: !secret new_wifi_ssid
password: !secret new_wifi_password
captive_portal:
web_server:
port: 80
esp32:
board: esp-wrover-kit
framework:
type: esp-idf
external_components:
- source:
type: git
url: https://github.com/gnumpi/esphome_audio
ref: main
components: [ adf_pipeline, i2s_audio]
refresh: 0s
- source: github://mulcmu/esphome-es8388-a1s@nightdav-patch
components: [es8388]
refresh: 0s
es8388:
i2c:
sda: GPIO33
scl: GPIO32
scan: true
i2s_audio:
id: i2s_in_out
i2s_lrclk_pin: GPIO25
i2s_bclk_pin: GPIO27
adf_pipeline:
- platform: i2s_audio
type: audio_out
id: adf_i2s_out
i2s_audio_id: i2s_in_out
i2s_dout_pin: GPIO26
media_player:
- platform: adf_pipeline
id: adf_media_player
name: s3-dev_media_player
internal: false
pipeline:
- self
- adf_i2s_out
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thank you for this component and your efforts, really appreciate.
Hardware:
DAC: es8388
Board: AI-Tinker ESP32-A1S
Problem 1: Audio output is much too slow
Audio output of the media player is working but is much too slow, maybe half of the original velocity.
Problem 2: Microphone not running
The log message
is present as soon as I start the voice-assistant. But no other error message on the log.
config.yaml
As reference: working config.yml without adf-pipeline
log output slow audio output
log output microphone not running
Are external DACs already supported? Do I have to specify this in the config?
If you need further details, feel free to ask.
The text was updated successfully, but these errors were encountered: