Split stereo channels into different prediction calls

[fherb2]

I use the BirdNetPi since March with a stereo microphone setup. Especially in the morning hours a lot of bird calls are in parallel. If I check the original sound, so I see the amount of overlaid calls. But a part is simple to separate by use the left and right channel independent. It looks like, the stereo sound is used by the BirdNet model as a sum, a mono signal. I found this information about the mono configuration in the original projects of Stefan Kahl. I would wish a feature to switch from mono signal prediction into a parallel prediction for each stereo channel. After the prediction from BirdNet model, the BirdNetPi should merge both results. If the prediction is the same in left and right channel, so it would be one event. But if the results are different, so we get two single events.

Have some other users the same interest in this feature?

Best regards,
Frank

[HolgerTs]

Have some other users the same interest in this feature?

Yes, I do!

Checking some of the shell and python scripts, birdnet is made from, I can see, that number of channels set in birdnet.conf file, does not have an impact at all. I already took into account to create a bug report by this.

Example 1) In file extract_new_birdsounds.sh we ca see following line:
sox -V1 "${NEWSPECIES_BYDATE}/${NEWFILE}" -n remix 1 rate 24k spectrogram ...
"remix 1" means, only first channel is used to create spectrogram pictures. This is done for all the samples.

Example 2) In file server.py we can see following line:
sig, rate = librosa.load(path, sr=sample_rate, mono=True, res_type='kaiser_fast')
"mono=True" means, stereo samples are converted to mono when loading sound file for prediction.

Maybe one of the developers can explain, what was the aim of introducing CHANNELS variable.

I strongly support fherb2's idea as described above.

Best regards,
holger

[ehpersonal38]

Yeah I really can't answer this... Patrick put channels in the conf for some reason even though the BirdNET code converts all stereo tracks to mono with mono=True. Does the BirdNET model actually support stereo files? @kahst

[fherb2]

Hi,

I think it makes no sens to have a model for stereo, so long the model doesn't recognize and use context information from the sound short before or after the analyzing range of the 3 seconds chunk. But it makes sens to analyze the sound in different directions by using a mono-channel model, I think. Last would be only like a direction depending loudness filter against an omni-Sensor before species prediction.

Any algorithm to separate bird song sources depending on the direction in space should not be a part of the recognition algorithm (AI model). I think such a separation based on simple physics and can be done by using algorithm approach like beam forming stuff. There are amplitude-phase interrelationships in time or frequency domain, what can help to separate the single bird song sound source. Maybe the classification for the singer source inside the 3D space of a time-frequency-singingPlace could be made by a AI as a special part of the recognition AI, but the preprocessing to get a space information like an angle or some coordinates, can get only by a physics based algorithm.

Best regards, Frank

[kahst]

Yeah I really can't answer this... Patrick put channels in the conf for some reason even though the BirdNET code converts all stereo tracks to mono with mono=True. Does the BirdNET model actually support stereo files? @kahst

No, the only way to meaningfully process stereo files is to run inference for each channel separately and then do some sort of score pooling or post-processing.

[HolgerTs]

I think, processing as stereo makes it more complicated than necessary and does not make sense. If the microphones are located side by side, a down to mono mix before analysing should be o.k.

Another setup (like mine) is: one microphone in front of the house, the second one in the backyard. So from my point of view, it is sufficient to process each channel separately.

My proposal for processing:

• Use arecord as before with the difference of recording a different file name, e.g insert mc (multi channel) in the record name.

arecord -f S16_LE -c${CHANNELS} -r48000 -t wav --max-file-time ${RECORDING_LENGTH} \ 
--use-strftime ${RECS_DIR}/%B-%Y/%d-%A/%F-birdnet-%H:%M:%S_mc.wav 

• split the recorded file into as many wav files as channels were defined:

for ((i = 1; i <= CHANNELS; i++)); do 
	  sox ${base_filename}_mc.wav ${base_filename}_ch$i.wav remix $i
	  done

• analyse each of .._chX.wav file

[fherb2]

"remix 1" means, only first channel is used to create spectrogram pictures.

Its interesting, Holger!

My stereo stream is also evaluated by BirdWeather (Station https://app.birdweather.com/stations/1036). If I monitor the sound and the results of my BirdNetPi against the results of BirdWeather live, so I can see some differences. BirdWeather shows more hits. Maybe Tim Clark from BirdWeather uses another model version from Stefan Kahl. But, maybe, the reason could be the channel / stereo problem.

Maybe the reason to use only one channel by BirdNetPi was some performance problems. But my monitoring of the system performance shows that two calls of the analyze routine should not be a problem with Raspberry Pi 4. In my setup I'm straining my RasPi to record a 18s stream and than the analyzing runs with 3s chunks but by overlapping 2s. That are analyzing steps of one second all over the stream. That are 15 of 3s-chunk analyzes each 18s. And the load of the ARM is not high: After 18s of input data acquisition, the analyzing needs about 3 seconds with a load around 50%. But this load contains also the next acquisition chunk. A stereo analyzing should not be an performance problem.

Best regards,
Frank

[fherb2]

I marked this topic as an issue.

@mcguirepr89:

Maybe this is not a real issue, but a request for a missing function. During the last weeks I was monitoring my stream and the result of my BirdNetPi station. (Since my Locu Sonus Stream, I could hear the sound at my working place in parallel to my daily job. And I could analyze the detecting of BirdNetPi.) Some days I used some time to compare additional the results of BirdWeather by using my stereo stream against the BirdNetPi results. BirdWeather has really more recognition events during the main bird singing time.

I have only such simple monitoring results to can check the BirdNetPi. Maybe, there are more good recognitions in case the BirdNetPi would analyze left and right audio channel separated. Get us this feature, please, Patrick.