Automatic file purging and database growth over time?

[hgoudey]

After several months of detections, my BirdNET-Pi device running on a 32MB SD card filled the card to the point that it no longer functions. I think I read that there is a function for automatic purging of old detection files. Do I have to activate that or is it default behavior? Is inevitable growth of the database over time something I have to manage manually, or is there an automated way to keep it within a memory limit? Obviously, I can use larger storage media, but I feel like I will eventually have the same problem, unless I can reconfigure how it uses disk space to work within a limited amount of storage.
Thanks,
Howdy

[ehpersonal38]

Hi,

BirdNET-Pi automatically removes old audio files that haven't been manually excluded from being purged, once the disk becomes 95% full. You're free to modify this number to your liking by changing it in ~/BirdNET-Pi/scripts/disk_check.sh.

Check Tools -> Settings -> Advanced Settings to make sure your full disk behavior is set to purge, and you won't need to do anything else.

The database file becoming large really isn't a huge concern, on my machine with a few hundred thousand detections it's still under 100MB. You can always copy it over to your local drive and then clear all your data and start anew if it ever becomes a problem.

[oliverk71]

Output of du -hs ~/BirdSongs/*:

16G     /home/oliver/BirdSongs/August-2022
9.8M    /home/oliver/BirdSongs/Extracted
4.0K    /home/oliver/BirdSongs/July-2022
4.0K    /home/oliver/BirdSongs/June-2022
22G     /home/oliver/BirdSongs/November-2022
49G     /home/oliver/BirdSongs/October-2022
360K    /home/oliver/BirdSongs/Processed
17G     /home/oliver/BirdSongs/September-2022

Output of df -h:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       117G  113G     0 100% /
devtmpfs        325M     0  325M   0% /dev
tmpfs           455M     0  455M   0% /dev/shm
tmpfs           182M  824K  182M   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
/dev/mmcblk0p1  253M   31M  222M  13% /boot
tmpfs            91M  4.0K   91M   1% /run/user/1000

[ehpersonal38]

Okay the problem is that the audio files inside of the folders from August-November aren't getting moved to /home/oliver/BirdSongs/Processed after analysis.

Before we go further, are you on the latest version of BirdNET-Pi?

[oliverk71]

Before we go further, are you on the latest version of BirdNET-Pi?

Since I always made update from time to time, I guess I am. Only the last update is not yet installed.
Terminal says I am on 5.15.61-v8+

[ehpersonal38]

That appears to be your Pi kernel version which is different from your BirdNET-Pi installation. To update, navigate to Tools -> System Controls -> Update, as you can see in this screenshot:

[oliverk71]

Got this output:

+ sudo -u oliver git -C /home/oliver/BirdNET-Pi reset --hard
fatal: Could not write new index file.
+ set +x
+ sudo -u oliver git -C /home/oliver/BirdNET-Pi fetch origin main
From https://github.com/mcguirepr89/BirdNET-Pi
 * branch            main       -> FETCH_HEAD
+ set +x
+ sudo -u oliver git -C /home/oliver/BirdNET-Pi switch -C main --track origin/main
error: unable to write file README.md
fatal: unable to write new index file
+ set +x
+ my_dir=/home/oliver/BirdNET-Pi/scripts
+ sudo systemctl stop birdnet_server.service
+ sudo systemctl stop birdnet_recording.service
+ services=(chart_viewer.service spectrogram_viewer.service icecast2.service extraction.service birdnet_recording.service birdnet_log.service)
+ for i in "${services[@]}"
+ sudo systemctl restart chart_viewer.service
+ for i in "${services[@]}"
+ sudo systemctl restart spectrogram_viewer.service
+ for i in "${services[@]}"
+ sudo systemctl restart icecast2.service
+ for i in "${services[@]}"
+ sudo systemctl restart extraction.service
+ for i in "${services[@]}"
+ sudo systemctl restart birdnet_recording.service
+ for i in "${services[@]}"
+ sudo systemctl restart birdnet_log.service
+ sudo systemctl start birdnet_server.service
+ sleep 5
+ for i in {1..5}
+ systemctl is-active --quiet birdnet_server.service
+ grep 5050 /dev/fd/63
++ netstat -tulpn
+ sleep 5
+ for i in {1..5}
+ systemctl is-active --quiet birdnet_server.service
+ grep 5050 /dev/fd/63
++ netstat -tulpn
+ sleep 5
+ for i in {1..5}
+ systemctl is-active --quiet birdnet_server.service
+ grep 5050 /dev/fd/63
++ netstat -tulpn
+ sleep 5
+ for i in {1..5}
+ systemctl is-active --quiet birdnet_server.service
+ grep 5050 /dev/fd/63
++ netstat -tulpn
tcp        0      0 127.0.0.1:5050          0.0.0.0:*               LISTEN      3735606/python3     
+ logger '[/usr/local/bin/restart_services.sh] birdnet_server.service is running'
+ break
+ systemctl is-active --quiet birdnet_server.service
+ grep 5050 /dev/fd/63
++ netstat -tulpn
tcp        0      0 127.0.0.1:5050          0.0.0.0:*               LISTEN      3735606/python3     
+ status=0
+ ((  status != 0  ))
+ sudo systemctl restart birdnet_analysis.service

Then I removed the folder of June, July, September, October using rm -r <folder>

Now, I will try updating, again.

[oliverk71]

One update installed. Output:

+ sudo -u oliver git -C /home/oliver/BirdNET-Pi reset --hard
HEAD is now at 5767409 Update weekly_report.php
+ set +x
+ sudo -u oliver git -C /home/oliver/BirdNET-Pi fetch origin main
From https://github.com/mcguirepr89/BirdNET-Pi
 * branch            main       -> FETCH_HEAD
+ set +x
+ sudo -u oliver git -C /home/oliver/BirdNET-Pi switch -C main --track origin/main
Reset branch 'main'
Branch 'main' set up to track remote branch 'main' from 'origin'.
Your branch is up to date with 'origin/main'.
+ set +x
+ my_dir=/home/oliver/BirdNET-Pi/scripts
+ sudo systemctl stop birdnet_server.service
+ sudo systemctl stop birdnet_recording.service
+ services=(chart_viewer.service spectrogram_viewer.service icecast2.service extraction.service birdnet_recording.service birdnet_log.service)
+ for i in "${services[@]}"
+ sudo systemctl restart chart_viewer.service
+ for i in "${services[@]}"
+ sudo systemctl restart spectrogram_viewer.service
+ for i in "${services[@]}"
+ sudo systemctl restart icecast2.service
+ for i in "${services[@]}"
+ sudo systemctl restart extraction.service
+ for i in "${services[@]}"
+ sudo systemctl restart birdnet_recording.service
+ for i in "${services[@]}"
+ sudo systemctl restart birdnet_log.service
+ sudo systemctl start birdnet_server.service
+ sleep 5
+ for i in {1..5}
+ systemctl is-active --quiet birdnet_server.service
+ grep 5050 /dev/fd/63
++ netstat -tulpn
+ sleep 5
+ for i in {1..5}
+ systemctl is-active --quiet birdnet_server.service
+ grep 5050 /dev/fd/63
++ netstat -tulpn
+ sleep 5
+ for i in {1..5}
+ systemctl is-active --quiet birdnet_server.service
+ grep 5050 /dev/fd/63
++ netstat -tulpn
+ sleep 5
+ for i in {1..5}
+ systemctl is-active --quiet birdnet_server.service
+ grep 5050 /dev/fd/63
++ netstat -tulpn
+ sleep 5
+ for i in {1..5}
+ systemctl is-active --quiet birdnet_server.service
+ grep 5050 /dev/fd/63
++ netstat -tulpn
tcp        0      0 127.0.0.1:5050          0.0.0.0:*               LISTEN      3743499/python3     
+ logger '[/usr/local/bin/restart_services.sh] birdnet_server.service is running'
+ break
+ systemctl is-active --quiet birdnet_server.service
+ grep 5050 /dev/fd/63
++ netstat -tulpn
tcp        0      0 127.0.0.1:5050          0.0.0.0:*               LISTEN      3743499/python3     
+ status=0
+ ((  status != 0  ))
+ sudo systemctl restart birdnet_analysis.service

[ehpersonal38]

Great, you'll be able to see if the update fixed the issue by checking to make sure there aren't a lot of files here:

ls ~/BirdSongs/November-2022/06-Sunday

[oliverk71]

Yes, it seems so! Great!

birdnetpi-elsen:~/BirdSongs$ ls ~/BirdSongs/November-2022/06-Sunday
2022-11-06-birdnet-22:24:08.wav  2022-11-06-birdnet-22:24:37.wav

But why? The last commit was just an update of README.MD

[ehpersonal38]

The update includes all the commits we've made since your system was last updated. So it really depends on when that was. A LOT has changed since April-ish, probably hundreds of commits since then.

[oliverk71]

Btw: Last chart is from 2nd of October.

Now I have many non playable recording under 'recordings' because the files are gone. Is there an easy way to fix it?

[ehpersonal38]

Btw: Last chart is from 2nd of October.

What's the output of: sudo systemctl status chart_viewer.service

Now I have many non playable recording under 'recordings' because the files are gone. Is there an easy way to fix it?

O_o ... how? Recordings only shows detections that have actual files, otherwise it just skips showing that one. Can I see a screenshot?

[oliverk71]

True, I meant under 'best recordings', not 'recordings':

[oliverk71]

Sorry, I missed the first question somehow. Here is the output. Could not copy it from web terminal, so here is the crop of the screenshot:

[ehpersonal38]

Hmm yeah I don't know the best way to fix the Best Recordings thing, I have it on my system as well. You could: edit the database and lower the confidence of your detections that are > 0.95 confidence, in hopes that you will eventually get a new, higher confidence detection that will replace it.

UPDATE detections SET confidence = 0.95 WHERE confidence > 0.95

It's not great but that's the best solution I can think of right now.

Also in regards to the chart problem, what's the output of: ls -lh /home/pi/BirdSongs/Extracted/Charts ?

[oliverk71]

In the charts folder there are files from June to today. Right now, I can only take a screenshot, but the list is longer. Here is the screenshot: