Testing of Verification Branch

[CaiusX]

Hi All

For THE BRAVE (@mcguirepr89 , @ehpersonal38, @DD4WH @nilspupils ) I've eventually updated the verification branch to actually do some verifications (plus some other goodies) and need ALPHA TESTERS.

There's a fair amount that a tester needs to prepare behind the scenes

1. Make sure the birdnet python virtual environment has the latest version of Streamlit
2. Ensure that the following other packages are installed (pip install into the birdnet python virtual environment)

• wikipedia
• streamlit_aggrid

3. Run sun.py (which will hopefully install its required packages- otherwise look at the error messages and install astral, tzlocal etc as required) which will hopefully produce a daylight table birds.db for your location.
4. MAKE A BACKUP COPY OF BIRDS DB
5. Update birds.db to add Rating and Manual_ID fields with a default for Manual_ID of Not_Reviewed
   
6. Restart the system.
7. Hope like hell it works.
8. Test it and report errors/improvements.

If it works to a reasonable degree, automate the above in bash and put it into the wild......

Some screenshots of what can be expected:

The main System Overview file is still called plotly_streamlit.py and appears in the sidebar. This can be changed to "Overview.py" if the php script is updated?

Selecting a specie in the dropdown gives a specie overview

with selectable dates and subsampling and a wiki-pic of the fowl.

Selecting Multi Day View gives

with sunrise/sunset/dusk/dawn so you can see the sleeping habits of the specie

Selecting Sample Review gives

where you can select a sample, listen to it, listen to other recordings on Xeno-Canto....

and then mark it as a true positive/false positive, and if a false positive, give it the correct specie or other noise or unidentified to send to your friends to identify.

The sample can also be given a rating.

The data on the right - can be filtered/sorted by clicking on the column headers, and samples for review selected based on confidence etc.

The single day view is as per the original African Penguin daily phenology chart, but with the added bonus of seeing as many species as you want -

pop it out if too small with the double arrows if too small

The Weather tab does nothing yet..........

Looking forward to y'all feedback.

Best

AP

[DD4WH]

AP, that looks very promising and I can't wait to test 😀
However, it seems my python knowledge is too poor to do so.
I think I can manage steps 3 to 7. But could you explain what I have to do in steps 1 and 2 ? Sorry for the trouble!

[CaiusX]

Hi @DD4WH

Maybe leave it to the people who can put it back together again if its broken after the update (@mcguirepr89 @ehpersonal38 ), but if you are brave and have your system backed up:

1. go to terminal
2. switch to the verification branch

• ~/BirdNET-Pi/$ git fetch
• ~/BirdNET-Pi/$ git branch -r
• ~/BirdNET-Pi/$ git checkout verification

3. get python virtual environment going & install required packages

   • ~/BirdNET-Pi$ source birdnet/bin/activate
   • (birdnet) birdpi1:~/BirdNET-Pi$ pip install --upgrade streamlit
   • (birdnet) birdpi1:~/BirdNET-Pi$ pip install wikipedia streamlit_aggrid astral tzlocal

4. Get the sunrise/sunset data for your location: go to the scripts directory and run

   • (birdnet) birdpi1:~/BirdNET-Pi$ cd scripts
   • (birdnet) birdpi1:~/BirdNET-Pi$ python sun.py

then follow the rest of the steps above.

Hope it works!!

Best AP

[Vertikar]

Got it sorted, I think as the default installer is limited to cloning shallow, it wasn't able to grab the verification branch.
Ran this:

git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*

and that fixed it up.

The default limits it to main

remote.origin.fetch=+refs/heads/main:refs/remotes/origin/main

[CaiusX]

Hi @Vertikar

Very pleased you got it going, not sure why the repo is shallow cloning? (@ehpersonal38 , @mcguirepr89).

I look forward to your feedback. I am close to pushing a big update, but thought there was little rush due to a lack of interest from users - this will inspire me :).

Best
AP

[Vertikar]

Hey @CaiusX Definitely enjoying using it, but I've flicked back for now as I was missing my weekly reports.

I think some of the notifications for infrequent birds may have been missing too. Haven't had a chance to dive and try and debug, but do want to come back to it again soon.

One idea, is to maybe look to actually change the name of the bird when it has been manually ID, and just mark it as manually reviewed.
Guessing that'd require renaming of the file also.

[nilspupils]

Sorry i couldn't test so far since i don't have a spare system availeable. It is not a lack of interest but a lack of resources. I hope to be able to do so soon.

[Vertikar]

I'm now running on my own fork, with weekly reports seemingly working successfully.
trying to bring the verification branch back up to date with the main branch, but I focused on the weekly reports first.

[DD4WH]

@CaiusX AP,
Thanks a lot for the excellent instructions!
With your help I could manage all the necessary steps with ease!
First observations:

• Works perfectly !
• the new Stats page is awesome and self-explanatory, very well done!
• I really like the new year overview graph
• species ID review: very good feature, here are some small issues with this new feature:
• if I review several recordings of the same species, the button "Commit to DB" gets duplicated, sometimes I could see 5 buttons.
• it would be good, if the recording list would propose the recordings sorted by highest confidence, not by date
• may I propose to rename "Current ID" into "Manual ID"
• Rename "What species?" Into "Manual ID: "
• it's a little irritating, that the Verification is already ticked as "True positive" or "False Positive ", even before I have reviewed the recording
• a nice feature would be a simple species list of all the manually verified species
• Its a good thing that I can review a species ID in the Recordings menu also! But I fail to save my manual ID, it remains as Not Reviewed in the spectrogram

These are my first impressions, will test more over the weekend.

Excellent work, thanks a lot, AP !!!

[DD4WH]

Additonal observations:

• Spectrogram on a large monitor only gives the static jpg, not the running spectrogram. Also on small screens like my smartphone. --> Hmm, it seems the ffmpeg and icecast2 services were down. A restart of the services brought them to function again! However, this is something I also observe with the version in the main branch !!! it does not seem to be specific with the verification branch
• images for some species in species stats will not load ("cant access"), maybe search by scientific name?
• the possibility to listen to Xeno-Canto is suberb!
• on the left side, where I can choose the four options: may I suggest to rename those options?

1. Quick analysis (overview & phenology)
2. Longterm graph (Year)
3. Manual ID review
4. Daily Charts

• BTW: if I choose option 2, there is some magic code appearing (below the graph, if you scroll down): is that intended ? Looks like this:
  

[CaiusX]

Hi @DD4WH

Thanks for your thoughtful comments as always Frank - I will implement your suggestions where capable.

Which species images don't come up? I've noticed that those with an apostrophe (e.g. Barrat's Warbler) get parsed incorrectly. I'll give scientific name a bash as that shouldn't have that problem.

I assume the daylight data downloaded OK for your location? The next step is to use the same logic to download weather (temp, humidity, rainfall etc) on an hourly basis and produce similar "overlays" so we can see what species don't talk in the rain or when its too cold....

Other tasks on my list are:

• Learn more about caching and statefulness to get faster performance and "memory" on the reviewed species table, so that one can filter/sort and it will maintain the view after review update with the process operating faster - currently it re-opens the database, writes to it, and reloads again twice to the default view (most recent detection) - quite annoying on a big database. If I get it right the table will remain with the existing filter/sort (eg on confidence) at the latest sample reviewed.
• Probably password protect the Manual_ID form so that only authorized users can update
• Put additional data/info on each page where appropriate - e.g. last 5 detections on the main Overview page, reviewed species summary as per your suggestion maybe with a confusion matrix for each specie
• Incorporate the "confidence" color coding, and be able to filter on confidence on each page
• Do something with the Manual_ID's, maybe have an option to show only reviewed species on the charts
• ?????

I doubt I've messed with the live spectrogram, I only upload the existing static images - I don't touch the bash/php side, leave it up to the wizards @mcguirepr89 and @ehpersonal38.

Looking forward to further comments.

Best
AP

[DD4WH]

• Yes, daylight data did donwload without problems
• Species that did not come up as images: Elster, Hausrotschwanz
• Species that did come up: Haussperling, Nebelkrähe, Mauersegler
• Yes, password-protection for Manual ID review is important (also for review from the Recording view)
• not sure, if we need more info on the pages, they seem very well organized at the moment. Confusion matrix only makes sense, if all detections have been reviewed, I doubt anybody will do that. From my personal point of view the manual ID review is important to have at least ONE manually identified detection per species. Only then can we output a list of verified species.
• And that leads to your next suggestion: Yes, very much appreciated would be an option to only display reviewed species on the charts. Even better would be the option to simply output a species list of all the manually verified (identified) species. Not sure if we need the "only reviewed species" option for all the charts . . . I guess that would not be very informative, if only verified detections would be taken for the charts.
• Yes, I am pretty sure the ffmpeg and icecast2 pauses have been there before you implemented the changes.

Dennis, Thank you very much for this major improvement of the software !!! Keep up the good work!

[CaiusX]

hi @DD4WH & other alpha testers.

I have updated the streamlit suite as per some of Frank's suggestions.

If you have jumped through the hoops in #470 (comment) above, simply pulling the verification branch should run OK.

I couldn't implement renaming plotly_streamlit.py to Quick_analysis_(overview_&_phenology).py because it needs to updated in the bash calling service and I don't want to mess with things I don't understand.

Duplicated buttons don't happen on my laptop but do on my tablet, so it may be a browser caching issue

Wikipedia now looks for scientific name

Option to filter review samples and sort by Confidence implemented

Preliminary summary table of Manual_ID implemented - suggestions on better presentation required.

I'll continue improving this, bugs and improvement suggestions are encouraged!!

Best
AP

[DD4WH]

Thanks Dennis, for this quick update !!! Here my first observations on the new version:

• Xeno-Canto is unavailable most of the time, only when I click on "Commit to database", it appears for a short time and then vanishes again (I can not click on it, because it greys out so quickly)
• Sorting by confidence works very well and makes things so much easier with manual ID --> thanks for that !!!
• Wikipedia photographs now are found for every species I have tested !
• in "plotly_streamlit", the number of false and true positives shown is not correct, even when I reload the page
• the review summary table is OK. But is it really interesting to see "False Positives" ? What I would like to see is a table with only the species with at least one true positive. Even better would be a table of all the species manually identified. It could theoretically be (however not very likeley, but not impossible) that a species has never been detected by BirdNET, but is identified manually from a recording. That is a false positive for the species BirdNET assigned, but is a "Manual ID" for the manually identified species. So we need another category.
• May I suggest that we use the field "Manual ID" for this ? An overview table of all manually identified species should show only those species that contain a species name in that field and not any of the other categories "baby", "noise", "wind", "Unknown", "Not_Reviewed" etc.
• That overview table is so important in its information content that -in my opinion- we should have a fifth category on the left side in the "Species Stats" for this table. What do you think?
• bug ? Rating showing NULL vs. None
  

Thank you very much for your hard work! BirdNET-Pi is now such a valuable tool, it is incredible !

[ehpersonal38]

Just tested this out and it looks very good! Great to see all this functionality coming to BirdNET-Pi. Thanks for all your hard work!

I would like some sort of way to speed up the manual verification though. I can accurately determine whether a species is a true or false positive in the time it takes me to listen to the recording, and often times almost instantly, as I've come to recognize the way spectrograms look for different bird species. The flow to actually fill out the form, wait for "Commit to DB" to work, and load a new recording takes something like 15-20 seconds though. That's a lot of time wasted and puts me off using the feature at all.

I guess I'm thinking something like this? Where we can batch identify recordings and then have one "Commit to DB" at the end.

This will be really useful for something I'm thinking of, allowing the user to export manually verified detections into a CSV for importing into eBird checklists.

[willkuhn]

Hey all, I'm late to this party, but I've recently become a BirdNET-Pi fanatic and I think verification would be a great addition to the platform. I have a BirdNET-Pi setup in my Knoxville, TN backyard and it's detected lots of great finds, like screech owls in the middle of the night! I work for a nonprofit that does biodiversity research in Great Smoky Mountains National Park and we're about to pilot a couple of devices there, as well.

As y'all know, most BirdNET detections are excellent, and some are very bad. It would be nice to be able to review them and tag the ones I'm confident in as True Positives and those that aren't correct as False Positives. Data from the True Positives would be very useful as species presence datapoints, and False Positives could be useful as well.

I recently came across Cole et al. 2022, which showed promising results using different confidence thresholds for each species (as @raybellis suggests in #516 ) based on the highest confidence on a False Positive of that species. That's something I'd like to try, but I'd need those False Positives.

So I've just started playing around with the features in the verification branch and like what I see! I wanted to share some initial feedback and musings and plan to keep playing with it.

Species Stats: Manual ID Review B

• Xeno-Canto Comparison feature is super useful and it's working great for me!
• side by side comparison option is also very useful
• I think the order of the Rating slider options in the rating = st.select_slider() function needs to be reversed, so options=["NULL", "*", "**", "***", "****", "*****"].
  Also, the max excellent rating in the help attribute should be 5 stars: help="Select your rating for the sample quality - *=bad; *****=excellent"

Species Stats: Manual ID Review

• same suggestions for the options and help variables
• value should be "NULL"

Review Sample Selection table:

• needs some kind of pagination or load-as-you-scroll for this table. I have 1000s of Carolina Wren observations and it wants to list them all
• it would be nice to be able to check multiple filename checkboxes at once and review them all together.

Recordings view

• love the idea of an Edit button on each recording
• it would be nice if the Manual ID field would autocomplete with BirdNET labels, also there's no way to back out of the Edit menu without hitting "Submit"
• I like the "Manually Verified" filter checkbox at the top, but it doesn't seem to be working
• I would like to be able to add/edit Rating and Manual ID in this view as well

General thoughts

• I personally don't find the star rating system very useful. I could see utility in it for selecting future training data or recordings to submit to the likes of the Macaulay Library, but that seems like a pretty niche use case. Maybe there's another reason users might like it?
• To me, it's more important to be able to "ground-truth" detections as True of False Positives
• BirdWeather's simple "thumb up" or "thumb down" approach seems like a simple and elegant approach that could be placed in the top-right corner of each recording in the Recordings view (like in BirdWeather), though I do really like the ability to manually relabel the recording as well...so maybe "thumb up", "thumb down", plus your "edit" symbol that brings up the manual relabel window?
• it would also be great if recordings marked as False Positives would get demoted somehow, for example falling the end end of the list in the confidence-ordered Recordings view

I hope this is helpful feedback. Thanks to @mcguirepr89 @ehpersonal38 @DD4WH @nilspupils @CaiusX and others that have contributed to this project! It's well-designed, functional and surprisingly seamless to install. Well done!

--Will--