Updated Species Stats

[CaiusX]

Hi All

There's a new (and hopefully improved) Species Stats Streamlit page.

Changes are:

• Ability to report species on resampled time periods (e.g. if you select 10 minutes & a species had multiple detections in a 10 minute period it will log only one detection). This should (hopefully) improve response time and reduce the impact of overtly noisy individuals).
• New Daily Species plot, which plots the detections on 15 minute intervals throughout the day, with color reflecting how many detections in that 15 minute period (thanks to @DD4WH for this idea).

The Daily plot will only become meaningful when there are many days data, from my NatureStation (note will take quite a long time on first load), you can start seeing the seasonality (days getting shorter as we head to winter here).

Please give feedback & suggestions for improvement.

Things on my list:

Include Dusk/Dawn, Sunrise/Sunset lines
Temperature data
Rainfall/weather data

Best
AP

[DD4WH]

@CaiusX AP
Dennis, this is so awesome!!!!
Just installed the new version and playing around with it . . . looks very good, indeed!
Will comment in the next days, I need a few more moments to think about improvements ;-).

And: I will need a way how to combine my data from different versions of BirdNET-Pi (MariaDB & SQLite . . .), so I can really display data from a few months . . .

[DD4WH]

what is definitely needed is:

• Species Stats --> Daily diagram --> a slider that the user can shift the color gradient at the right side of the diagram: in my diagrams the color is a bit too dark in many cases

[CaiusX]

Thanks for the feedback @DD4WH , I'm looking forward to more ideas!!

In order to speed things up and get reasonable smoothing I think we should agree on a "standard resample time", maybe 15 minutes, and then create a separate DB table for that?

I'll put some code together where one can get one's own color gradient and then users can suggest best aesthetics .

I find the best way to combine databases is to export from MariaDB, SQLite etc to csv files, combine in a spreadsheet (or VIM if you're @mcguirepr89 and his ilk), export again as a csv and read back into SQLite. Should be a once off/infrequent process so doesn't need to be automated.

Best

[DD4WH]

AP, thanks for your quick reply!

• from my point of view, there is no need to put integrated data information into the database. The detections are the base data and everything else can be created on-the-fly from that. Of course, you are right, it will take some time (maybe several minutes with large data sets), but I would like to keep the database "clean" and only contain the raw data, not some integrated data
• my wish for a slider was not for a different color scheme, my comment was not clear enough: I would like the same color scheme, but a way to add some "gain" or a multiplication factor to the data, so that I can emphasize small or large values in the graph.

Thanks for the hint on combining database data! I am not at all familiar with MariaDB/SQLite format and also I am confused about which file I have to choose, the txt file or the *.db file. But this is offtopic here. I will eventually open up a new thread for that. And - Yes - you are right, that is an infrequent process without the need for any automation. All that would be needed is some lines in the Wiki describing what is needed and how to squeeze csv files back into a SQLite file.

[CaiusX]

@DD4WH Frank, thanks for the pointers;

With regard to including "contrast/gain" etc. that would be digging too far into the Plotly API than I think worth it unless I can find some code that someone else has done.

I've put an updated version onto the main branch where the user can select the colour pallet from the Plotly built-ins - probably best you play around, select a scheme that works for you, and report back which you like best so we can make that a default.

My favourites at the moment are "deep" and "algae" - keeping somewhat within the BirdNET-Pi colour schemes.

[srchild]

This looks a great way to visualise the habits of different birds - they have different patterns of activity at different times of day, and variations in activity from day to day (and season to season).

One thing that strikes me as "wrong", though it may just be my personal aesthetics, is the direction of time on the Y-axis.

If it was a graph then you might show increasing time as you ascend the Y-axis, in the way that you have plotted it. But really it is a presentational chart rather than a graph, and in that situation it seems more natural to invert it, so that time progresses as you read down the page. We read from top to bottom (as well as left to right), and when reading this chart from top to bottom time is going backwards which confuses me!

Great work

[DD4WH]

Thanks AP for your great work and sorry for my late reply!!!

• color scheme for me is difficult to choose. Many of them seem well suited. High contrast seems to be present with "turbid", so that could be the default, maybe ?

• time on the x-axis proceeds from left to right (date) as in all common graphs

• time on the y-axis is a little more difficult. @srchild: You are right that there is no obvious reason to do it either way.

• If I look at the scientific literature, it seems scientists more often use time proceeding from left to right AND from bottom to top, e.g. https://www.researchgate.net/profile/Reinhold-Hill-2/publication/313239319_Migration_phenology_and_behaviour_of_bats_at_a_research_platform_in_the_south-eastern_North_Sea/links/58946133aca27231daf7ba1d/Migration-phenology-and-behaviour-of-bats-at-a-research-platform-in-the-south-eastern-North-Sea.pdf

• https://link.springer.com/article/10.1186/s12862-022-01962-6

• but sometimes they even swap x and y axes :-), see this paper:
  https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0193345

Personally I would prefer time proceed from left to right AND from bottom to top. BTW: this is also how evolutionary time proceeds in phylogenetic trees (from root to tip of the branches) . . .

[srchild]

These further comments in the interest of clarity and exploration and not to push my own view!

@DD4WH It is interesting that all your links above are showing years on the Y-axis. For me, most recent at the top works ok when Y-axis is for year - as I read down the page I am looking back in history.

With time on the Y-axis rather than year (as on Birdnet-pi species stats page) I think it is different. That isn't looking back in history as I read down from row to row. Rather it is showing features of an individual hour of the day. To look back in time I would read from right to left on the X-axis.

I think with day/date on the X-axis and time on the Y-axis it is more like a calendar, or a programme of events. Does the calendar on your phone show 11pm near the top of the screen and 3am near the bottom of the screen? Does a conference programme of events show the afternoon sessions at the top of a page and the morning sessions at the bottom of the page? Might reading a species stats daily plot "read" more naturally if it was presented more like a programme of events (bird song events), with morning chorus to be found near the top of the plot and daytime/evening activity lower down?

[srchild]

Here's a fun example for @DD4WH

Scroll down this page and look at the chart of your contributions in the last year i.e. your "songs" charted by week / day of week:

https://github.com/DD4WH

Would you expect the Y-axis on that chart to be

Saturday
Friday
Thursday
Wednesday
Tuesday
Monday
Sunday

Or as it is, the reverse of that?

[DD4WH]

@srchild Thanks for your input and your contribution to the discussion!

As I said in my older comment, an agreed standard way to graphically display time axes does not seem to exist.

For me personally, it seems more logical and readable to have x and y axes to proceed from 0 to something positive. That implies that if time is displayed on an axis (x or y or both), time proceeds from left to right and from bottom to top. Many (but not all) scientists seem to do it this way and publish in that way.

But it is perfectly OK to do it the other way round.

And the great thing in BirdNET-Pi is, that the code is open source and that AP (thanks @CaiusX !) has provided a means to easily change the way the data is presented. So, everyone can build his/her own data representation for everyones´ needs and personal preferences.

[CaiusX]

@srchild

Thanks for the discussion - next one is on color pallets and additional functionality.

I am going to keep the default for this as is because in my mind the sun rises at the bottom of the chart and sets at the top :) but feel free to change yours around as indicated.

[srchild]

Two additional thoughts, on the bird/period selector.

Currently it defaults to 1 minute view, which is the slowest option to load. Might it be better to default to a more responsive option such as 10minutes or hourly? The view would load more quickly, and then load quickly again on selection of a different bird. The 1/5 minute options could still be offered for someone who wants more detail.

It seems that the dropdown selector starts on the bird with the most recordings. That makes sense but creates some anomalies.

For example, for me, it defaults to loading 1minute data on Blackbird. If I change the selector to 10minutes then it shows reed bunting data instead, because reed bunting are represented more in the 10 minute data than the blackbird which leads the one minute data. (Reed bunting song is sustained throughout the day whereas my blackbird is very busy in the mornings but less so later in the day).

I think I would expect that changing the period radio button would keep the same bird selected. So far as I can see the bird selection is preserved when changing between 5/10/hourly options, but selecting either 1minute or daily views resets the bird selected to the one with most records for that aggregation

Thanks.

[CaiusX]

Hi @srchild, @DD4WH and all the gang

Thanks for the feedback.

I've done some work on this and uploaded to a separate species_stats branch if you want to give it a try, or look at NATURESTATION.

Some of the changes

• Eliminated 1 minute interval, but included RAW data. The 1 min interval was taking ages because of an enormous amount of redundant up sampling.
• Included 15min, 1 hour and daily.
• Included Single Day stats, as per the Overview and Daily Charts but with the facility to resample and to have user selected number of species to show (needs some aesthetic improvement). Suggestions what else to put here welcomed, maybe replicate the info on the current Overview page?
• Made the DAILY page a single chart, which can be viewed in full-screen quite nicely. This needs a title and axis labels. Any ideas on what else to put on this page, it looks a bit sparse - maybe the species photographs/spectrogram/sound files?
• Defaulting on species selection to the most common RAW species for the selected time period.
• Tried to get color theming to the "BirdNET-Pi Green"

I'm diving down various rabbit holes trying to get as much info onto a single app (without the facility in Streamlit for separate tabs).

Further comments are welcomed.

Best
AP

[mcguirepr89]

I love this iteration a LOT 😍