Long list of possible improvements

[miboka]

After trying Chirpity for a few days, I have to say I am deeply impressed! Please continue to develop this app, it has huge potential.

A few things that I like in particular are that you included the two different models, and that your own Chirpity model distinguishes between calls and songs - calls are a common cause for false-positives in BirdNET. I like the retrospective adjustments (species filters, confidence) and other filter options and ebird export. The processing speed for the Chirpity model is remarkable!

Still, I found a few things that I, from my personal view, would love to see improved with Chirpity. I have been a prolific user of BirdNet (via the command line) and validate BirdNet detections in Kaleidoscope Pro (csv conversion with a script to load detections into Kaleidoscope). This is a workflow that has been somewhat cumbersome on one hand (audio file conversion to wav, csv conversion), but also very efficient on the other hand (validation process itself is very fast in Kaleidoscope). I have processed TBs of soundscape audio in this way. This is just to give you some background on myself.

This notes and wishes list is very long, I apologise. Some things that I noted below may be inaccurate (not tested long enough) or are in fact already present. Some things should be easy to implement, some would certainly require a lot of work and others will be outright rejected by you, which is of course totally fine. But it makes no sense to just keep the list for myself to dream on, so here I go:

Model, settings

• Please include some missing central-eastern European birds in the native Chirpity model: examples are River Warbler, Thrush Nightingale, Tengmalms Owl, Hazel Grouse and Great Reed Warbler (there may be more). These are all not particularly rare in central-eastern Europe. It would be nice to get to a model that includes all (central) European birds. I would love to work with the Chirpity model more than I did so far, but the species inventory needs to be more complete. It is also a pity that we cannot easily use both models simultaneously in the app.

• Detections classes for Ballions Crake and Red-brested Flycatcher (song) could be improved, if possible. Frogs and Spotted Flycatcher produce huge numbers of false positive detections for these.

• Using BirdNet a lot, I know that the sensitivity setting can be important. I found no way to change this setting for the BirdNet model within Chirpity, nor is there any hint of what sensitivity setting is being used for the analysis? The sensitivity cannot be changed after the analysis is done (unlike species or confidence filtering of the results), so it is important to set it before the analysis starts.

• Can the 'overlap' option from BirdNet be combined with the 'context' setting that Chirpity offers? (how does the context option work, anyway?)

Appearance, usability

• for me (personally) the font size could be reduced a little bit to make more room.

• how about a freely resizeable spectrogram window? (make the horizontal split to the species and detections list flexible up-down?)

• Some keyboard shortcuts are 'hard-wired' to the english/us keyboard layout. I noted the + and - adjustments for the spectrogram

• very helpful (for me): I really like to pan left and right in the spectrogram with a two-finger swipe on the trackpad when the pointer is over the spectrogram. Audacity does this and the spectrogram in the 'compare with reference calls' pup-up window works like this as well. Only the main spectrogram in Chirpity doesn't, and in fact there seems no way to pan left or right other than letting it play beyond the screen or using arrow keys a lot. Alternatively, a double-tap on the spectrogram could recenter on the click position.

• show frequency indications on spectrogram (the video and manual shows overlaps on left side of the spectrogram, but I don't see these?

• very helpful (for me): make the displayed frequency in the spectrogram adjustable. When validating detections, it is extremely helpful to limit the displayed frequency in the spectrogram to the relevant species' song or call. You could even automate this to some degree, i.e. automatically zoom the spectrogram to the relevant frequency range of the current bird species, for example zoom to 100-1000Hz for booming Bitterns or zoom in on high frequencies for kinglets and 'zip' calls.

• essential (for me): For fast validation it is extremely helpful to have keyboard shortcuts for preset labels. numbers 1-9 (or any other keys) should be available to assign self-defined labels in the settings menu. When reviewing/validating detections, a keypress puts the corresponding label into the label field and Chirpity then automatically jumps to the next detection. This is how several validating softwares for bats do it, and it works reeeally fast. Just as an example, these labels could be 'reviewed: id correct', 'reviewed: id rejected', 'reviewed: id uncertain', 'reviewed: id correct (playback!)'; or a simple quality rating of 1 to 5. This method could also be used to make super-fast species changes with a single keypress (flexible database field that gets updated with a keypress?).

• 'command-v' marks a detection as 'validated' (ie reviewed & correct). But I can't find a way to undo this? On the lower left side (species overview) the checkmark could be accompanied by the number of validated detections of that species. Validation overwrites the confidence number in the csv export. I am not sure is this is the perfect way longer term (see below).

• allow multi-selection of detections to edit them at once. I can edit a single detection, or all detections of a species (by right-clicking on the species summary), but not a selected subset of, say, 30 of 120 detections of a species.

• very helpful (for me): option to show all detections (of all species) on spectrogram, all detections of the selected species, or just the one selected detection.

Database, data organisation, filtering:
Unlike for bats, the passive acoustic monitoring world for birds still lacks a 'Lightroom', ie a comprehensive app with extended database functionality for audio files, various model detections, metadata, and validations.

• To me, it is not always clear, if I am looking at a current (new, unsaved) analysis of files, or if I am seeing (old) detections from the archive database. Also, when I open files, it would be nice to have an option to automatically retrieve the saved detections if there are any in the archive database and analyse remaining files if there are no detections in the archive db. Saving detection data to archive could overwrite valuable validation data if these are present in the archive? is there a check for that? (e.g. doing a chirpity analysis of a number of files, and then a birdnet analysis on the same files will overwrite the chirpity validations?) Is it possible to keep both birdnet and chirpity model detections for a file in the archive? Switching to 'explore archive' or 'view charts' seems to discard current (unsaved) detections/validations?

• Metadata window popup when opening files or folders: location, project, and recording equipment (device manufacturer/type, device number, microphone type and number), the recording settings and scheme are the bare minimum I can think of, which should be entered for a dataset of audio files before analysis. Support user tags for detections, audio files, folders or projects. Support the wav formats that include metadata (wamd and guano). There are python libraries that help with this iirc.

• Allow a separate database file for a project instead of adding everything into a main archive db. Ability to open an existing database files. Split database files (i.e. save a filtered subset) and merge database files.

• show the number of files and folders that are open/in the database and filtered for display. show the current project name, active database name and db file location. When opening multiple folders, make them selectable/filter option.

• support imports from Raven and Kaleidoscope or BirdNet csv result files.

• Enable detections of the two models (or several runs of the same model, but with different settings) in the database. Include the detections of both models in the detections lists & show them at the same time in the spectrogram (maybe in different color shading?). It would be very helpful to have the results of both models shown in the list and spectrogram, as they do produce somewhat different results, detect different species and sometimes one is significantly better than the other.

• greatly enhanced filter options: Time of day, date, folders, locations, projects, any metadata or tags, model type, species Ids, confidence, validated or not, etc... (I am aware that some of this is present already. It would be nice to have all filters in one place)

• keep track of validations, species changes and deleted detections during the validation process. The database could still remember the original detection (with all the data). Medium- to longer-term, such data could be massively useful to determine real-world efficiency, precision and recall rates and confusion species to improve the model. Give the user an option to share the validation data (or even recordings) with you, so you can improve the Chirpity model.

Evaluation, visualisation, reporting

• more detailed time scale (not only whole year/ months, but zoom in on the timeframe of selected / filtered detections). Show recording times, night/daylight and plot the detections.

• show a map with recording locations and overlays for each location, visualising the selected / filtered detections (this could be for example species diversity at a location or avg. number of detections per day for a certain species at each location, etc.)

• provide summary tables (maybe flexible, like pivot tables in excel-type programs)

[miboka]

• The species lists languages are sometimes inconsistent: Names in the lists are shown in German, but when editing a detection, the selection list is in English.
  I believe this could have something to do with my custom species list and sometimes it suddenly works right again - inconsistent, I don't know why. Proceeding to change the species for a detection then messes up lists (everything disappears) until species filter is pressed/toggled. The main Problem (different languages) usually persists, though. I also get an error message when toggling through the species filter and hitting the custom list: 'no detections found using the custom list' - and then it goes on to show all the detections with the custom list...

[Mattk70]

This is likely because you exported / created a custom list in English. If your language is German, the custom list should use German labels. I've found the bug that causes the 'no detections found using the custom list' notice and fixed that today.

[Mattk70]

After trying Chirpity for a few days, I have to say I am deeply impressed! Please continue to develop this app, it has huge potential.

Thank you. This is a long list! However, it's a well-considered one, so thanks for putting it together! I'll comment below on some of your requests...

Model, settings

• Please include some missing central-eastern European birds in the native Chirpity model: examples are River Warbler, Thrush Nightingale, Tengmalms Owl, Hazel Grouse and Great Reed Warbler (there may be more). These are all not particularly rare in central-eastern Europe. It would be nice to get to a model that includes all (central) European birds. I would love to work with the Chirpity model more than I did so far, but the species inventory needs to be more complete. It is also a pity that we cannot easily use both models simultaneously in the app.

Happy to do this. It would be good to get a complete list since training is expensive and piecemeal additions a) require a full retrain each time, and b) the way the current archive is designed, if you add a new species, it will generate a new archive, and a migration of any existing archive would be necessary to pull in your records from previous chirpity versions.

• Detections classes for Ballions Crake and Red-brested Flycatcher (song) could be improved, if possible. Frogs and Spotted Flycatcher produce huge numbers of false positive detections for these.

Very few frog vocalisations were included in the training data, If you know of any datasets that contain these, I'd be grateful for the pointer and add them to the training. I'm not clear if you mean Red-breasted flycatcher song and spot fly song is mixed up or something else?

• Using BirdNet a lot, I know that the sensitivity setting can be important. I found no way to change this setting for the BirdNet model within Chirpity, nor is there any hint of what sensitivity setting is being used for the analysis? The sensitivity cannot be changed after the analysis is done (unlike species or confidence filtering of the results), so it is important to set it before the analysis starts.

It's my understanding that BirdNETs sensitivity setting simply applies a log/sigmoid function to the confidence levels output by their model. By default, i.e. with sensitivity=1 (which is what Chirpity uses) these have a linear profile, if you adjust sensitivity, you will get more of an s-shape, like this:

In practical terms, you may start with a 60% confidence detection, but using sensitivity it could be reported as 80%, and conversely, 40% may become 20%. I haven't made this possible in Chirpity as I don't believe it to be useful - at least not as a global setting. I have found the detection confidence and level at which you can generally take it to be accurate varies from species to species. A species specific sensitivity makes much more sense to me, but it's not a trivial task.

• Can the 'overlap' option from BirdNet be combined with the 'context' setting that Chirpity offers? (how does the context option work, anyway?)

Overlap in BirdNET and context mode in chirpity are similar. I'll assume you understand overlap. Context-mode for the chirpity model processes each batch-sized section of audio as follows:

1. Perform an analysis of the audio, split into 3-second chunks. store this result.
2. Pad the beginning and end of the audio with 1.5 seconds silence. Do another split at 3 second intervals (this is like a 50% overlap)
3. Analyse the padded audio segments.
4. Remove detections that do not meet the required confidence threshold in at least 2 of 3 adjacent/overlapping audio segments.
   The idea here is to mitigate against the false positives which occur when a call or song is clipped at the boundary of a 3 second chunk. Both Chirpity and BirdNET have this issue, where for example the first note of a bird song is misidentified as perhaps the call of another species. Adding overlap just increases the number of boundaries, and make this issue more likely, context mode uses overlap in a way that reduces false positives, rather than increasing them.

Appearance, usability

• for me (personally) the font size could be reduced a little bit to make more room.

Since version 1.14.0, the font size can be adjusted in Settings.

• how about a freely resizeable spectrogram window? (make the horizontal split to the species and detections list flexible up-down?)

Since version 1.14.0, the spectrogram can be resized by dragging the transport control panel.

• Some keyboard shortcuts are 'hard-wired' to the english/us keyboard layout. I noted the + and - adjustments for the spectrogram

This was fixed in 1.13.0.

• very helpful (for me): I really like to pan left and right in the spectrogram with a two-finger swipe on the trackpad when the pointer is over the spectrogram. Audacity does this and the spectrogram in the 'compare with reference calls' pup-up window works like this as well. Only the main spectrogram in Chirpity doesn't, and in fact there seems no way to pan left or right other than letting it play beyond the screen or using arrow keys a lot. Alternatively, a double-tap on the spectrogram could recenter on the click position.

This has been requested before, but I don't think I am going to do it. Audacity is able to do this, and it works for the comparison specs because the whole file is loaded into memory. That's why if can take minutes for Audacity to just open a file, where Chirpity can open it instantly - and can even analyse that file in less time than Audacity takes to open it. It's easy to page through the spec using the keyboard, and enabling the two finger scroll would slow the whole application down unacceptably.

• show frequency indications on spectrogram (the video and manual shows overlaps on left side of the spectrogram, but I don't see these?

I removed these so they didn't obscure the spectrogram where they appear. You can easily move the mouse cursor to a point on the spectrgram to see the frequency at that point. In version 1.12.1, the frequency labels were put back by default. They can be removed in settings.

• very helpful (for me): make the displayed frequency in the spectrogram adjustable. When validating detections, it is extremely helpful to limit the displayed frequency in the spectrogram to the relevant species' song or call. You could even automate this to some degree, i.e. automatically zoom the spectrogram to the relevant frequency range of the current bird species, for example zoom to 100-1000Hz for booming Bitterns or zoom in on high frequencies for kinglets and 'zip' calls.

As of version 1.13.0, you can do this.

• essential (for me): For fast validation it is extremely helpful to have keyboard shortcuts for preset labels. numbers 1-9 (or any other keys) should be available to assign self-defined labels in the settings menu. When reviewing/validating detections, a keypress puts the corresponding label into the label field and Chirpity then automatically jumps to the next detection. This is how several validating softwares for bats do it, and it works reeeally fast. Just as an example, these labels could be 'reviewed: id correct', 'reviewed: id rejected', 'reviewed: id uncertain', 'reviewed: id correct (playback!)'; or a simple quality rating of 1 to 5. This method could also be used to make super-fast species changes with a single keypress (flexible database field that gets updated with a keypress?).

I like the idea of self assigned labels added with a keypress. 1-9 are available. For consistency, I think it will still need a tab keypress, to move to the next detection.

• 'command-v' marks a detection as 'validated' (ie reviewed & correct). But I can't find a way to undo this? On the lower left side (species overview) the checkmark could be accompanied by the number of validated detections of that species. Validation overwrites the confidence number in the csv export. I am not sure is this is the perfect way longer term (see below).

I don't think you can undo this, but you can change or delete the record.

• allow multi-selection of detections to edit them at once. I can edit a single detection, or all detections of a species (by right-clicking on the species summary), but not a selected subset of, say, 30 of 120 detections of a species.

Possible but complex. It will take some time.

• very helpful (for me): option to show all detections (of all species) on spectrogram, all detections of the selected species, or just the one selected detection.

Possible but difficult. There's a lot of logic in the code that relies on there being just one area highlighted in the spectrogram.

Database, data organisation, filtering: Unlike for bats, the passive acoustic monitoring world for birds still lacks a 'Lightroom', ie a comprehensive app with extended database functionality for audio files, various model detections, metadata, and validations.

I agree this would be useful, but have not yet delivered a database management tool that even allow you to update the location of moved or renamed files.

• To me, it is not always clear, if I am looking at a current (new, unsaved) analysis of files, or if I am seeing (old) detections from the archive database. Also, when I open files, it would be nice to have an option to automatically retrieve the saved detections if there are any in the archive database and analyse remaining files if there are no detections in the archive db. Saving detection data to archive could overwrite valuable validation data if these are present in the archive? is there a check for that? (e.g. doing a chirpity analysis of a number of files, and then a birdnet analysis on the same files will overwrite the chirpity validations?) Is it possible to keep both birdnet and chirpity model detections for a file in the archive? Switching to 'explore archive' or 'view charts' seems to discard current (unsaved) detections/validations?

If you open one file and it is known to the database, the file name will appear blue and there will be an option to retrieve the saved results in the analysis menu. Likewise if all open files have had analyses saved, you can retrieve results. If some files are new in the open file list, this won't happen. Conducting a new analysis on a file that has been analysed and saved previously will not overwrite what has been saved, unless you choose to save the new analysis. Even then, I think existing records will not be overwritten as there are database constraints that prevent duplicate records for the same species & time & place.

I think I'll stop there, for now. You have lots of good ideas for how to improve Chirpity, thanks. I hope you understand that as something I do in my spare time, implementing these feature will take a long time - I think we're over a year's worth of effort to do the best bits of what I've responded with so far. Much of what's below is also solid, but so far ahead in the development timeline, it doesn't make a lot of sense to think about it now!!

• Metadata window popup when opening files or folders: location, project, and recording equipment (device manufacturer/type, device number, microphone type and number), the recording settings and scheme are the bare minimum I can think of, which should be entered for a dataset of audio files before analysis. Support user tags for detections, audio files, folders or projects. Support the wav formats that include metadata (wamd and guano). There are python libraries that help with this iirc.
• Allow a separate database file for a project instead of adding everything into a main archive db. Ability to open an existing database files. Split database files (i.e. save a filtered subset) and merge database files.
• show the number of files and folders that are open/in the database and filtered for display. show the current project name, active database name and db file location. When opening multiple folders, make them selectable/filter option.
• support imports from Raven and Kaleidoscope or BirdNet csv result files.
• Enable detections of the two models (or several runs of the same model, but with different settings) in the database. Include the detections of both models in the detections lists & show them at the same time in the spectrogram (maybe in different color shading?). It would be very helpful to have the results of both models shown in the list and spectrogram, as they do produce somewhat different results, detect different species and sometimes one is significantly better than the other.
• greatly enhanced filter options: Time of day, date, folders, locations, projects, any metadata or tags, model type, species Ids, confidence, validated or not, etc... (I am aware that some of this is present already. It would be nice to have all filters in one place)
• keep track of validations, species changes and deleted detections during the validation process. The database could still remember the original detection (with all the data). Medium- to longer-term, such data could be massively useful to determine real-world efficiency, precision and recall rates and confusion species to improve the model. Give the user an option to share the validation data (or even recordings) with you, so you can improve the Chirpity model.

Evaluation, visualisation, reporting

• more detailed time scale (not only whole year/ months, but zoom in on the timeframe of selected / filtered detections). Show recording times, night/daylight and plot the detections.
• show a map with recording locations and overlays for each location, visualising the selected / filtered detections (this could be for example species diversity at a location or avg. number of detections per day for a certain species at each location, etc.)
• provide summary tables (maybe flexible, like pivot tables in excel-type programs)