refact!: discovery module + revised overview endpoints

[davidlougheed]

BREAKING CHANGE: new formats for various overview endpoints in order to share more code.

beacon: bento-platform/bento_beacon#79

web: bento-platform/bento_web#392

[codecov]

Codecov Report

Attention: Patch coverage is 92.04340% with 44 lines in your changes are missing coverage. Please review.

Project coverage is 88.08%. Comparing base (a1a7305) to head (3a0f715).
Report is 21 commits behind head on develop.

Files	Patch %	Lines
chord_metadata_service/phenopackets/utils.py	51.16%	19 Missing and 2 partials ⚠️
chord_metadata_service/discovery/fields.py	90.97%	6 Missing and 7 partials ⚠️
chord_metadata_service/discovery/fields_utils.py	94.02%	2 Missing and 2 partials ⚠️
chord_metadata_service/discovery/stats.py	90.47%	2 Missing and 2 partials ⚠️
chord_metadata_service/discovery/api_views.py	97.01%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #496      +/-   ##
===========================================
+ Coverage    87.70%   88.08%   +0.37%     
===========================================
  Files          116      127      +11     
  Lines         4467     4566      +99     
  Branches       672      671       -1     
===========================================
+ Hits          3918     4022     +104     
+ Misses         393      390       -3     
+ Partials       156      154       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[v-rocheleau]

First pass, nice asyncio usage for summaries

[davidlougheed]

still working on tests for this, although project coverage may be lower now due to testing of branching.

[v-rocheleau]

minor: make low_counts_censored default to True as a safeguard

[v-rocheleau]

LGTM!

[gsfk]

Looking good... I was thrown off at first by the new nesting experiment.data_type_specific.experiments but it seems to make sense. Two issues / questions:

• katsu has an endpoint api/search_overview that is used both by beacon and by bento_web: it takes a list of individual ids as input, and returns (uncensored) overview statistics for them. The format is the same as the old front page overview with some outer nesting removed. Now overview and search_overview are out of sync. They don't particularly need to be in sync, but it seems inelegant somehow if they're not. In the long term I wonder about the usefulness of a hardcoded overview fomat as opposed to the configurable public one.

• Is public overview always censored with count_threshold, or are there permissions that can remove the threshold?

[davidlougheed]

regarding point 2, permissions haven't been done yet but will be done in the future

[gsfk]

One oddity about overview censorship (including current master) is it will wipe out entire categories if they are empty. This is fine for pie charts but not necessarily ideal for bar charts (where you may want to show categories with value zero.... think of the age distribution chart that becomes confusing if you remove bars from the middle of the chart if they happen to be empty)

... I guess the same point can apply to sparsely populated categories even when uncensored.

[davidlougheed]

Yeah that's not ideal handling - although if it's in master I won't bother with it here for now