Features/beacon network #93

gsfk · 2024-06-11T16:32:50Z

First pass at code for networking Bento beacons together.

On startup, contacts all beacons in a list of urls passed in from a config file.
routes requests/responses for any beacon in the network.

Aggregation is done in the client rather than here. This allows for real-time updates in the frontend, rather than waiting for the slowest beacon to respond. However, aggregation is easy and should come in a later version, since it lets us:

treat the network like a single beacon
add the network to an existing network
search the network from a non-Bento client that doesn't know how to aggregate the data

Some features are still experimental:

uses bento_public search config for filtering terms discovery (eg sex=MALE or BMI<18) since that's what our beacons currently use. Once changes are rolled out to all beacons, we can use beacon spec filtering terms instead.
The network itself is hosted on a particular beacon instance, which will be expected (but not required) to be part of the network, so there is some extra handling to avoid non-terminating circular http requests. Whether we are better off hosting the network separately is left as an exercise for the reviewer.

Related PRS:

Todos for next version:

init network at first network request rather than at app startup
make network init calls async
back-end aggregation (see above)

davidlougheed

first pass

davidlougheed · 2024-09-03T19:07:29Z

bento_beacon/authz/headers.py

@@ -2,6 +2,9 @@


 def auth_header_getter(r: Request) -> dict[str, str]:


type hint seems wrong here if request can be false-y - should be Request | None?

this one is fiddly.. r is not actually a request, it's a werkzeug LocalProxy that acts like a request when a request happens to present. When no request is present it's just LocalProxy rather than none.

I'm considering just removing this and handling the no-request edge case elsewhere. Seems inelegant to ask for headers when there's no request.

bento_beacon/network/bento_public_query.py

bento_beacon/config_files/config.py

bento_beacon/network/bento_public_query.py

davidlougheed · 2024-09-03T19:31:36Z

bento_beacon/network/utils.py

+from ..endpoints.variants import get_variants
+from .bento_public_query import fields_intersection, fields_union
+
+PUBLIC_SEARCH_FIELDS_PATH = "/api/metadata/api/public_search_fields"


it'd be better to accept the katsu URL from a configuration variable and then build from there (so just katsu URL + /api/public_search_fields)

ah, hmm. These are calls to other katsus in the network:

PUBLIC_SEARCH_FIELDS_PATH probably belongs in config rather than hidden in this file. I probably left it here because this entire file will be gone in the next version.

beacon network is configured by passing in beacon base urls. We find the katsu search fields by figuring out the base bento url (ie remove /api/beacon) and adding on PUBLIC_SEARCH_FIELDS_PATH. So, you have your choice of which is least inelegant:
  - url surgery as above
  - configure network by using bento base urls instead of beacon urls
  - configure network by passing both beacon and katsu urls

In the next iteration this will all be replaced by calls to network beacons only, so this sort of url modification will no longer be needed.

if it's temporary it's fine to leave in, but in general I think not relying on Bento's particular URL setup at time of code-writing (so configuring Katsu URL separately) is the best long-term approach if any contact is needed.

bento_beacon/network/utils.py

davidlougheed · 2024-09-03T19:34:58Z

bento_beacon/network/utils.py

+    return fields
+
+
+def public_search_fields_url(beacon_url):


this shouldn't rely on katsu being under the portal. subdomain forever in bento. instead, take the Katsu URL as an env var (similar to aggregation or WES) and build it from Katsu + whatever subpath is required.

davidlougheed

from a quick scan of the code. i should also try this out... is there an easy way I can set this up locally?

davidlougheed · 2024-10-03T15:47:05Z

bento_beacon/utils/katsu_utils.py

-def get_katsu_config_search_fields():
-    # Use forwarded auth for getting available search fields, which may be limited based on access level
-    fields = katsu_get(current_app.config["KATSU_PUBLIC_CONFIG_ENDPOINT"], requires_auth="forwarded")
+def get_katsu_config_search_fields(requires_auth):


would be good to type hint this argument the same as the other one - you could make a type RequiresAuthOptions = Literal["none", "forwarded", "full"] and use that as the type hint both here and above.

davidlougheed · 2024-10-03T16:00:39Z

bento_beacon/config_files/config.py

@@ -1,5 +1,6 @@
 import json
 import os
+from ..constants import GRANULARITY_COUNT, GRANULARITY_RECORD


it's nice to have local imports below module imports (urllib3 here)

gsfk · 2024-10-03T16:07:49Z

for local test, use Bento v17 branch and see setup instructions here.

You'll probably need to change bento_public version to pr-165

Edit: also requires this patch if not already merged

davidlougheed

tested with the default config for now - seems to work well.

gsfk added 22 commits May 27, 2024 20:04

add first draft of network utils

4cb54fe

first draft of endpoints for beacon network

4d10ceb

don't crash if a network beacon is down

c21d71d

handle filters

f2b6a48

temp network config

912b2e2

network fixes

3e458aa

lint

b100bef

workaround for flask url_prefix issues is to not use it

69c7221

tinkering with querySections

ae63b28

add katsu rule changes

378a3e8

tweak network timeouts

8b320d7

Merge branch 'master' into features/beacon-network

901e1f1

lint

aace127

temp network test

b4a3877

temp bento public query handling

8b91bbe

headers edge case

25e4e36

config additions for network

3426ab9

network config

7098261

delete temp hardcoded stuff

76bab6b

network updates

2d33fde

network utilts

e5579f9

add network timeout params

ee184ef

gsfk marked this pull request as ready for review August 15, 2024 19:38

gsfk mentioned this pull request Aug 15, 2024

Add beacon network bento-platform/bento#259

Merged

4 tasks

gsfk added 5 commits August 23, 2024 17:13

completed todo

1cb958e

completed todo

4f47193

parameterize beacon network urls

0f4cb38

don't allow filters with empty options list

0d01716

lint

ee67e8f

gsfk requested a review from davidlougheed September 3, 2024 13:03

davidlougheed requested changes Sep 4, 2024

View reviewed changes

gsfk added 12 commits September 4, 2024 10:41

unused config

d472877

snake case

51dcd6f

better handling for missing request headers

a09a117

merge description changes

2417c65

fix variants count for beacon network host

20d1005

Merge branch 'master' into features/beacon-network

a2cdb71

fix for network query sections auth

631eae2

use "none" auth for beacon network

7ad1134

beacon network init fixes

c473f2b

comments

cc3b528

lint

0ec58be

fixes for katsu private overview changes

6b5d553

gsfk requested a review from davidlougheed October 3, 2024 15:35

davidlougheed reviewed Oct 3, 2024

View reviewed changes

gsfk mentioned this pull request Oct 3, 2024

feat: Beacon network UI bento-platform/bento_public#165

Merged

gsfk added 4 commits October 3, 2024 18:50

reorder imports

cd5833b

auth type hint

7f2c7f6

lint

0346e92

correct network config file name

2478ba1

davidlougheed approved these changes Oct 3, 2024

View reviewed changes

gsfk merged commit d278832 into master Oct 3, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Features/beacon network #93

Features/beacon network #93

gsfk commented Jun 11, 2024 •

edited

Loading

davidlougheed left a comment

davidlougheed Sep 3, 2024

gsfk Sep 4, 2024

davidlougheed Sep 4, 2024

gsfk Sep 4, 2024 •

edited

Loading

gsfk Sep 4, 2024

davidlougheed Sep 3, 2024

gsfk Sep 4, 2024

davidlougheed Sep 9, 2024

davidlougheed Sep 3, 2024

gsfk Sep 5, 2024

davidlougheed left a comment

davidlougheed Oct 3, 2024

davidlougheed Oct 3, 2024

gsfk commented Oct 3, 2024 •

edited

Loading

davidlougheed left a comment

		@@ -2,6 +2,9 @@


		def auth_header_getter(r: Request) -> dict[str, str]:

Features/beacon network #93

Features/beacon network #93

Conversation

gsfk commented Jun 11, 2024 • edited Loading

davidlougheed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gsfk Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidlougheed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gsfk commented Oct 3, 2024 • edited Loading

davidlougheed left a comment

Choose a reason for hiding this comment

gsfk commented Jun 11, 2024 •

edited

Loading

gsfk Sep 4, 2024 •

edited

Loading

gsfk commented Oct 3, 2024 •

edited

Loading