NWB from DANDI no longer load?

[yarikoptic]

original report by @memartone:

• Couldn't launch the NWB explorer dandi/dandi-archive#2052

which reproduced for me by trying to open nwb explorer via

• http://nwbexplorer.opensourcebrain.org/nwbfile=https://dandiarchive.s3.amazonaws.com/blobs/bae/79b/bae79b98-0484-469b-ab50-aa6f96eed831 which redirects to
• https://nwbexplorer.v2.opensourcebrain.org/user/a-0a800fdc-08b2a/geppetto which keeps just spinning without any error feedback to user:

and only in console we can observe

[filippomc]

Hi @yarikoptic, there's been some overloading on cluster resources. Did some cleanup and could load the file from the link above

[yarikoptic]

unfortunately for me the situation didn't change a bit - same behavior and same errors in console.

[sanjayankur31]

We had a look at this and we're basically running into a DoS situation here. We're constantly seeing ~200 pods for anonymous nwbe usage and that basically gets the google cloud cluster at its max levels for anonymous accounts and prevents it from spawning more.

[yarikoptic]

are there any limits in place, e.g. no more than a pod from IP?

in either case -- might be nice to provide user an error message (e.g. "Maximum capacity of the cluster is reached. Try later") if such situation could be detected at that interface level.

[sanjayankur31]

From what I know, the limits aren't IP based. Google cloud clusters can be provided limits to how many nodes/storage/etc. the cluster should use and what they can be scaled up to. So, once the cluster hits the limit, it won't spawn new pods until some resources are freed up. Ideally, these limits should be high enough so that this situation never arises but at the moment since we're between grants, we need to keep these limits in check to manage the cluster expenses (higher limits cost more, even if they're not completely in use) :/