Allow users to browse s3 buckets in JupyterLab interface

[batpad]

It would be useful to have a browser similar to the local file browser in JupyterLab for s3 buckets that the users on the Hub have access to.

From speaking to @yuvipanda this feature might already exist and maybe just some configuration to setup. @yuvipanda whenever possible, if you could point in the right direction, I can fill out this issue in more detail - thanks!

cc @wildintellect @abarciauskas-bgse

[wildintellect]

I think it was suggested we try https://github.com/IBM/jupyterlab-s3-browser <- is this maintained?

[yuvipanda]

That one isn't compatible with JupyterLab 4. https://github.com/Navteca/jupyterlab-bxplorer seems to be, and perhaps is worth trying. Needs an image with it in there to be tried out.

[j08lue]

Bucket explorer could be helpful to some users, so if there are easy and sustainable pathways, we should consider adding such a feature.

However, we do want to promote workflows where users share and discover assets via (user-group-specific or public) STAC. File names can only hold that much information and there are no standard conventions, so discovery of assets works a lot better (and interoperable) via STAC.

(NB we are connected to Navteca via the NASA SMCE, which they support, and have earlier on in the project been talking to them about developing solutions that VEDA needs. It should be easy to open up those direct channels again, if need be.)

[wildintellect]

@j08lue this feature request is not for sharing, it's for users to find their own files since VEDA does not auto-mount buckets like MAAP. ➕ we want open sharing to be done via STAC.

[wildintellect]

See also MAAP-Project/Community#768

[yuvipanda]

@minrk, the primary creator of JupyterHub, is also exploring this for another project - destination-earth/DestinE_ESA_GFTS#13 has more information. It uses https://github.com/jpmorganchase/jupyter-fs and seems to look good?

[batpad]

This is probably a requirement for the MAAP migration - #43

Next steps here:

• Try out the same approach as this: enable browsing s3 via jupyter-fs destination-earth/DestinE_ESA_GFTS#13 (mentioned in @yuvipanda's comment above)
• Add the JupyterLab extension and config and enable browsing specific s3 buckets on the VEDA hub

@sunu let's add this to our agenda to chat about when we meet next.

cc @yuvipanda

[wildintellect]

@batpad can we get one of these options deployed at least in staging before Aug 14 (demo to JPL/HQ of VEDA Hub as a future of MAAP)?

cc: @jsignell @yuvipanda @maxrjones

[batpad]

I created an image to test with the bxplorer extension installed.

I see the extension show up in the left panel and it shows me s3 buckets, lets me browse them, etc.

@wildintellect would you be able to help test and we can figure out if this works / what else we might need?

Steps to test:

• Select the Bring Your Own Image option
• Use this image: public.ecr.aws/nasa-veda/pangeo-notebook-veda-image:683db507c7a5
• You should see the bxplorer icon in the left panel - clicking on it should show an s3 browser in the left pane

I didn't spend a lot of time on this, so there are probably things we can configure, etc. - this is pretty much the default install currently.

(this is the branch / PR for the image building, in case anyone wants to try any changes to config etc: NASA-IMPACT/pangeo-notebook-veda-image#18 )

cc @sunu

[yuvipanda]

I just tried it out and it works. yay.

Double clicking a file explored this way doesn't do anything though. I'd expect it to open in the jupyterlab browser but i suppose that's not a universal expectation. But I'd like it to at least do something. Downloading is a bit clunky but that's probably ok too. I'd love for it to have a 'copy URI' option so we can use it in code. All these can be contributed upstream of course.

But i'd love to hear from actual MAAP users if this is good enough :)

[yuvipanda]

@batpad I'd also love for you to take a quick look at the code (https://github.com/Navteca/jupyterlab-bxplorer) to see if it feels like something we can contribute to.

[wildintellect]

Noting that Navteca (Ramon) is the main support behind SMCE.

I gave bxplorer a try:

1. adding a bucket is on the Favorites page - bigger issue is that permissions to such a bucket have to be pretty open, I tried maap-ops-workspace and nasa-maap-data-store, neither worked, though technically the role the hub uses should be able to see some of the files in those buckets. Anyone know another bucket to test with? I'm not sure it's using our role, lp-prod-protected also didn't work.
2. Right click gets you the Download option, we don't want that, we want copy URI for inserting into code. I'm glad in way it doesn't open files, that could be painful way to lockup the system.

[j08lue]

I think it is fine that publicly catalogued buckets do not support listing (only reading).

What kinds of buckets do MAAP users browse via the explorer, @wildintellect? Some kind of shared temp space?

[wildintellect]

@j08lue maap-ops-workspace is the critical one right now, which includes both dps_output and shared spaces.
Longer term (later this quarter), different groups like EIS fire have their own buckets they'd want to access. It's hard to predict but other buckets come up occasionally that don't have catalog service over them. So you're right that this is for buckets that lack catalogs.

[batpad]

@batpad I'd also love for you to take a quick look at the code (https://github.com/Navteca/jupyterlab-bxplorer) to see if it feels like something we can contribute to.

Gave things a quick look and it looks all great - the frontend seems standard React and easy to work with if we need to.

Right click gets you the Download option, we don't want that, we want copy URI for inserting into code. I'm glad in way it doesn't open files, that could be painful way to lockup the system.

Is this something we should ticket adding? The code seems reasonable enough that we should be able to submit a PR for this.

I also took a bit of a look at https://github.com/jpmorganchase/jupyter-fs - @wildintellect do you think it's worth trying out as well or does bxplorer seem workable? the jupyter-fs extension seems like a bit more work to setup with some required configuration, etc. - am happy to spend some time early next week if that seems useful @wildintellect .

[wildintellect]

bxplorer seems fine for now, we can show it off in our VEDA demo, and talk about how it comes from another part of NASA ...
later we should try other options to compare (sometime this PI)

[anayeaye]

@stephenkilbourn just shared this with data services: https://aws.amazon.com/about-aws/whats-new/2024/09/storage-browser-amazon-s3-alpha-release/

[wildintellect]

@anayeaye interesting but I don't think that applies to this use case which require python interoperability, unless we want to write our own Jupyter Extension in JS.

This might apply to some VEDA front end ideas around browsing catalog items for download.