AWS S3 Enabled Downloads

[jmchilton]

• Modify client/galaxy/scripts/mvc/upload to add a new upload button for cloud accounts.
• Add API endpoint to list accounts configured(?), buckets, and files. Look at FTP files as an example.
• Serialize selection as a URI - s3://bucket/path
• Update tools/data_source/upload.py to dispatch s3 type (look for "url" and "urlopen").

[hexylena]

@VJalili don't you have something for this? I guess we can close after that, then?

[VJalili]

Does the following API implement the expected?

https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/webapps/galaxy/api/cloud.py

We do not have a UI at the moment for getting data from cloud, but for sending to cloud, we have the following tool:

https://github.com/galaxyproject/galaxy/tree/dev/tools/cloud

[MikeKroell]

We are looking for the same functionality. We have all our data in S3 and currently, our users need to download to their machine and upload to Galaxy which is pretty inefficient. If they could drop in the S3 URI and Galaxy could download directly into it's datastore, it would be streamlined. Having the HTTPS/FTP section accept S3 URIs would be acceptable. It should also be able to use IAM roles for access.

[VJalili]

@MikeKroell the backend functionality is there, here are the docs on how to use it: https://galaxyproject.org/cloud/storage/#send-data-to-cloud

It's been a long while I've not tested it, so please let us know if you see any issues. This will require OIDC (only Google backend will work) to be enabled on the Galaxy server, so it will not yet work on Galaxy main (usegalaxy.org). OIDC is required because these APIs do not ask for user credentials (e.g., access key and secret key), rather it obtains the required authorization on the fly through OIDC and role-based access. Also, one downside is that we still do not have UI for getting data from S3 to Galaxy, but we have UI to send data from Galaxy to S3. But if you're comfortable with API-based interaction, it should work.

[MikeKroell]

Thanks for the quick response @VJalili. Our users don't know how to use the API, and it doesn't use IAM roles. If users could copy-paste an S3 URI into the GUI, this would work well. Added bonus if it had an option for recursive.

Example:

[VJalili]

I guess it should work if you paste signed-urls.

[MikeKroell]

I guess it should work if you paste signed-urls.

Yea, this is what we thought of as well.

[nuwang]

With the unified handling we have now, I think we should be able to add support for directly downloading public s3 URLs. Are there any security concerns etc?

[nuwang]

@jmchilton Will user-importable object-stores add support for s3 filesources like this? Or will that need to be handled separately?

[jmchilton]

That PR won't address this - I would think we need a URI handler for this using your unified handling. I don't have specific security concerns