Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Next Steps to Bring Your Own Storage and Data #18128

Open
7 of 21 tasks
jmchilton opened this issue May 11, 2024 · 9 comments
Open
7 of 21 tasks

Next Steps to Bring Your Own Storage and Data #18128

jmchilton opened this issue May 11, 2024 · 9 comments

Comments

@jmchilton
Copy link
Member

jmchilton commented May 11, 2024

#18127 is a monster PR - a classic John framework (times two) and in rare form contains both a wide variety of well tested plugins and deep documentation - but it still could use a lot of polish and there are a lot of things we could potential do to make it more palatable for the usegalaxy * servers.

Verify before Create

#18222

  • Implement “Verification” green/red dots before creating a plugin (think like wifi/connection testing vibes). Need to check a configuration before persisting it.
    • Green/red dot for plugin verification.
    • Green/red dot for variables expand to create valid configuration.
    • Green/red dot for connection established.
    • Test then create, would prevent a lot of misconfigured instances from being created I think.
  • Allow verification re-checks after the fact from the manage menu. Verify the persisted thing instead of the payload.

OAuth Use Cases

  • Work through a full oauth configuration.
    • Revise Dropbox file source plugin to use modern, functional APIs.
    • Work through functional template examples for Dropbox and Google Drive that work using auth.
    • Decide what the framework needs to support these use cases and implement it.

UI QOL Enhancements

  • Disable form submission during testing and create/update/upgrade.
  • User object store index should allow selecting an object store as the user default - feels odd to still have to navigate to another user option to make that the default.
  • Revise removing instance - removing a plugin should maybe give a few options. The API makes a distinction between “hidden” (still works but not shown in the file tree for instance) - which is useful if you are done adding to something but you still want old dataset or old URIs to work, “inactive” which disable the use of the plugin (maybe you want to go a step further and disable the possibility for charges or whatever), and “purged” which also deletes all the secrets associated with the plugin. I’m not sure what @martenson will let me get away with.
  • There is a log of good text in the description of the YAML files but it doesn’t appear in the form. It should maybe appear at the top of the form inside the the UI card.
  • Implement a help section the way tools have and place below the form. Markdown instead through obviously.
  • Implement “Agree to Terms” of some kind - make the downsides of sharing credentials clear.

Underlying Plugin Improvements

Testing:

  • Unit or integration testing for disable_templating. Make sure Cheetah templating doesn’t apply after the fact to the Jinja templated file sources.
  • Extend user object store selenium test cases with selection and use and verify
  • Extend user file source selenium test cases with verify output
  • Selenium tests for upgrading user object stores and user file sources.

Collect and work through existing issues around object store selection & private object stores

@itisAliRH
Copy link
Member

itisAliRH commented May 16, 2024

I configured my local using the Hands-On Production Examples from the main PR description and have the following comments on it in addition to your comment:

Configuration and Documation

  • I created config/file_source_templates.yml and config/object_store_templates.yml but the object store buttons (history and tool) are not showing up (because object_store_allows_id_selection from the config object is false).
  • Also created object_store_conf.yml file using examples from object_store_conf.sample.yml but only the type: distributed ones are working (neither disk nor hierarchical)

UI

  • I have created a new object store using AWS credentials in user settings and ran a tool by choosing this location to store the results; but it raised an error and the dataset detailed fetcher is returning 500 only and it blocks the Ui (the error is not handled here and it brakes the UI but not related to these changes)
  • Preferred storage location is not visible to the user until hovering on the icon (I'd put the selected name next to its icon)
  • It's not clear what will happen to the existing datasets in history when the user changes the preferred location store. Either if they are moving to the new place or only the new datasets will be stored in the new location, we should inform them (showing a storage icon in the dataset card with the name of the object store that it is stored in would be an idea in this case)
  • I found some things in the user preferences that could be improved and will work on enhancing this if you agree in follow up PRs.

@jmchilton
Copy link
Member Author

It's not clear what will happen to the existing datasets in history when the user changes the preferred location store.

This is not related to this PR right? You could say the same thing about storage options on main. I've updated the hover message to be bold in 24.0 (both of these message are from 24.0 and usegalaxy.org):

Screenshot 2024-05-16 at 11 58 05 AM

The message during creation also makes it clear the selection applies only to new datasets.

Screenshot 2024-05-16 at 11 59 11 AM

I wonder if making this a history option was a mistake - maybe it should just be a user option and an option on the tools and workflows.

@jmchilton
Copy link
Member Author

jmchilton commented May 16, 2024

[ ] I created config/file_source_templates.yml and config/object_store_templates.yml but the object store buttons (history and tool) are not showing up (because object_store_allows_id_selection from the config object is false).

[ ] Also created object_store_conf.yml file using examples from object_store_conf.sample.yml but only the type: distributed ones are working (neither disk nor hierarchical)

Lets move this conversation to #18157 ? I think both of those points are related to needing a distributed object store for this to work (which again main and EU both have). I think I've discussed this there but I will definitely updated the documentation inside the PR right now to make the current state of affairs clear.

@itisAliRH
Copy link
Member

This is not related to this PR right? You could say the same thing about storage options on main. I've updated the hover message to be bold in 24.0 (both of these message are from 24.0 and usegalaxy.org):

You're right, my bad. I apologize! Having an icon next to each dataset would make it easier to understand their storage location.
Bolding the important parts is a good way.

@martenson
Copy link
Member

martenson commented May 16, 2024

I wonder if making this a history option was a mistake - maybe it should just be a user option and an option on the tools and workflows.

It is a useful option, but maybe for the most advanced users out of the four locations. However the most visible.

@jmchilton
Copy link
Member Author

Updated Docs about the distributed object store:

Screenshot 2024-05-16 at 12 15 38 PM

@jmchilton
Copy link
Member Author

@martenson , @itisAliRH I appreciate the comments regarding the history default indicator - including "Preferred storage location is not visible to the user until hovering on the icon (I'd put the selected name next to its icon)". It is clearly a pressing issue. In response to @itisAliRH's "You're right, my bad. I apologize!" - there is no need to apologize - everyone says this - there is an issue the help text does not resolve and I'd love to work with the UI/UX team on figuring out a solution.

Also though this is a pre-existing issue - I've created a discussion for it here #18163. Lets discuss this the context of object store selection instead of user object stores - scratch storage on main has the same issue for instance. None of this is an attempt by me to side step this though - we've got to get this resolved more urgently for it being an issue under more conditions than just user defined object stores.

@jmchilton
Copy link
Member Author

I have created a new object store using AWS credentials in user settings and ran a tool by choosing this location to store the results; but it raised an error and the dataset detailed fetcher is returning 500 only and it blocks the Ui (the error is not handled here and it brakes the UI but not related to these changes)

This seems like a combination backend/frontend issue:

For the backend, any chance you can track down some kind of backend stack traces to help track down the problem?

In terms of the frontend...

I've seen a similar problematic lock up behavior on bad object stores - but for me it kills the backend - but I think the source is probably still the problamatic loop related to exception handling in the client (is there a pinia store that is looping infinitely or something do you think?).

My gut is also this is a pre-existing issue in some ways but this code makes it trivial to hit the issue.

I think every time I have observed this behavior - I had a misconfigured object store that got serialized and setup. My top priority in the above list is testing user defined objects before they are created for a reason (I've made some good initial progress this morning - 0bb53fd - it provides some nice feedback on a variety of issues). It would catch so many problems. Even once we validate object stores on creation though... the core frontend issues need to be resolved... invariably keys will be invalidated, remote APIs will disappear, connections will be bad - and we need to not crash the UI or the backend in those cases.

@jmchilton
Copy link
Member Author

@itisAliRH In response to: "I found some things in the user preferences that could be improved and will work on enhancing this if you agree in follow up PRs.". I'm still working hard on the UI for this ahead of the GCC - any advice would be great. I'm happy to accept help too but if you're serious I would scope out the issues so we can divide them up and so I can warn you if you're working on something I'm touching.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants