Allow installation of visualizations from toolshed

[hexylena]

Whether in this incarnation of the TS or the next, we need to have visualizations in the toolshed.

Given that some visualizations are Galaxy version specific (either relying on JS included with Galaxy, or GIEs), I'm open to hearing how y'all want to handle that.

As an admin, I'd rather not have to uninstall one repo and install another during Galaxy upgrades. Maybe this combined with #1188 filtering which versions are available to be installed will make this possible.

[nsoranzo]

👍

xref: https://trello.com/c/YLA70rZx/977-visualizations-framework-use-the-tool-shed-for-distribution

[gregvonkuster]

@erasche @nsoranzo @jmchilton @bgruening Its great to see this in the planning stages with a 16.04 goal, but this time frame is quite a ways off and it is uncertain whether automatic installation of visualization plugins associated with tool installations will actually be available in that time frame.

There are several tools now available in the Tool Shed (with more currently under development) whose use is either significantly restricted or non-existent without an associated visualization plugin. The constructive solid geometry tools that I've been working on are a good example.

Perhaps we can find a reasonable way to document the tool's requirements on a separate plugin that doesn't come with the tool when it is installed, including its location and some installation instructions. But I'm wondering if a better interim solution would be to allow PRs that contain working visualization plugins to be accepted into the Galaxy code base. These are different than tools in that they would not need to be migrated out of the code base at some point. Reproducibility is not a factor, and when automatic installation of these plugins is available, they can just be deleted from the Galaxy code base at that time. What are your thoughts on this?

[bgruening]

@gregvonkuster I guess the biggest problem is to figure out how to handle namespaces in visualisations. What happens if two visualisations have the same name?

Following suggestions for the time being: Please try to change visualization_plugins_directory to point to tool_dependency_dir/visualizations. Then we use my Docker-Vis hack but make is a little bit more pretty and create a script that we ship with Galaxy, that downloads a given visualisation and puts it in this location. We could even create our own simple Visualisation registry, similar to cargo-port, where we can control the name space and collect visualisations and the correct URL.

[hexylena]

@bgruening

handle namespaces in visualisations

same way we handle other TS naming issues. By adding a long prefix for /shedurl/owner/repo-name/version. Or is there more going on here that I'm not aware of?

We could even create our own simple Visualisation registry

no, please no... we already have the toolshed, let's do it all in one place. We're just unpacking a tarball and maybe handling dependencies (if we're brave). I worry that yet-another-registry means yet-another-level-of-complexity

@gregvonkuster

this time frame is quite a ways off and it is uncertain whether automatic installation of visualization plugins associated with tool installations will actually be available in that time frame.

they're just another type of repository. I cannot imagine it'll be that tough to do. It's been a while since I've looked at legacy TS code, so maybe this will take longer than projected. I've been known for being a bit too optimistic with my timeframes. I'm going to have this done by GCC2016, for sure though.

But I'm wondering if a better interim solution would be to allow PRs that contain working visualization plugins to be accepted into the Galaxy code base.

you're welcome to open the PR, I have no opinions on that matter. If the datatypes are in, I see no reason to refuse a visualization plugin

Perhaps we can find a reasonable way to document the tool's requirements on a separate plugin that doesn't come with the tool when it is installed, including its location and some installation instructions.

not sure I follow on this point.

These are different than tools in that they would not need to be migrated out of the code base at some point. Reproducibility is not a factor, and when automatic installation of these plugins is available, they can just be deleted from the Galaxy code base at that time. What are your thoughts on this?

yep! That was the plan. Get TS support for viz, then migrate everything there, forcing the galaxy team to solve out the "we need to install these tools/viz plugins on first run" problem

[bgruening]

handle namespaces in visualisations

same way we handle other TS naming issues. By adding a long prefix for /shedurl/owner/repo-name/version. Or is there more going on here that I'm not aware of?

I guess this is more complex from a usability point of view. You don't want to have two times the same name in this small visualisation list, aren’t you :) Afaik this hierarchy is currently not supported so we need to touch the visualisation code as well.

We could even create our own simple Visualisation registry

no, please no... we already have the toolshed, let's do it all in one place. We're just unpacking a tarball and maybe handling dependencies (if we're brave). I worry that yet-another-registry means yet-another-level-of-complexity

Let's see what happens to the TS in the next month. This is more or less what I'm doing right now to get visualisations into Docker flavours and it works quite well for my small use case.

If we don't take the namespace issue into account we can use what is already there. We extend https://github.com/galaxyproject/galaxy/blob/dev/config/galaxy.ini.sample#L243 to include $tool_dependency_dir/visualisations/ and create a new action type similar to download_by_url that copies content to $tool_dependency_dir/visualisations/. This should be sufficient for most of the plugins.

[hexylena]

I guess this is more complex from a usability point of view. You don't want to have two times the same name in this small visualisation list, aren’t you :) Afaik this hierarchy is currently not supported so we need to touch the visualisation code as well.

Good point, but is this a real concern, or are we fixing a future hypothetical bug?

If we don't take the namespace issue into account we can use what is already there. We extend https://github.com/galaxyproject/galaxy/blob/dev/config/galaxy.ini.sample#L243 to include $tool_dependency_dir/visualisations/ and create a new action type similar to download_by_url that copies content to $tool_dependency_dir/visualisations/. This should be sufficient for most of the plugins.

sounds great!

[gregvonkuster]

@erasche @bgruening Based on this conversation and a separate conversation with @blankenberg I was preparing to submit a PR for a visualization plugin, but noticed that Galaxy's .gitignore file include this entry: config/*, so I'm wondering if that can be removed so that I can successfully add a visualization plugin. Thoughts?

[nsoranzo]

@gregvonkuster You can explicitly add files to the git index with git add FILE.

[gregvonkuster]

@nsoranzo Thanks for the info: I had to use git add -f from the command line. This doesn't work from my Eclipse IDE without even more work, but works fine in a term window, ;)

[hexylena]

In response to #2861 (comment), moving this discussion off @guerler's PR, since it is getting increasingly back to this particular issue rather than relevant to his PR.

@dannon

Admins have complete control, always, but what comes 'out of the box' is a balancing act, right?

It would be a lot nicer for those in, e.g. engineering domains, to never have to know the fasta format exists and be able to use the framework. As much as some of my biologists get freaked out when they see options they don't understand, I have heard other disciplines feeling similarly during framework selection and looking at other projects because "it's filled with all of these biological assumptions."

One of the awesome things about Galaxy when I very first tried it ~7 years ago was how I literally just did 'hg clone', 'run.sh', and I was able to do things right out of the box.

And surely that would still be the case, if you had a list of defaults and bootstrapped for that with things you wanted? Just another line in run.sh, and you'd shift a tiny bit of download time from your git clone step to your startup?

[jxtx]

And surely that would still be the case, if you had a list of defaults and bootstrapped for that with things you wanted? Just another line in run.sh, and you'd shift a tiny bit of download time from your git clone step to your startup?

Yes, and this was always the plan! We needed to move tools out of the main Galaxy codebase so they could be versioned independently through the toolshed, but when a new Galaxy was initialized we would automatically bootstrap it with a core (but pretty minimal set of tools). This will be even better than before because we can also install dependencies, etc.

[dannon]

I don't think moving discussion to this separate issue helps, since the actionable item right now is @guerler's #2861 PR.

I also don't think there isn't any disagreement that the right way to do it is to have separate repositories. It's my fault that I conflated the issue a bit when I was trying to ask the question of whether or not this is a must have it now out-of-the-box visualization, or whether it's ok for admins to install it manually, as all the pluggable stuff is done.

[jgoecks]

There appears to be broad agreement about how to set up a minimal Galaxy instance and bootstrap from the toolshed and potentially other data/visualization/tool/dependency sources. I concur with @erasche @jxtx @dannon as well.

Given this agreement, I would +1 @dannon's point: we should return to #2861 because we have to make a decision on it knowing that while we agree on the long-term approach, this PR has forced a decision about what to do right now.

[jmchilton]

I started writing something for #2861 - but I don't think it really made sense there. I doesn't argue for inclusion or now but I just love myself and so I wanted to place the ... pragramatic @jmchilton perspective into the record.

Tool shed migrations were tough - not because it is tough to remove something but because dependencies were tough. Tools in Galaxy pre-tool shed and tools post-tool shed were different things - one included context and dependencies and we spent the time building that stuff up.

I don't think the lesson of the tool shed should be it is hard to remove stuff from Galaxy - I think it should be that it is hard to build registry for dynamic plugins to Galaxy. We should not even attempt to build a registry for a new type of plugin until...

• ... we have a solid interface - notice that in this PR each chart plugin was updated because the interface changed. Having centralized code is great because it lets us evolve the interface like that - I'm not convinced that we have an interface we can support indefinitely (but I'm open to be proven wrong).
• ... we have solid tooling - where is the application to test if a viz or chart plugin is compatible with a version of Galaxy, structure correctly, to install it, etc...
• ... we have a solid testing strategy - I don't know how to test viz and I know there are some tests for the frameworks - but if we have a plugin interface we are committed to it should be testable and have great documentation and examples for testing right?
• ... we have a built at least one good registry. No one is yet super happy with the tool shed's ability to serve tools. Once we have done that - once as a project we can demonstrate that we can build and maintain a suite of repository functionality that we are satisfied with and proud of - I think we should try again. At that point - I'd argue the next target should be workflows - they would be more user-facing than viz and so we'd probably get more different and more customized workflows than viz - and hence there is more of a need for a registry.

I'd really like to see 4 issues created for the above topics and closed out - before we attempt #1191 as a project.

[martenson]

With TS2.0 out I am closing this issue since we are not likely to work in its predecessor.

At the moment visualization framework is not being actively considered for a rewrite.