Check plugins for dependencies requiring compilation and mark them

[jni]

One of the issues flagged in napari/napari#3223 is that we will always have issues installing plugins that depend on a compilation step, because they don't provide a wheel or (in the future) a conda-forge package.

@tlambert03 writes:

it's very unlikely that we (napari) are going to be able to swoop in and make it all smooth for beginner napari plugin developers. I'd say the best thing you can do to help at this point is encourage developers to consider their dependencies carefully.

To scale such encouragement, napari-hub could systematically attempt to either introspect plugin dependencies (this should be easier with our new setup.cfg-based cookiecutter), OR to install each plugin on a vanilla machine on each OS (ie one that doesn't have developer tools installed). If installation fails, this would be marked on the plugin page. Plugin developers would be provided instructions on how that mark could be removed.

The whole process needs to be carefully designed so that this is indeed encouragement rather than discouragement. 😂

CC @sofroniewn @tlambert03

[tlambert03]

See also #331

[DragaDoncila]

To scale such encouragement, napari-hub could systematically attempt to either introspect plugin dependencies (this should be easier with our new setup.cfg-based cookiecutter)

This would actually be easy to do even with "old" setup.py plugins on the preview page, as we build the wheel and can therefore trivially inspect dependencies.

OR to install each plugin on a vanilla machine on each OS (ie one that doesn't have developer tools installed). If installation fails, this would be marked on the plugin page.

This would probably be useful regardless of whether we introspect dependencies or not as there's always going to be gremlins we can't foresee.

Plugin developers would be provided instructions on how that mark could be removed.

This is the bit that I think is trickier. I feel like at most we could provide a link to the failing workflow run, and maybe also a link to a workflow file they could use themselves to mimic the failure. We could possibly support custom instructions for some very common failures if we know how to fix them, but I worry that would be a big can of worms.

The whole process needs to be carefully designed so that this is indeed encouragement rather than discouragement. 😂

I agree here to an extent. I think we should provide opportunities for the developers to fix these issues before surfacing them to users, and provide as many tools as we can to help the developers investigate these issues, but as the plugin ecosystem grows I think we'll start needing to be prescriptive e.g. flagging plugins if they consistently fail installs for multiple releases.

I think the preview page can be a good middle ground for communicating these issues to developers before we flag them to users.

[tlambert03]

I agree here to an extent. I think we should provide opportunities for the developers to fix these issues before surfacing them to users, and provide as many tools as we can to help the developers investigate these issues, but as the plugin ecosystem grows I think we'll start needing to be prescriptive e.g. flagging plugins if they consistently fail installs for multiple releases.

+1 ... don't need to be rude about it, but also don't need to be delicate about it. If a plugin is more or less guaranteed to fail in certain scenarios. I see no reason to sugar coat it. Send them a failing log, offer to help them fix it if they're lost, and then flag it visibly on the hub and in the app

[psobolewskiPhD]

Does enabling a way to do the plugin pip install as --no-deps make sense in some of these cases? Obviously combined with a warning for the user of course. Something that informs that (compiled) depends need to be installed/provided by the user and just the plugin is being installed.
This is apropos things like tensorflow, pyopencl etc that often require a bit to install due to complex dependencies and pins. See:
stardist/stardist-napari#3
On a related note, such an option could be useful when updating an existing plugin; it would allow the user to just say that the depends are there and get on with it. E.g. heres's a stardist-napari issue I ran into when updating the plugin:
stardist/stardist-napari#5

[tlambert03]

That feels like a recipe for problems. If you let a gui user install with no deps, they're almost guaranteed to get a crash when they actually run the program. If they have to go to the command line to install the deps, they might as well just go to the command line to install the plugin itself too. Same goes for updates... if the plugin has changed its dependencies it will crash

[psobolewskiPhD]

True true, I guess I forget that the plugins can also be installed from the command line using pip. Thanks.

[neuromusic]

OR to install each plugin on a vanilla machine on each OS (ie one that doesn't have developer tools installed). If installation fails, this would be marked on the plugin page. Plugin developers would be provided instructions on how that mark could be removed.

does this approach (explicitly trying a vanilla install on each OS) effectively cover the scenarios that #331 addresses?

[tlambert03]

Yeah, I think this is certainly a good brute force way to start. may catch some things that a more "algorithmic" approach (#331) might miss. Would be nice to run this on all OSes too, and make sure that there's no compiler in PATH

[neuromusic]

yeah, that's my thought. though the algorithmic approach could be helpful as a utility for devs (so they can see what dependencies might cause issues for users)

[DragaDoncila]

I think a general message of "this plugin may not be installable" is better than nothing, but ideally we'd be able to point to specific reasons for it - like the compiled dependencies. An experienced user might not be deterred by a "This plugin has a dependency that requires compiling C code and may not be trivial to install" message, but they may not even consider the plugin if it's just a general warning.

I guess what I'm saying is we definitely want to try clean installs on the different OSs and flag if that fails, but we should also take the time to implement specific checks if we know of practices that can cause issues, and have a conceivable algorithm that can surface them (like the compiled dependencies).

a utility for devs (so they can see what dependencies might cause issues for users)

Definitely +1 to this.

[tlambert03]

here is a workflow (borrowing @DragaDoncila's plugin matrix json strategy 🙏) that I made to try to install every plugin and do some basic discovery and health checks:

• does the plugin install?
• is the plugin actually detected by napari.plugins.plugin_manager
• if it has a dock widget is it at least a callable? (silly test for now, but wanted to be conservative)
• does it not potentially mess up the environment by installing its own backend

About a third (35/111) of plugins fail one or more of these tests on mac and/or windows (Note, I haven't yet disabled dev tools, so some of these may actually be successfully building sdists, and this might be an optimistic summary). That's a bit disappointing. Some might have custom install instructions (conda, etc.), that would make them work, but they would fail in the builtin installer. As we promote rapid adoption of napari, I would love to see us put more resources into keeping the expected success rate up!

expected to fail on both

napari-mahotas-image-processing
PlatyMatch
napari-lazy-openslide
napari-labelimg4classification
napari-micromanager
napari-tomoslice
napari-webcam
napari-cupy-image-processing
napari-spacetx-explorer
napari-deepmeta
napari-clusters-plotter
napari-DeepSpot
napari-time-slicer
napari-simpleitk-image-processing
napari-live-recording
napari-omero
napari-mri
napari-pyclesperanto-assistant
napari-subboxer
napari-sdeconv
napari-imc
napari-script-editor
napari-segment-blobs-and-things-with-membranes
napari-yapic-prediction
napari-layer-details-display

expected to fail on mac

brainreg-segment
misic-napari-plugin
devbio-napari
napari-allencell-segmenter
napari-apr-viewer
napari-accelerated-pixel-and-object-classification
napari-lfdfiles
napari-nucleaizer

expected to fail on windows

napari-autolign
napari-clemreg

[DragaDoncila]

Thanks for putting this together so quickly @tlambert03. I think using this action (or some iteration of it) would be a very clean solution as the "backend" for surfacing these errors on the plugin preview page, though I also think we should promote this as a separate action the community can use to find these things out before they go to release.

[tlambert03]

I also think we should promote this as a separate action the community can use to find these things out before they go to release.

fully agree!