[BUG?] Versions >=0.4.0 cannot be installed using only conda-forge channels

[willGraham01]

Comment:

Version >=0.4.0 is noarch, and depends on tensorflow >=2.5.0.

However, only version 1.14.0 of tensorflow is available via conda-forge for Windows OSs (this appears to conflict with the claims of the tensorflow-feedstock badges though, but conda cannot locate any versions on Windows above 1.14.0).

This means that any packages to be uploaded to conda-forge that require cellfinder-core >=0.4.0 have unresolvable dependencies on Windows.

Workarounds
For conda-forge recipes: forcing Windows to use cellfinder-core<0.4.0 will bypass the issue but leave the user with a broken package, unless they follow the step below to install a compatible version of tensorflow first. This also makes any recipes illegible for noarch.

A compatible version of tensorflow can be installed via pip on Windows, so it is possible to install cellfinder-core>=0.4.0 manually by:

• pip install tensorflow
• conda install cellfinder-core>=0.4.0

But of course, conda-forge recipes cannot use non-conda channels, so this is not available when uploading new recipes to staged-recipes.

[adamltyson]

Based on this page it looks like TensorFlow is kinda no-longer supported on Windows (at least using a GPU). Can we specify the cpu version for Windows (and is that available?).

[willGraham01]

We can't install any tensorflow version above 2.0.0 using only conda channels on Windows, and cellfinder-core requires at least v2.5.0. In turn, it means that anything that depends on cellfinder-core>=0.4.0 can't be conda-installed on Windows, since conda installs can only use conda channels to fetch dependencies.

There's this hack that I've employed for the meta-package that (seems) to work; since cellfinder-core v3.1.0 didn't actually install tensorflow, so conda can resolve the dependencies and "install" the package. Then the manual pip-install trick I described needs be done by users manually to get it to work. Far from an ideal solution.

[hmaarrfk]

Anything that has a hard dependency on tensorflow cannot be installed using only conda-forge.

To fix this, we need to build tesnsorflow at conda-forge, but this isn't really easy.

PRs welcome at the tensorflow feedstock.

[adamltyson]

@willGraham01 I think we can make TF optional for now. It will also make downloads much quicker for most users (TF is only used for cellfinder).

[willGraham01]

I think we can make TF optional for now. It will also make downloads much quicker for most users (TF is only used for cellfinder).

Conda-forge doesn't allow optional dependencies - TF is either in or out. TF is also a hard dependency of cellfinder-core on PyPI, which we're pulling from here.

Couple of options:

• I can make another package like cellfinder-core-no-TF and upload that to the forge if we want? BUT we'd be allowing users to install a broken package into their environment if they're not careful. And we'd be unable to run any kind of tests in the corresponding recipe since they'd all fail at the import / pip check step.
• We modify the cellfinder-core pypi package to have an optional tensorflow dependency; cellfinder-core[tensorflow], and then make the conda package the one that only installs the base package? Although in doing that we'd also need to update cellfinder-napari and cellfinder's package info too.

Happy to proceed with either option. The latter is safer but will be touching a lot of our existing packages (and might temporarily break things). Former option is quick but not very safe (and I'm not sure what the conda-forge review team will say if we put a package up with no tests!)

[adamltyson]

We modify the cellfinder-core pypi package to have an optional tensorflow dependency; cellfinder-core[tensorflow], and then make the conda package the one that only installs the base package? Although in doing that we'd also need to update cellfinder-napari and cellfinder's package info too.

I think this is probably the best. None of the options seem particularly satisfactory though!

[willGraham01]

I think this is probably the best. None of the options seem particularly satisfactory though!

Sadly the real blocker appears to be conda-forge itself. Everything we have is completely fine if pip install is used instead!

Am going to move/link this issue onto brainglobe/BrainGlobe so that all the required changes are recorded in one place, then will get started on the necessary changes 👍

[dstansby]

What about reverting back from being a noarch package to just uploading packages for Linux and macOS? In retrospect I think this is what I should have done when I released version 0.4.0.

[adamltyson]

Would that make all the upstream dependencies linux and macOS only though?

[dstansby]

You could specify platform-dependent dependencies in dependent packages, e.g. in a brainglobe meta package you could specify something like - cellfinder-core # [not win]

[willGraham01]

I guess that also leaves certain users (potentially unknowingly) without some BrainGlobe tools after installing the meta-package - also relevant is Adam's comment here.

I guess it's which evil do you want:

• conda-installing doesn't fetch all the tools, but the ones you get all work.
• conda-installing fetches all the tools, but some are broken until you manually install something else.

[adamltyson]

conda-installing fetches all the tools, but some are broken until you manually install something else.

This gets my vote. Most users will be using BrainGlobe tools via napari plugins, so this would mean the tools are visible, and we can provide explicit instructions on how to get them working.

[hmaarrfk]

You may want to consider noarch_paltforms to have the best of both world:
https://conda-forge.org/docs/maintainer/knowledge_base.html#noarch-packages-with-os-specific-dependencies

Discussion here:
conda-forge/conda-forge.github.io#1840

[willGraham01]

Thanks for the suggestion Mark!

noarch_platforms look to resolve our problems here (see #14) and address some of our tensorflow woes at the same time.

With #14, cellfinder-core is still be un-importable on Windows due to missing tensorflow, but we're already prepared to acknowledge this:

• Forbid import when tensorflow is not installed brainglobe/cellfinder-core#186, introduced changes to guard against this possibility anyway.
• Update README in light of possible noarch build brainglobe/cellfinder-core#191, we can alert Windows users trying to install via conda to this problem.

So with these changes to cellfinder-core, and new PyPI upload, #14 will close this issue, albeit with the Windows caveat.