Installation of pywwt breaks JupyterLab under TLJH (The Littlest Jupyter Hub)

[hickst]

Using the successful formula from recent fixes (#272), I tried to install pywwt in the TLJH implementation of JupyterHub (https://tljh.jupyter.org/en/latest/index.html). Unfortunately, pywwt installs its own version of JupyterLab (and related libraries) and something conflicts with the versions in TLJH, causing JupyterLab to stop working.

Why does pywwt include JupyterLab in the first place?

Is there a version of pywwt that does not include the JupyterLab and ipywidget (and related) libraries?

[hickst]

The errant behavior is that executing the first cell of a newly spawned notebook hangs. After about 30 seconds the JL interface reports that you are no longer connected to a viable Python kernel (which seems like the kernel has crashed):

The only error that shows is in the log entry for the user (no errors in JH log or reverse proxy log):

Oct 26 18:49:46 hector.cyverse.org bash[11561]: [I 2020-10-26 18:49:46.766 SingleUserNotebookApp restarter:110] KernelRestarter: restarting kernel (4/5), new random ports
Oct 26 18:49:47 hector.cyverse.org bash[11561]: Traceback (most recent call last):
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/runpy.py", line 193, in _run_module_as_main
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     "__main__", mod_spec)
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/runpy.py", line 85, in _run_code
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     exec(code, run_globals)
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/site-packages/ipykernel_launcher.py", line 15, in <module>
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     from ipykernel import kernelapp as app
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/site-packages/ipykernel/__init__.py", line 2, in <module>
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     from .connect import *
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/site-packages/ipykernel/connect.py", line 13, in <module>
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     from IPython.core.profiledir import ProfileDir
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/site-packages/IPython/__init__.py", line 49, in <module>
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     from .terminal.embed import embed
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/site-packages/IPython/terminal/embed.py", line 18, in <module>
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     from IPython.terminal.interactiveshell import TerminalInteractiveShell
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/site-packages/IPython/terminal/interactiveshell.py", line 16, in <module>
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     from prompt_toolkit.document import Document
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/site-packages/prompt_toolkit/__init__.py", line 16, in <module>
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     from .interface import CommandLineInterface
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/site-packages/prompt_toolkit/interface.py", line 19, in <module>
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     from .application import Application, AbortAction
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/site-packages/prompt_toolkit/application/__init__.py", line 1, in <module>
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     from .application import Application
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 41, in <module>
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     from prompt_toolkit.buffer import Buffer
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/site-packages/prompt_toolkit/buffer.py", line 9, in <module>
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     from .completion import Completer, Completion, CompleteEvent
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/site-packages/prompt_toolkit/completion/__init__.py", line 1, in <module>
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     from .base import (
Oct 26 18:49:47 hector.cyverse.org bash[11561]:   File "/opt/tljh/user/lib/python3.7/site-packages/prompt_toolkit/completion/base.py", line 7, in <module>
Oct 26 18:49:47 hector.cyverse.org bash[11561]:     from prompt_toolkit.eventloop import generator_to_async_generator
Oct 26 18:49:47 hector.cyverse.org bash[11561]: ImportError: cannot import name 'generator_to_async_generator' from 'prompt_toolkit.eventloop' (/opt/tljh/user/lib/python3.7/site-packages/prompt_toolkit/eventloop/__init__.py)

[pkgw]

I believe that you're correct that the problem is that new, incompatible versions of the jupyterlab packages are getting installed.

The source of the issue isn't pywwt's dependency on these packages per se (it needs their libraries to create Jupyter widgets and whatnot), it's that a "conda" based installation will overwrite versions installed with "pip". In the TLJH case it seems that the installation only partially replaces some of the Jupyter-related packages, leading to internal inconsistencies and the crashes that you're experiencing.

I can imagine that TLJH might require specific or specialized versions of the core packages, so I think the safest strategy is to make sure not to overwrite those. This can be accomplished by making sure to install pywwt (and most other Python packages) with "pip" rather than "conda".

On the other hand, pywwt does have some dependencies that involve compiled code, and I think it's generally safer (and certainly faster) to install those through conda.

It took me a few tries so I haven't tested this incantation from a pristine environment, but I think that an installation plan along the following lines should get things going without causing any breakage:

sudo -E conda install astropy beautifulsoup4 lxml matplotlib pyqt qtpy
sudo -E conda install -c conda-forge reproject
sudo -E pip install ipyevents ipywidgets pywwt

Let me know if that helps.