Component: IPyWidgets

IPyWidgets (both the default and custom widgets) are supported.

TOC

• Basic classes involved
  • Extension side
• Kernel Syncing
• Loading 3rd party source
• Debugging this mess

Basic classes involved

There are some basic parts involved in IPyWidgets support

Extension side

On the extension side, there's a number of classes involved in talking to jupyter:

IPyWidgets is supported by rewiring the WebSocket in the picture above like so:

This websocket rerouting allows us to intercept both inbound and outbound messages to the kernel.

Kernel Syncing

Why do we need to do this? IPyWidgets require talking to a kernel within the UI. Since our kernel actually resides on the Extension side, we need to forward messages from our real kernel to a dummy kernel on the UI side:

We achieve this syncing by replacing the websocket in each kernel and making each one wait for the other.

Under the covers this is done with two classes and bunch of messages between them.

On the UI side there's a file here: https://github.com/microsoft/vscode-python/blob/dd82412ef119247a4fee2c3e045996a3319d8768/src/datascience-ui/ipywidgets/kernel.ts#L22

This contains a ProxyKernel that listens for messages from the extension. It forwards these messages onto a real kernel implementation, making sure its state matches the state of the kernel on the extension side.

On the Extension side there's two files: https://github.com/microsoft/vscode-python/blob/dd82412ef119247a4fee2c3e045996a3319d8768/src/client/datascience/ipywidgets/ipyWidgetMessageDispatcher.ts#L24 https://github.com/microsoft/vscode-python/blob/dd82412ef119247a4fee2c3e045996a3319d8768/src/client/datascience/jupyter/jupyterWebSocket.ts#L14

The IPyWidgetMessageDispatcher takes messages from the extension's kernel and forwards them onto the UI side. It accomplishes this by adding a couple of hooks into the JupyterWebSocket.

• Send Hook - this hook listens to when the extension is sending a message to the real kernel. This hook allows us to setup the kernel on the UI side to have the same set of futures that we have on the extension side. This is necessary so that message hooking on the UI side finds those futures. Message hooks are used in a number of ipyWidgets to control data and where it's displayed.
• Receive Hook - this hook listens to messages that come back from the real kernel but before the extension receives them. We send each message to the UI side so that any listeners for messages in the UI will get them before we use them in the extension.

Loading 3rd party source

Besides kernel syncing, IPyWidgets requires loading JS files into the UI that are not shipped with our extension. We handle this with some special hooks when we create our UI side WidgetManager.

These hooks essentially do the following:

1. Try loading the widget source from memory. This works for the standard jupyter widgets.
2. Try loading the widget source from a known CDN (as of right now we're using https://www.jsdelivr.com/ & https://unpkg.com) if the user is okay with that.
3. Try loading the widget from where jupyter finds it as best we can (this has problems with dependencies not getting loaded)
4. Try loading the widget from remote Jupyter Notebook (/nbextensions//index.js). Though this doesn't work with Jupyter Labs.

Loading Widgets and registering with requirejs

Diagram

```mermaid sequenceDiagram participant WidgetManager participant Notebook (WebView) Notebook (WebView)->>WidgetManager: Render Widget WidgetManager->>Notebook (WebView): Request Widget Source Notebook (WebView)->>IPyWidgetScriptSource: Request Widget Source IPyWidgetScriptSource->>Notebook (WebView): Return Widget Source loop RequireJs Notebook (WebView)->>Notebook (WebView): Register Scripts with requirejs end Notebook (WebView)->>WidgetManager: Return Widget Source loop Load Widget WidgetManager->>WidgetManager: Load using requirejs end Note right of WidgetManager: If widget is not
registered, then
loading will fail,
and error displayed. ```

Loading from various sources

Diagram

```mermaid sequenceDiagram participant IPyWidgetScriptSource participant CDN participant LocalFS participant RemoteJupyter IPyWidgetScriptSource->>CDN: Request Widget Source loop Look for Widget CDN->>CDN: jsDelivr.com CDN->>CDN: unpkg.com end CDN->>IPyWidgetScriptSource: Return Widget Source Note right of IPyWidgetScriptSource: If we got everything
return to WebView.
Else try others. IPyWidgetScriptSource->>LocalFS: Request Widget Source loop Look for Widget LocalFS->>LocalFS: Look for widgets in folder. end Note right of LocalFS: If widgest are found
copy .js to
/tmp
folder. Requirement
of WebView LocalFS->>IPyWidgetScriptSource: Return Widget Source Note right of IPyWidgetScriptSource: If we got everything
return to WebView.
Else try others. IPyWidgetScriptSource->>RemoteJupyter: Request Widget Source loop Look for Widget RemoteJupyter->>RemoteJupyter: Look for widgets in
/nbextension//index.js end RemoteJupyter->>IPyWidgetScriptSource: Return Widget Source ```

Notes:

• The widget sources located from (CDN/localFS/remote jupyter) needs to be registered with requirejs.
  • IPyWidgets uses requirejs to locate/load widget sources.
• Loading from local file system (widgets located in <python sysprefix>/share/jupyter/nbextensions/<widget>) does not always work due to the fact that:
  • Widgets might have other dependencies defined in extension.js.
  • Widgets might have style sheets embedded in extension.js file.
  • Loading extension.js is not possible due to the following reasons:
    • It is specific to jupyter lab/notebooks.
    • These files can add module dependencies only available in jupyter lab/notebooks require(["jupyterlab/ui", etc])
    • These files expect DOM elements to be available that are available in jupyter lab/notebooks.
    • Some widgets do not have plain js in extension.js, they have js as strings that gets evaluated.
• Loading from remote Jupyter server doesn't work in the case of Jupyter Labs.
  • Jupyter labs creates new bundles as and when widgest (python packages) are installed.

Debugging this mess

If for some reason ipywidgets isn't working, the way to debug this is to :

1. Setup logpoints in the extension side in either jupyterNotebook.ts onHandleIOPub or in the send/receive hooks setup in ipyWidgetDispatcher. (The ipyWidgetDispatcher hooks are at a lower level)
2. Setup similar logpoints in the kernel.ts file on the UI side
3. Run your scenario
4. Compare the messages each side gets and the order.

Alternatively you might also run 'jupyter lab' or 'jupyter notebook' with the same code and see what the difference is their implementation. Generally you can set breakpoints in chrome in their kernel implementation too.