Component: IPyWidgets

IPyWidgets (both the default and custom widgets) are supported.

TOC

• Kernel Syncing
• Loading 3rd party source
• Debugging this mess

Architecture

IPyWidgets in general is a very complex piece of functionality hence requires a lot of moving parts and work to get it working in VS Code. Following is a summary of what's involved in adding support for IPyWidgets in VS Code:

• The sample project https://github.com/jupyter-widgets/ipywidgets/blob/master/examples/web3 is used as a basis for hosting Widgets in VS Code. The source for that repo can be found here https://github.com/microsoft/vscode-jupyter-ipywidgets. More information about that repo can be found in the corresponding README.md file.
• The Widget Manager from Jupyter Lab is hosted in the UI layer (renderer).
• Introduce a kernel in the UI layer (renderer). Widget Managers are responsible for rendering widgets, and they require access to the kernel (Kernel.IKernelConnection). In VS Code the real kernel (Kernel.IKernelConnection) is hosted in the extension host process which does not have access to the UI layer (dom/renderer). Thus, a kernel needs to be re-introduced in the UI layer.
• Synchronize kernel messages between the extension host & renderer process. Essentially the kernel in the UI layer is no different from the kernel in the extension host. Widgets can use this same kernel (Kernel.IKernelConnection) to execute code against the backend kernel and use all of its methods.
• Base Widgets sources are bundled and shipped with the VS Code extension. All built-in Widgets such as sliders, textboxes, and the like work out of the box.
• The UI layer (renderer) is sandboxed and does not allow execution and loading of resources from any arbitrary resource. All 3rd party widget scripts are downloaded and saved into a specific location and then loaded into the UI layer (renderer).
• 3rd party scripts are loaded from the following 3 sources:
  • CDN (jsdelivr, unpkg)
  • Local file system
  • Remote Jupyter Server

Legend (for diagrams)

• Widget Manager - This is the Jupyter Lab Widget manager found in the following package @jupyter-widgets/jupyterlab-manager
• Jupyter Lab Kernel - This is the Kernel connection interface found here Kernel.IKernelConnection in @jupyter-widgets/jupyterlab-manager
• Jupyter Notebook - This is the Notebook (loosely refers to the UI component & all of the corresponding backend components) in Jupyter Notebooks associated with a single notebook file.
• Proxy Web Socket - Getting access to the Kernel connection interface Kernel.IKernelConnection from @jupyter-widgets/jupyterlab-manager requires a WebSocket class. As VS Code doesn't always start Jupyter Servers, we end up providing an object that looks like a Web Socket object (hence the term proxy web socket). VS Code starts the kernel process and communites with it using zmq using the encode and decode methods found in @nteract/messaging/lib/wire-protocol

Kernel Syncing

Why do we need to do this? IPyWidgets require talking to a kernel within the UI. Since our kernel actually resides on the Extension side, we need to forward messages from our real kernel to a dummy kernel on the UI side:

We achieve this syncing by replacing the websocket in each kernel and making each one wait for the other.

Under the covers this is done with two classes and bunch of messages between them.

On the UI side there's a file here:

• kernel.ts

This contains a ProxyKernel that listens for messages from the extension. It forwards these messages onto a real kernel implementation, making sure its state matches the state of the kernel on the extension side.

On the Extension side there's two files:

• ipyWidgetMessageDispatcher.ts
• JupyterWebSocket

The IPyWidgetMessageDispatcher takes messages from the extension's kernel and forwards them onto the UI side. It accomplishes this by adding a couple of hooks into the JupyterWebSocket.

• Send Hook - this hook listens to when the extension is sending a message to the real kernel. This hook allows us to setup the kernel on the UI side to have the same set of futures that we have on the extension side. This is necessary so that message hooking on the UI side finds those futures. Message hooks are used in a number of ipyWidgets to control data and where it's displayed.
• Receive Hook - this hook listens to messages that come back from the real kernel but before the extension receives them. We send each message to the UI side so that any listeners for messages in the UI will get them before we use them in the extension.

Loading 3rd party source

Besides kernel syncing, IPyWidgets requires loading JS files into the UI that are not shipped with our extension. We handle this with some special hooks when we create our UI side WidgetManager.

These hooks essentially do the following:

1. Try loading the widget source from memory. This works for the standard jupyter widgets.
2. Try loading the widget source from a known CDN (as of right now we're using https://www.jsdelivr.com/ & https://unpkg.com) if the user is okay with that.
3. Try loading the widget from where jupyter finds it as best we can (this has problems with dependencies not getting loaded)
4. Try loading the widget from remote Jupyter Notebook (/nbextensions//index.js). Though this doesn't work with Jupyter Labs.

Loading Widgets and registering with requirejs

Diagram

# To recreate, open an issue and paste this code into the issue. Wikis don't support mermaid yet but issues do. ```mermaid sequenceDiagram participant WidgetManager participant Notebook (WebView) Notebook (WebView)->>WidgetManager: Render Widget WidgetManager->>Notebook (WebView): Request Widget Source Notebook (WebView)->>IPyWidgetScriptSource: Request Widget Source IPyWidgetScriptSource<<->>ScriptUriConverter: Convert to Webview URI IPyWidgetScriptSource->>Notebook (WebView): Return converted Widget Source loop RequireJs Notebook (WebView)->>Notebook (WebView): Register Scripts with requirejs end Notebook (WebView)->>WidgetManager: Return Widget Source loop Load Widget WidgetManager->>WidgetManager: Load using requirejs end Note right of WidgetManager: If widget is not
registered, then
loading will fail,
and error displayed. ```

Loading from various sources

Diagram

```mermaid sequenceDiagram participant IPyWidgetScriptSource participant CDN participant LocalFS participant RemoteJupyter IPyWidgetScriptSource->>CDN: Request Widget Source loop Look for Widget CDN->>CDN: jsDelivr.com CDN->>CDN: unpkg.com end CDN->>IPyWidgetScriptSource: Return Widget Source Note right of IPyWidgetScriptSource: If we got everything
return to WebView.
Else try others. IPyWidgetScriptSource->>LocalFS: Request Widget Source loop Look for Widget LocalFS->>LocalFS: Look for widgets in folder. end Note right of LocalFS: If widgest are found
copy .js to
/tmp
folder. Requirement
of WebView LocalFS->>IPyWidgetScriptSource: Return Widget Source Note right of IPyWidgetScriptSource: If we got everything
return to WebView.
Else try others. IPyWidgetScriptSource->>RemoteJupyter: Request Widget Source loop Look for Widget RemoteJupyter->>RemoteJupyter: Look for widgets in
/nbextension//index.js end RemoteJupyter->>IPyWidgetScriptSource: Return Widget Source ```

Notes:

• The widget sources located from (CDN/localFS/remote jupyter) needs to be registered with requirejs.
  • IPyWidgets uses requirejs to locate/load widget sources.
• Loading from local file system (widgets located in <python sysprefix>/share/jupyter/nbextensions/<widget>) does not always work due to the fact that:
  • Widgets might have other dependencies defined in extension.js. (extension.js is the main entrypoint for the widget UI in jupyter. It usually has an index.js as well)
  • Widgets might have style sheets embedded in extension.js file.
  • Loading extension.js is not possible due to the following reasons:
    • It is specific to jupyter lab/notebooks.
    • These files can add module dependencies only available in jupyter lab/notebooks require(["jupyterlab/ui", etc])
    • These files expect DOM elements to be available that are available in jupyter lab/notebooks.
    • Some widgets do not have plain js in extension.js, they have js as strings that gets evaluated.
• Loading from remote Jupyter server doesn't work in the case of Jupyter Labs.
  • Jupyter labs creates new bundles as and when widgest (python packages) are installed.

Debugging this mess

If for some reason ipywidgets isn't working, the way to debug this is to :

1. Setup logpoints in the extension side in either jupyterNotebook.ts onHandleIOPub or in the send/receive hooks setup in ipyWidgetDispatcher. (The ipyWidgetDispatcher hooks are at a lower level)
2. Setup similar logpoints in the kernel.ts file on the UI side
3. Run your scenario
4. Compare the messages each side gets and the order.

Alternatively you might also run 'jupyter lab' or 'jupyter notebook' with the same code and see what the difference is their implementation. Generally you can set breakpoints in chrome in their kernel implementation too.

Architecture

Due to the complex nature of IPyWidgets, writing highlevel unit tests isn't sufficient. We have end to end tests that test the following functionality:

• Rendering widgets
• Interacting with interactive widgets (verifying updates to the UI)
• Rendering & interaction of complex 3rd party widgets such as matplotlib, ipyvolume, ipysheets, etc.

Note: A large number of the tests haven't been migrated to support the new Notebook UI in vscode. This is currently a WIP.

Outdated

Basic classes involved

There are some basic parts involved in IPyWidgets support

Extension side

On the extension side, there's a number of classes involved in talking to jupyter:

IPyWidgets is supported by rewiring the WebSocket in the picture above like so:

This websocket rerouting allows us to intercept both inbound and outbound messages to the kernel.