Intellisense for notebooks

The new and improved way

This page describes how intellisense works for notebooks (and the interactive window) when the following experiment is enabled:

  "python.pylanceLspNotebooksEnabled": true

If that setting is false, the old way is used instead.

Table of Contents

Overview

Notebooks Architecture

Concatentation of the notebook cells

New LSP Messages

Pylance can now understand notebooks

Active interpreter?

• Custom API

Tooling

Intellisense Overview

Intellisense in VS code works by sending LSP requests to a separate process (well in most cases, see this for more info)

Something like so:

Intellisense for notebooks

Intellisense for notebooks works pretty much the same way but with each cell of a notebook being a text document:

Concatentation of the notebook cells

This poses a problem for the language server (Pylance) because code from one cell can be referenced in another.

Example:

In that example, the pandas import crosses the cell boundary.

This means pylance cannot just analyze each cell individually.

New LSP Messages

The solution was changes to LSP:

Message	When it is sent	What it's for
DidOpenNotebookDocument	On notebook open	Sends the cells and their contents on open
DidChangeNotebookDocument	On moving, deleting or adding cells	For cell changes, moves, deletes, and adds
DidSaveNotebookDocument	On notebook save	When the notebook is saved its relative path is established. This is necessary to determine location with respect to imports.
DidCloseNotebookDocument	On notebook close	Server can clean up in memory state about the notebook.

Pylance can now understand notebooks

These 4 new messages means that pylance knows the relative order of all of the cells in a notebook. Here's generally the sequence of events:

From this pylance knows that:

• There are 3 cells
• They belong to the notebook foo.ipynb
• Their order is 3, 1, 2

So if the notebook looked like so:

Internally pylance could have a concatenated document like so (pylance may or may not concat the cells):

# Cell 1
import pandas as pd
# Cell 2
df = pd.read_csv('./data/test.csv')
# Cell 3
df.head()

Active interpreter?

There's still one missing part in this design. How does pylance know what interpreter to use?

When Pylance is started, the Python extension tells it what the pythonPath is through the workspace/configuration message:

But a notebook uses a kernel. The kernel isn't tied to the global pythonPath.

Custom API

To workaround this problem, pylance sends the workspace/configuration for every notebook opened.

The Python extension handles this message and then asks if it knows the pythonPath for a URI.

If Jupyter has a kernel active for a notebook, it returns the known pythonPath for that kernel.

Tooling

Sometimes things go wrong with intellisense. And one of the most difficult parts of diagnosing these failures is reproing the problem.

Pylance logs a lot of data about