Intellisense for notebooks

The new and improved way

This page describes how intellisense works for notebooks (and the interactive window) when the following experiment is enabled:

  "python.pylanceLspNotebooksEnabled": true

If that setting is false, the old way is used instead.

Table of Contents

Overview

Notebooks Architecture

Concatentation of the notebook cells

New LSP Messages

Pylance can now understand notebooks

Active interpreter?

• Custom API

Tooling

Intellisense Overview

Intellisense in VS code works by sending LSP requests to a separate process (well in most cases, see this for more info)

Something like so:

Intellisense for notebooks

Intellisense for notebooks works pretty much the same way but with each cell of a notebook being a text document:

Concatentation of the notebook cells

This poses a problem for the language server (Pylance) because code from one cell can be referenced in another.

Example:

In that example, the pandas import crosses the cell boundary.

This means pylance cannot just analyze each cell individually.

New LSP Messages

The solution was changes to LSP:

Message	When it is sent	What it's for
DidOpenNotebookDocument	On notebook open	Sends the cells and their contents on open
DidChangeNotebookDocument	On moving, deleting or adding cells	For cell changes, moves, deletes, and adds
DidSaveNotebookDocument	On notebook save	When the notebook is saved its relative path is established. This is necessary to determine location with respect to imports.
DidCloseNotebookDocument	On notebook close	Server can clean up in memory state about the notebook.

Pylance can now understand notebooks

These 4 new messages means that pylance knows the relative order of all of the cells in a notebook. Here's generally the sequence of events:

From this pylance knows that:

• There are 3 cells
• They belong to the notebook foo.ipynb
• Their order is 3, 1, 2

So if the notebook looked like so:

Internally pylance could have a concatenated document like so (pylance may or may not concat the cells):

# Cell 1
import pandas as pd
# Cell 2
df = pd.read_csv('./data/test.csv')
# Cell 3
df.head()

Active interpreter?

There's still one missing part in this design. How does pylance know what interpreter to use?

When Pylance is started, the Python extension tells it what the pythonPath is through the workspace/configuration message:

But a notebook uses a kernel. The kernel isn't tied to the global pythonPath.

Custom API

To workaround this problem, pylance sends the workspace/configuration for every notebook opened.

The Python extension handles this message and then asks if it knows the pythonPath for a URI.

If Jupyter has a kernel active for a notebook, it returns the known pythonPath for that kernel.

Tooling

Sometimes things go wrong with intellisense. And one of the most difficult parts of diagnosing these failures is reproing the problem.

Pylance logs a lot of data about each document and their changes in this output window here:

This window is almost like a history window. It contains the state of a notebook and every change that was typed into it.

Pylance log replay tool (old mode)

If you're in old mode, there's a separate language server output for each kernel path. Example:

You can ask the user to turn on full logging for this window with these settings:

  "notebook-intellisense.logLevel": "Trace",
  "notebook-intellisense.trace.server.verbosity": "Verbose",

The output in this window constitutes the log of everything sent to the language server. We can use this to replay what happened to a notebook to try and recreate the state that caused a bug.

To replay the log:

• Save output of Pylance to a .log file
• Build Jupyter extension locally
• Create an empty notebook
• Name it the same as the notebook in the original issue (you can generally find the name in the cell URIs in the log)
• Run the Jupyter (Dev): Replay Pylance Log command
• Open the log you saved
• This should fill in all the cells that were in the original file. Path matters so you might have to replicate the user's path.
• Run the Jupyter (Dev): Step Pylance Log command over and over

Theoretically it will make all the edits to the notebook that the user made. (Edit : it doesn't work with newer VS code because fragment URIs have changed, it uses this to compute indices)

Adapting to new mode

The replay tool understands textDocument/didOpen and textDocument/didChange events. If it was modified to understand the notebook messages, it could theoretically replay logs from customers in the 'new' mode for notebook intellisense without the fragment problems because the notebook messages should be enough to fully recreate the notebook.