Use Markdown or Python for writing Jupyter Notebooks

[amotl]

About

Markdown as the lingua franca for many technical writing tasks should be used more, as it is roughly interoperable with, for example, GitHub and Discourse. ¹

Conversion from Jupyter Notebooks

Just use nbconvert.

pip install nbconvert
jupyter nbconvert --to markdown automl_classification_with_pycaret.ipynb

Authoring Jupyter Notebooks

Instead of converting from, Jupyter Notebooks can be written in Markdown itself, see Notebooks written entirely in Markdown.

The easiest way to create a MyST notebook is to use Jupytext, a tool that allows for two-way conversion between .ipynb and a variety of text files. See also Notebooks as Markdown.

Footnotes

1. Also with HubSpot, when throwing https://github.com/crate-workbench/hubspot-tech-writing into the mix. ↩

[amotl]

Problem

When looking at this patch, it is immediately clear that the slight obfuscation added through the JSON format Jupyter Notebooks are stored in, properly reviewing diffs effectively becomes impossible, because they can't be deciphered well.

That other one isn't unreadable, but still it demonstrates that this form of authoring and editing easily attracts noise.

Solution

I think we should level up here and write Notebooks in Markdown in the future, using, for example, Jupytext.

/cc @marijaselakovic, @ckurze, @karynzv

[ckurze]

Sounds like a viable approach, without having reviewed the technologies linked in https://github.com/crate/tech-writing/issues/414.

Will it still be possible to link directly to these "markdowned" notebooks and open them in Google Colab, for example?

[amotl]

Hi. Thanks for also mentioning that other page about Quarto, Hex, and Marimo. Sure enough, those [commercial] offerings can also be evaluated for publishing content.

Here, we are mostly looking at solutions coming from the Jupyter FOSS communities, pretty close to baseline Jupyter, Binder, and Executable Books technologies.

Will it still be possible to link directly to these "markdowned" notebooks and open them in Google Colab, for example?

This is a very good point: It will probably not be possible to have that feature out of the box. We like that convenience very much, so rest assured we will provide a solution for that when switching over to an alternative authoring workflow.

Here, we will need to establish a little machinery to also provide/publish the rendered variants when the source file
has changed, i.e. as soon as we will start authoring notebooks like suggested here, using Jupytext and friends.

[amotl]

Thanks for also mentioning that other page about Quarto, Hex, and Marimo. Those commercial offerings can also be evaluated, but here, we are mostly looking at corresponding FOSS solutions.

While we started the discussion by referencing Jupytext, we would like to not forget mentioning Marimo once more: Contrary to Quarto and Hex, it is FOSS, and ticks roughly the same boxes we have on our list of requirements for improved Notebook authoring:

reproducible, git-friendly, executable as a script, and shareable as an app.

We wil rephrase the title of this issue accordingly, and will be happy about any relevant feedback. We think it should be considered and evaluated in the same spirit as we would do by exercising a corresponding authoring workflow using Jupytext.

[amotl]

Use Markdown or Python for writing Jupyter Notebooks.

All of those are right on the spot in one way or another, exploring both Marimo and MyST Notebooks, and relevant addons and extensions.

• Marimo: Basic notebook querying CrateDB #774
• CFR job statistics and slow-query exploration: First draw of marimo notebook cratedb-toolkit#350
• Add MyST-NB for rendering notebooks, and JupySQL for running SQL crate-docs-theme#566
• Notebook/Jupyter: Add example using JupySQL #797