big new image restructure

[shouples]

Reworks the image builds based on:

• https://docs.google.com/document/d/1y3G-BiB37KNOp_qHvLKquXKx2JkMXxaK_3Swl4627Jw/edit?pli=1
• https://www.figma.com/file/ByqWSGlYWY2IPZ3qpjTeQr/Kernel-spec-setup-%26-pod-creation?type=whiteboard&node-id=0-1&t=IuytY69O3VCTJrHg-0

Current images based on jupyter/datascience-notebook are using Python 3.9.x and R 4.2.x, but this will provide the following (all new aside from the ones marked with ✅):

graph LR
    python(Python)
    R(R)
    pythonBase(Base 3.9)
    pythonBaseGPU(Base 3.9 GPU)
    pythonDatascience(Datascience 3.9)
    pythonDatascienceGPU(Datascience 3.9 GPU)
    pythonNoteable("Noteable 3.9 ✅")
    pythonNoteableGPU("Noteable 3.9 GPU ✅")
    rBase(Base 4.3)
    rDatascience("Future: Datascience 4.3")
    rNoteable("Noteable 4.3 ✅")
    python --> pythonBase
    pythonBase --> pythonDatascience --> pythonNoteable
    pythonBase --> pythonBaseGPU --> pythonDatascienceGPU --> pythonNoteableGPU
    R --> rBase
    rBase --> rDatascience
    rDatascience --> rNoteable
    rBase --> rNoteable
    classDef dashedNode stroke-dasharray: 5, 5;
    class rDatascience dashedNode
    class pythonCustomerGPU dashedNode
    linkStyle 7 stroke-dasharray: 2 2;
    linkStyle 8 stroke-dasharray: 2 2;

Loading

To be extra clear, the image builds added in this PR are:

• Python 3.9 (base)

  • Python 3.9 (datascience)
    • Python 3.9 (noteable)
  • Python 3.9 (base-gpu)
    • Python 3.9 (datascience with GPU support)
      • Python 3.9 (noteable with GPU support)

• Python 3.10 (base)

  • Python 3.10 (datascience)
    • Python 3.10 (noteable)
  • Python 3.10 (base-gpu)
    • Python 3.10 (datascience with GPU support)
      • Python 3.10 (noteable with GPU support)

(No 3.11 noteable build since we need to handle some internal dependency resolution)

• Python 3.11 (base)

  • Python 3.11 (datascience)
  • Python 3.11 (base-gpu)
    • Python 3.11 (datascience with GPU support)

• R 4.3 (base)

  • R 4.3 (noteable)

Local testing

All images:

• Lock dependencies via piptools compile without conflicts
• Builds successfully
• Secrets are set as environment variables (if any exist; if none exist, kernel should still start up successfully)

Base images:

• Show correct language version (e.g. !python --version within an IPython environment)
• Install packages and import them without needing to soft-restart
• Execute basic code

Datascience images:

• Successfully import additional packages not included in the associated base image

Noteable images:

• Python: support SQL cells, dataframe -> DEX output, auto-setup of comms (variable explorer, form cells), Chinese/Japanese/Korean font support (matplotlib)
• R: support matrix / data frame -> DEX output

Note: for the *gpu builds, I was unable to test with my local kubernetes cluster, but was able to docker run --rm --gpus all local/kernel-python-3.9-base-gpu:dev (with the connection_file.json parts of run.sh commented out) and see nvidia-smi show the proper output.

[kafonek]

Current dependencies on/for this PR:

• main
  • PR big new image restructure #7 👈
    • PR rust base #8

This comment was auto-generated by Graphite.