[python] Support Python 3.12

[johnkerl]

Issue and/or context: #1849

Changes: Just a check to see what happens with 3.12. We did it! :)

Notes for Reviewer: This PR is not for review. Completed

Status as of 2024-02-05: numba 0.58 is not yet ready for Python 3.12.

$ docker run --rm -it --entrypoint=/bin/bash python:3.12-bookworm
root@20d715ac5ab7:/# pip install numba==0.58
Collecting numba==0.58
  Downloading numba-0.58.0.tar.gz (2.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 32.5 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-417bk_qg/numba_1954fc6d020f4c6087042050c24f72c3/setup.py", line 51, in <module>
          _guard_py_ver()
        File "/tmp/pip-install-417bk_qg/numba_1954fc6d020f4c6087042050c24f72c3/setup.py", line 48, in _guard_py_ver
          raise RuntimeError(msg.format(cur_py, min_py, max_py))
      RuntimeError: Cannot install on Python version 3.12.1; only versions >=3.8,<3.12 are supported.
      [end of output]

$ docker run --rm -it --entrypoint=/bin/bash python:3.11-bookworm
root@d8f7153f52c6:/# pip install numba==0.58
Collecting numba==0.58
  Obtaining dependency information for numba==0.58 from https://files.pythonhosted.org/packages/f7/b6/8800604a3380aa94af830c30839db93077c7fe2812a48ce7d4400d75ba9c/numba-0.58.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata
...
Installing collected packages: numpy, llvmlite, numba
Successfully installed llvmlite-0.41.1 numba-0.58.0 numpy-1.25.2

Status as of 2024-02-13: numba 0.59 is installable on Python 3.12.

[codecov]

Codecov Report

Merging #2108 (86f7cfa) into main (a4f68a8) will decrease coverage by 10.33%.
The diff coverage is n/a.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2108       +/-   ##
===========================================
- Coverage   82.10%   71.77%   -10.33%     
===========================================
  Files          87      101       +14     
  Lines        8112     6926     -1186     
  Branches        0      211      +211     
===========================================
- Hits         6660     4971     -1689     
- Misses       1452     1853      +401     
- Partials        0      102      +102     

Flag	Coverage Δ
libtiledbsoma	66.97% <ø> (?)
python	?
r	74.62% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
python_api	∅ <ø> (∅)
libtiledbsoma	48.27% <ø> (∅)

[johnkerl]

Re-trying with rebase on #2122 which has since been merged, for #1849

[johnkerl]

Most strange. At
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/7891284152/job/21535281973?pr=2108
I see

Run python scripts/show-versions.py
  python scripts/show-versions.py
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.12.2/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.12.2/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.2/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.2/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.12.2/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.12.2/x64/lib
Traceback (most recent call last):
  File "/home/runner/work/TileDB-SOMA/TileDB-SOMA/scripts/show-versions.py", line 13, in <module>
    import tiledbsoma
  File "/home/runner/work/TileDB-SOMA/TileDB-SOMA/apis/python/src/tiledbsoma/__init__.py", line 111, in <module>
    from ._general_utilities import (
  File "/home/runner/work/TileDB-SOMA/TileDB-SOMA/apis/python/src/tiledbsoma/_general_utilities.py", line 13, in <module>
    from pkg_resources import DistributionNotFound, get_distribution
ModuleNotFoundError: No module named 'pkg_resources'
Error: Process completed with exit code 1.

Yet when I run docker run --rm -it --entrypoint=/bin/bash python:3.12.2 and use

from pkg_resources import DistributionNotFound, get_distribution
try:
    print(get_distribution("tiledbsoma").version)
except DistributionNotFound:
    print("unknown")

I get no such error 🤔

[bkmartinjr]

See https://docs.python.org/3/whatsnew/3.12.html for information on why you may see this error. The default packages have changed in 3.12 for a virtual environment.

gh-95299: Do not pre-install setuptools in virtual environments created with venv. This means that distutils, setuptools, pkg_resources, and easy_install will no longer available by default; to access these run pip install setuptools in the activated virtual environment.

[bkmartinjr]

Rather than modifying _general_utilities.py, why not just add setuptools to the list of dependencies for the package?

On python 3.7-3.11, it will already be installed and available, so little to no overhead. and it lets you keep the version code identical over all releases.

[johnkerl]

Rather than modifying _general_utilities.py, why not just add setuptools to the list of dependencies for the package?

On python 3.7-3.11, it will already be installed and available, so little to no overhead. and it lets you keep the version code identical over all releases.

@bkmartinjr thanks for thinking of what I did not! I'll try that.

However, I would also note
https://stackoverflow.com/questions/76043689/pkg-resources-is-deprecated-as-an-api
so maybe we keep what I have as a transition, then drop the 3.7 compatibility block once we fully drop support for Python 3.7?

Thoughts?

[bkmartinjr]

Yep, just (actually) read the docs. What you did looks right to me - sorry for the noise!