Deprecate using Python 3.7 with default tool lockfiles, preparing for pip 24.1 / Python 3.13 support

[huonw]

Is your feature request related to a problem? Please describe.

Pip 24.1 is the first version of pip that supports Python 3.13. It also drops support for Python 3.7.

This puts us in an awkward position with built-in tools:

1. their lockfiles need to use a pip compatible with Python 3.x to be able to run using a Python 3.x interpreter; for instance supporting 3.12 required regenerating all lockfiles (Most built-in tool lockfiles do not work with Python 3.12 #20354 / Regenerate all Python tool lock-files (updating many) and update to pip_version = 24.0, for Py 3.12 support #20365)
2. we currently support running built-in most tools using Python 3.7 by default (our default tool interpreter constraints include Python 3.7:

   pants/src/python/pants/backend/python/subsystems/python_tool_base.py

   Line 65 in cfb3caf

   default_interpreter_constraints: ClassVar[Sequence[str]] = ["CPython>=3.7,<4"]

The first point means that supporting Python 3.7 and Python 3.13 is potentially impossible, with a single lockfile (i.e. they're mutually exclusive). The second point means that bumping our built-in tools may break any users happily using 3.7.

Python 3.7 is end-of-life now. Python 3.8 is almost end-of-life. See: https://devguide.python.org/versions/.

Describe the solution you'd like

Add deprecation warnings for what will fail in future, for at least one release: using a built-in tool with Python 3.7, and Python 3.8 or newer isn't available.

These warnings should tell the user they'll need to either generate their own lockfiles with their own resolve, or provide Python 3.8 or newer.

Once that deprecation has existed for at least one release, bump the default minimum interpreter constraints and regenerate lockfiles, a way that is compatible for Python 3.13.

Describe alternatives you've considered

Python 3.7 has been end-of-life for almost a year now, so we could potentially just publish new lockfiles directly, and emphasise the breakage and fix in release notes, rather than trying to ease do it in a less-breaking way.

Additional context

Related issues:

• Built-in lockfiles won't work with Python 3.13, due to pip version #20852

Reproducer of the issue that might surface

In https://gist.github.com/huonw/5e98b79ba03bcd7a019ad0a743fe2ccb, I set up an example of what might happen if we upgrade the default lockfiles. It sets up the black subsystem with a lockfile generated with pex3 lock create --pip-version=24.1 black==23.3.0 ..., similar to what the default lockfile could be when we upgrade.

Running it like this:

cd /tmp
git clone https://gist.github.com/5e98b79ba03bcd7a019ad0a743fe2ccb.git
cd 5e98b79ba03bcd7a019ad0a743fe2ccb
pants fmt ::

Fails on my machine, which has a Python 3.13 interpreter installed:

ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython<4,>=3.7' failed with exit code 1.

...

No supported version of Pip is compatible with the given targets:
cp310-cp310-macosx_14_0_arm64 interpreter at /Users/huon/.local/share/mise/installs/python/3.10.14/bin/python3.10
cp310-cp310-macosx_14_0_arm64 interpreter at /Users/huon/.pyenv/versions/3.10.1/bin/python3.10
cp310-cp310-macosx_14_0_arm64 interpreter at /Users/huon/.pyenv/versions/3.10.4/bin/python3.10
cp310-cp310-macosx_14_0_arm64 interpreter at /opt/homebrew/Cellar/[email protected]/3.10.12_1/Frameworks/Python.framework/Versions/3.10/bin/python3.10
cp311-cp311-macosx_14_0_arm64 interpreter at /Users/huon/.pyenv/versions/3.11.0/bin/python3.11
cp311-cp311-macosx_14_0_arm64 interpreter at /opt/homebrew/Cellar/[email protected]/3.11.9/Frameworks/Python.framework/Versions/3.11/bin/python3.11
cp312-cp312-macosx_14_0_arm64 interpreter at /Users/huon/.pyenv/versions/3.12.0/bin/python3.12
cp312-cp312-macosx_14_0_arm64 interpreter at /opt/homebrew/Cellar/[email protected]/3.12.2_1/Frameworks/Python.framework/Versions/3.12/bin/python3.12
cp313-cp313-macosx_14_0_arm64 interpreter at /Users/huon/.pyenv/versions/3.13.0b2/bin/python3.13
cp37-cp37m-macosx_14_0_arm64 interpreter at /Users/huon/.pyenv/versions/3.7.13/bin/python3.7
cp38-cp38-macosx_14_0_arm64 interpreter at /Users/huon/.pyenv/versions/3.8.7/bin/python3.8
cp38-cp38-macosx_14_0_arm64 interpreter at /opt/homebrew/Cellar/[email protected]/3.8.19/Frameworks/Python.framework/Versions/3.8/bin/python3.8
cp39-cp39-macosx_14_0_arm64 interpreter at /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/bin/python3.9
cp39-cp39-macosx_14_0_arm64 interpreter at /Users/huon/.pyenv/versions/3.9.1/bin/python3.9
cp39-cp39-macosx_14_0_arm64 interpreter at /Users/huon/.pyenv/versions/3.9.10/bin/python3.9
cp39-cp39-macosx_14_0_arm64 interpreter at /Users/huon/.pyenv/versions/3.9.17/bin/python3.9
cp39-cp39-macosx_14_0_arm64 interpreter at /Users/huon/.pyenv/versions/3.9.7/bin/python3.9

To simulate only having Python 3.7 installed: uncommenting the python-bootstrap lines, fill in a path and rerun pants fmt ::. This gives a different error, e.g. on my machine:

[python-bootstrap]
search_path = ["/Users/huon/.pyenv/versions/3.7.13/bin/python"]

ProcessExecutionFailure: Process 'Building black.pex' failed with exit code 1.
stdout:

stderr:
The Pip requested was pip==24.1 but it does not work with the interpreter selected which is CPython 3.7.13 at /Users/huon/.pyenv/versions/3.7.13/bin/python3.7. Pip 24.1 requires Python <3.14,>=3.8.

[kaos]

If we provide two lockfiles per tool, switching between them would be easier, and we could issue warnings when using the old one, which could be the default initially.

[cburroughs]

@huonw should we call this complete, or are their more outstanding strands here?

[huonw]

I don't know of any more strands here (other than #21389), so let's call this done! Thanks @cburroughs (and @krishnan-chandra for highlighting that this is closable)