implicit dependency on setuptools causes build failures on pip

[jaraco]

In pypa/pip#7831, I described an issue where attempting to build distlib with pep-571 support and --no-binary :all: leads to a build failure when pip attempts to create the build environment with setuptools and wheel, but wheel fails to build when setuptools isn't present. Perhaps this issue could be addressed by declaring the build dependency on setuptools.

[jaraco]

Repro:

$ pip-run --no-binary :all: wheel                                                                                                                                       
Collecting wheel
  Using cached wheel-0.34.2.tar.gz (58 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-kqex__en/wheel/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-kqex__en/wheel/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-kqex__en/wheel/pip-egg-info
         cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-kqex__en/wheel/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pip-run", line 8, in <module>
    sys.exit(run())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/deps.py", line 75, in load
    _installable(args) and subprocess.check_call(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/local/bin/python', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-uz4z12c7', '--no-binary', ':all:', 'wheel')' returned non-zero exit status 1.

[jaraco]

In 4f9d93d, I attempted to declare the dependency on setuptools, but that was insufficient. It seems pip was unable to honor it.

$ pip-run --no-binary :all: git+https://github.com/pypa/wheel@bugfix/344-explicit-setuptools                                                                            
  Running command git clone -q https://github.com/pypa/wheel /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-d31ewf1h
Collecting git+https://github.com/pypa/wheel@bugfix/344-explicit-setuptools
  Cloning https://github.com/pypa/wheel (to revision bugfix/344-explicit-setuptools) to /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-d31ewf1h
  Running command git checkout -b bugfix/344-explicit-setuptools --track origin/bugfix/344-explicit-setuptools
  Switched to a new branch 'bugfix/344-explicit-setuptools'
  Branch 'bugfix/344-explicit-setuptools' set up to track remote branch 'bugfix/344-explicit-setuptools' from 'origin'.
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-p3mtk_lh/normal --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- wheel
       cwd: None
  Complete output (11 lines):
  Collecting wheel
    Using cached wheel-0.34.2.tar.gz (58 kB)
      ERROR: Command errored out with exit status 1:
       command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/pip-egg-info
           cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/
      Complete output (3 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
      ModuleNotFoundError: No module named 'setuptools'
      ----------------------------------------
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-p3mtk_lh/normal --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- wheel Check the logs for full command output.
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pip-run", line 8, in <module>
    sys.exit(run())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/deps.py", line 75, in load
    _installable(args) and subprocess.check_call(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/local/bin/python', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-trtwrd5a', '--no-binary', ':all:', 'git+https://github.com/pypa/wheel@bugfix/344-explicit-setuptools')' returned non-zero exit status 1.

It seems pip is still trying to install wheel, even though it's not in the build-system.requires.

Edit: To replicate this behavior use pip-run --no-binary :all: git+https://github.com/pypa/wheel@4f9d93d.

[jaraco]

In cae8be5, I tried adding backend-path so that wheel had wheel, but that fails with a BackendInvalid error:

$ pip-run --no-binary :all: git+https://github.com/pypa/wheel@cae8be50488b007cbd6c261a86ca478b473631a9                               
Collecting git+https://github.com/pypa/wheel@cae8be50488b007cbd6c261a86ca478b473631a9
  Cloning https://github.com/pypa/wheel (to revision cae8be50488b007cbd6c261a86ca478b473631a9) to /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-i4zvhvwd
  Running command git clone -q https://github.com/pypa/wheel /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-i4zvhvwd
  Running command git checkout -q cae8be50488b007cbd6c261a86ca478b473631a9
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 331, in run
    resolver.resolve(requirement_set)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 515, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 96, in _setup_isolation
    reqs = backend.get_requires_for_build_wheel()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 151, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 257, in _call_hook
    raise BackendInvalid(
pip._vendor.pep517.wrappers.BackendInvalid
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pip-run", line 8, in <module>
    sys.exit(run())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/deps.py", line 75, in load
    _installable(args) and subprocess.check_call(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/local/bin/python', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-ql9o180s', '--no-binary', ':all:', 'git+https://github.com/pypa/wheel@cae8be50488b007cbd6c261a86ca478b473631a9')' returned non-zero exit status 2.

[agronholm]

What do you expect me to do here? Wheel already declares setuptools as a setup requirement. Creating a pyproject.toml has been tried too and its presence caused other issues.

[jaraco]

I'm not sure there's anything the wheel project can do here. It seems to be a bootstrapping issue - pip creates an implicit dependency on wheel for every project, including setuptools, so when pip attempts to build wheel from source, it attempts to download setuptools, which requires wheel (still from source), which can't build without setuptools. This issue probably needs to be addressed at the pip or packaging-problems level. The aforementioned pip issue is probably a suitable forum for the underlying issue.

Creating a pyproject.toml has been tried too and its presence caused other issues.

That seems problemmatic. Even if supplying a pyproject.toml doesn't address the bootstrapping issue, it shouldn't be the case that following PEP 517/518 should cause additional failures. Do you have any references as to what issues that caused?

[agronholm]

The relevant discussion can be found here.

[nanonyme]

Seems like one possibility towards bootstrapping solution here would be porting this project to be built through flit. As part of testing pypa/flit#441, I noticed that a simple bootstrapper PoC like https://gitlab.com/freedesktop-sdk/pep517_install/-/tree/main would be able to install as far as flit itself and could be used to install this project which would in term allow installing setuptools (whose build dep is wheel).

[jaraco]

I've been working on solving the boostrapping problem in general in pypa/packaging-problems#342. I'm closing this as won't fix. I've pushed the work in progress branch to refs/archive/bugfix/344-explicit-setuptools and deleted the branch.

[agronholm]

You actually closed this as completed, not wontfix. But I think this is actually correct, as wheel no longer depends on setuptools to install, given that it uses flit instead to overcome this specific problem. There is currently a run-time dependency on setuptools, for wheel convert, but I'm also working to remove even that dependency by refactoring the code.