Unable to build no-binary :all: with use-pep517

[jaraco]

Environment

• pip version: 20.0.2
• Python version: 3.8.1
• OS: macOS 10.15.3

Description

As distlib doesn't publish a wheel, and because it has an implicit dependency on setuptools, I'm attempting to use --use-pep517 to force pip to ensure that setuptools/wheel are present for building, but when I do, pip fails to install wheel:

~ $ pip install --use-pep517 --no-binary :all: distlib                                                                                                                        
Collecting distlib
  Using cached distlib-0.3.0.zip (571 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-6j9zlufg/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel
       cwd: None
  Complete output (13 lines):
  Collecting setuptools>=40.8.0
    Using cached setuptools-45.3.0.zip (862 kB)
  Collecting wheel
    Using cached wheel-0.34.2.tar.gz (58 kB)
      ERROR: Command errored out with exit status 1:
       command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-hd2m84od/wheel/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-hd2m84od/wheel/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-hd2m84od/wheel/pip-egg-info
           cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-hd2m84od/wheel/
      Complete output (3 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
      ModuleNotFoundError: No module named 'setuptools'
      ----------------------------------------
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-6j9zlufg/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel Check the logs for full command output.
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pip-run", line 8, in <module>
    sys.exit(run())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/deps.py", line 75, in load
    _installable(args) and subprocess.check_call(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/local/bin/python', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-mzykfthi', '--use-pep517', '--no-binary', ':all:', 'distlib')' returned non-zero exit status 1.

Expected behavior

pip should be able to assemble a build environment, either by disabling the no-binary option for those requirements or (preferably) by providing a bootstrap environment for these build dependencies.

Note: This may be an issue that can be addressed with wheel.

[jaraco]

I attempted to address this within wheel at pypa/wheel#344, but it seems that even declaring the build dependencies, pip attempts to install wheel from sdist and fails when setuptools isn't already present.

$ pip-run --no-binary :all: git+https://github.com/pypa/wheel@bugfix/344-explicit-setuptools                                                                            
  Running command git clone -q https://github.com/pypa/wheel /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-d31ewf1h
Collecting git+https://github.com/pypa/wheel@bugfix/344-explicit-setuptools
  Cloning https://github.com/pypa/wheel (to revision bugfix/344-explicit-setuptools) to /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-d31ewf1h
  Running command git checkout -b bugfix/344-explicit-setuptools --track origin/bugfix/344-explicit-setuptools
  Switched to a new branch 'bugfix/344-explicit-setuptools'
  Branch 'bugfix/344-explicit-setuptools' set up to track remote branch 'bugfix/344-explicit-setuptools' from 'origin'.
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-p3mtk_lh/normal --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- wheel
       cwd: None
  Complete output (11 lines):
  Collecting wheel
    Using cached wheel-0.34.2.tar.gz (58 kB)
      ERROR: Command errored out with exit status 1:
       command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/pip-egg-info
           cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-eouk8ka8/wheel/
      Complete output (3 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
      ModuleNotFoundError: No module named 'setuptools'
      ----------------------------------------
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-p3mtk_lh/normal --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- wheel Check the logs for full command output.
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pip-run", line 8, in <module>
    sys.exit(run())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip_run/deps.py", line 75, in load
    _installable(args) and subprocess.check_call(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/local/bin/python', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-trtwrd5a', '--no-binary', ':all:', 'git+https://github.com/pypa/wheel@bugfix/344-explicit-setuptools')' returned non-zero exit status 1.

I think pip probably should do something here to allow wheel to build when installed from source.

[pfmoore]

Is this not related to in-tree backend support (which pip doesn't yet include, IIRC)? I thought the idea of that PEP update was precisely to allow projects to be defined in a way that meant they worked even if the user requested that everything was built from source.

If it's not then I think we need to get a much better understanding of what pip is expected to do in the face of --no-binary :all: before deciding on a solution, rather than firefighting individual use cases when they arise.

[jaraco]

I believe this issue is unrelated to in-tree backend support, though I do think that support is present in pip 20). The issue may be described thus:

Attempting to build any package with --use-pep517 and --no-binary :all:, pip downloads setuptools and wheel and attempts to build those (not from wheels). When attempting to build wheel, it fails because setuptools is not present.

It might just be the case that when using pep517 to build a package, it should also use pep517 to build the package's dependencies also... except that wheel would need wheel, leading to an infinite loop.

we need to get a much better understanding of what pip is expected to do

Agreed. Maybe the answer is that no-binary :all: is unsupported.

[pfmoore]

It might just be the case that when using pep517 to build a package, it should also use pep517 to build the package's dependencies also... except that wheel would need wheel, leading to an infinite loop.

Eww. Yes, that is a variant of the bootstrapping issue that in-tree backend support was addressing, but sufficiently different to be a new problem in its own right.

Maybe the answer is that no-binary :all: is unsupported.

That would be my strong preference. I honestly don't see why pip has to be the tool to handle the "build everything from raw source" problem. But there were fairly strong voices arguing that we had to support that case, so I think they should be the ones defining the solution here.

[jaraco]

In my use-case, I wasn't concerned with the "build everything from raw source" use case, but was rather attempting to build a single package from source and I used --no-binary :all: as a shorthand for "please build from source". In my case, I could have used --no-binary distlib, which would have been more selective.

[jaraco]

In the referenced vendoring issue, I learned that it's not just system packagers that want to build from source but also packages that wish to verify licenses.

[McSinyx]

Coming from ActiveState/appdirs#141,

Attempting to build any package with --use-pep517 and --no-binary :all:, pip downloads setuptools and wheel and attempts to build those (not from wheels). When attempting to build wheel, it fails because setuptools is not present.

How about letting setuptools and wheel be hard runtime dependencies? Just like we don't build pip when we want to use pip to build other wheels, then by the same logic, other build tools should not be built during the build process of other packages.

[uranusjr]

I think the whole idea of build isolation is to ultimately remove setuptools and wheel from needing to present in the runtime environment, because we can pull it if needed in the isolated build environment instead.

[McSinyx]

Thank you @uranusjr, I totally missed that point.

[jaypadariya]

chatboat python

if you want to install chatterbot

pip install chatterbot --no-binary :all:

you should use "pip install chatterbot --no-binary :all:"

before that run under three codes and install it.

conda remove --force PyYAML
pip install pint
pip install mathparse

[jaraco]

I ran into this issue again today, this time with pip download. I was attempting to download the sources for xattr:

pip download --no-binary :all: xattr

Because I'm trying to be a good citizen and operate without setuptools being installed, that operation failed with:

ImportError: cannot import name 'setup' from 'setuptools' (unknown location)

When setuptools tries to build from source, which tries to build wheel from source, and setuptools isn't present.

 draft $ py -m pip download --no-binary :all: xattr
Collecting xattr
  Downloading xattr-0.10.0.tar.gz (15 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [48 lines of output]
      DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
      Collecting setuptools>=40.8.0
        Using cached setuptools-65.5.0.tar.gz (2.6 MB)
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'error'
        error: subprocess-exited-with-error
      
        × pip subprocess to install backend dependencies did not run successfully.
        │ exit code: 1
        ╰─> [26 lines of output]
            DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
            Collecting wheel
              Using cached wheel-0.37.1.tar.gz (66 kB)
              Preparing metadata (setup.py): started
              Preparing metadata (setup.py): finished with status 'error'
              error: subprocess-exited-with-error
      
              × python setup.py egg_info did not run successfully.
              │ exit code: 1
              ╰─> [6 lines of output]
                  Traceback (most recent call last):
                    File "<string>", line 2, in <module>
                    File "<pip-setuptools-caller>", line 34, in <module>
                    File "/private/var/folders/sx/n5gkrgfx6zd91ymxr2sr9wvw00n8zm/T/pip-install-eca5mryf/wheel_44b7c370255345dd8b9fc33a8f13d4d7/setup.py", line 2, in <module>
                      from setuptools import setup
                  ImportError: cannot import name 'setup' from 'setuptools' (unknown location)
                  [end of output]
      
              note: This error originates from a subprocess, and is likely not a problem with pip.
            error: metadata-generation-failed
      
            × Encountered error while generating package metadata.
            ╰─> See above for output.
      
            note: This is an issue with the package mentioned above, not pip.
            hint: See above for details.
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: subprocess-exited-with-error
      
      × pip subprocess to install backend dependencies did not run successfully.
      │ exit code: 1
      ╰─> See above for output.
      
      note: This error originates from a subprocess, and is likely not a problem with pip.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I tried instead to employ pep517, but that failed similarly:

 draft $ py -m pip download --use-pep517 --no-binary :all: xattr
Collecting xattr
  Using cached xattr-0.10.0.tar.gz (15 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [48 lines of output]
      DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
      Collecting setuptools>=40.8.0
        Using cached setuptools-65.5.0.tar.gz (2.6 MB)
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'error'
        error: subprocess-exited-with-error
      
        × pip subprocess to install backend dependencies did not run successfully.
        │ exit code: 1
        ╰─> [26 lines of output]
            DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
            Collecting wheel
              Using cached wheel-0.37.1.tar.gz (66 kB)
              Preparing metadata (setup.py): started
              Preparing metadata (setup.py): finished with status 'error'
              error: subprocess-exited-with-error
      
              × python setup.py egg_info did not run successfully.
              │ exit code: 1
              ╰─> [6 lines of output]
                  Traceback (most recent call last):
                    File "<string>", line 2, in <module>
                    File "<pip-setuptools-caller>", line 34, in <module>
                    File "/private/var/folders/sx/n5gkrgfx6zd91ymxr2sr9wvw00n8zm/T/pip-install-_c2chljg/wheel_1e5d23c0a353431cac90e861087cc190/setup.py", line 2, in <module>
                      from setuptools import setup
                  ImportError: cannot import name 'setup' from 'setuptools' (unknown location)
                  [end of output]
      
              note: This error originates from a subprocess, and is likely not a problem with pip.
            error: metadata-generation-failed
      
            × Encountered error while generating package metadata.
            ╰─> See above for output.
      
            note: This is an issue with the package mentioned above, not pip.
            hint: See above for details.
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: subprocess-exited-with-error
      
      × pip subprocess to install backend dependencies did not run successfully.
      │ exit code: 1
      ╰─> See above for output.
      
      note: This error originates from a subprocess, and is likely not a problem with pip.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

The only way I know to make it work is to pre-install setuptools (works with or without --use-pep517). I was surprised that --use-pep517 succeeded.

 draft $ py -m pip-run -q setuptools -- -m pip download --use-pep517 --no-binary :all: xattr
Collecting xattr
  Using cached xattr-0.10.0.tar.gz (15 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting cffi>=1.0
  Using cached cffi-1.15.1.tar.gz (508 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pycparser
  Using cached pycparser-2.21.tar.gz (170 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Saved ./xattr-0.10.0.tar.gz
Saved ./cffi-1.15.1.tar.gz
Saved ./pycparser-2.21.tar.gz
Successfully downloaded xattr cffi pycparser

[pfmoore]

Is this not the situation that in-tree build backends were designed to address? Does setuptools configure itself as its own in-tree build backend? If not, then maybe it needs to for the --no-binaty :all: case.