[email protected]: bootstrap with ensurepip #67030

alebcay · 2020-12-16T22:57:26Z

Have you followed the guidelines for contributing?
Have you checked that there aren't other open pull requests for the same formula update/change?
Have you built your formula locally with brew install --build-from-source <formula>, where <formula> is the name of the formula you're submitting?
Is your test running fine brew test <formula>, where <formula> is the name of the formula you're submitting?
Does your build pass brew audit --strict <formula> (after doing brew install <formula>)?

This will (hopefully) address the original issue mentioned in #59802. We now use ensurepip to bootstrap an older version of setuptools/pip before installing our desired version of setuptools, pip, and wheel using the bootstrapped pip.

The result is essentially that the pip script is no longer version locked and should not break if the user attempts to upgrade pip on their own (pip install -U pip). See the linked issue for more details on that breaking behavior.

This PR does not patch ensurepip to provide our chosen version of setuptools and pip (as some other package managers do) for end users who use ensurepip (e.g. in the process of creating a venv). If we deem that a desirable change, it should happen in a different PR.

If/when this gets merged, it may also be prudent to backport to [email protected] and [email protected].

carlocab · 2020-12-18T00:52:50Z

10.14

Error: 13 failed steps!
brew test --retry --verbose cargo-edit
brew test --retry --verbose cucumber-cpp
brew fetch --retry deno
brew fetch --retry diff-pdf
brew test --retry --verbose glyr
brew test --retry --verbose hyperkit
brew test --retry --verbose mariadb
brew test --retry --verbose [email protected]
brew test --retry --verbose [email protected]
brew test --retry --verbose [email protected]
brew test --retry --verbose [email protected]
brew test --retry --verbose [email protected]
brew test --retry --verbose pcl

Nothing too new (#66450). Except the mariadb + friends fix seems to not have taken. @gromgit

~~I was also under the impression that hyperkit was broken only on Big Sur.~~

rgoldberg

Why do we still have the following?

resource "setuptools" do
    url "https://files.pythonhosted.org/packages/12/e1/b9a2926a3c5a3fb055b8f85052f5baa890106a0e21b64a977c10affea751/setuptools-51.0.0.zip"
    sha256 "029c49fd713e9230f6a41c0298e6e1f5839f2cde7104c0ad5e053a37777e7688"
end

resource "pip" do
    url "https://files.pythonhosted.org/packages/cb/5f/ae1eb8bda1cde4952bd12e468ab8a254c345a0189402bf1421457577f4f3/pip-20.3.1.tar.gz"
    sha256 "43f7d3811f05db95809d39515a5111dd05994965d870178a4fe10d5482f9d2e2"
end

resource "wheel" do
    url "https://files.pythonhosted.org/packages/d4/cf/732e05dce1e37b63d54d1836160b6e24fb36eeff2313e93315ad047c7d90/wheel-0.36.1.tar.gz"
    sha256 "aaef9b8c36db72f8bf7f1e54f85f875c4d466819940863ca0b3f3f77f0a1646f"
end

I assume that these files are later used by the call to system bin/"pip3", "install", …, but why not just run something like:

pip3 install --upgrade setuptools==51.0.0
pip3 install --upgrade pip==20.3.1
pip3 install --upgrade wheel==0.36.1

I imagine that pip should always return a consistent binary for every package & version pair.

If we install a version of pip for people to use, we should be able to trust that it will download correct packages to install.

If we don't trust it to install correct packages of versions, why are we installing it?

carlocab · 2020-12-18T01:35:57Z

Does pip3 install verify source integrity? resource blocks do, and that would be one reason to keep them if pip3 install does not.

rgoldberg · 2020-12-18T01:41:43Z

I haven't read it all, but here's https://pip.pypa.io/en/stable/reference/pip_install/#hashes-from-pypi

carlocab · 2020-12-18T01:48:58Z

Since this hash originates remotely, it is not a useful guard against tampering and thus does not satisfy the --require-hashes demand that every package have a local hash.

I think we want a local hash, so pip3 install --upgrade isn't good enough on its own. I think the formula also needs control over where and how this installation happens (since I'm pretty sure pip3 install has its own ideas of where things go), and resource blocks are a good way of managing that.

gromgit · 2020-12-18T02:28:57Z

Except the mariadb + friends fix seems to not have taken

That's disturbing indeed, since the CI post-install "gate" should still be shut according to the current formula. Both my local Homebrew instance and the formula file embedded in a freshly-downloaded bottle (/usr/local/opt/mariadb/.brew/mariadb.rb) agree on this. CI=1 brew install -d mariadb also doesn't run post-install.

@Homebrew/core, is CI ever automatically unset during testing for any reason?

rgoldberg · 2020-12-18T03:03:33Z

Since this hash originates remotely, it is not a useful guard against tampering and thus does not satisfy the --require-hashes demand that every package have a local hash.

I think we want a local hash, so pip3 install --upgrade isn't good enough on its own. I think the formula also needs control over where and how this installation happens (since I'm pretty sure pip3 install has its own ideas of where things go), and resource blocks are a good way of managing that.

We should be able to use a --hash=sha256:… option for each of the 3 packages as described earlier on that documentation web page.

I've used Homebrew-installed pip3 to install & update packages that the [email protected] formula doesn't install. I've also upgraded the packages it does install. They all seem to have the same base directory, so I don't think that Homebrew installs (or should install) packages in a non-standard location.

carlocab · 2020-12-18T03:09:46Z

Actually, it looks like what you're asking is essentially what is already happening in the post-install block.

Without the resource blocks, then there is no pip3 you can call to do pip3 install --upgrade {setuptools,pip,wheel}.

alebcay · 2020-12-18T03:54:11Z

We should be able to use a --hash=sha256:… option for each of the 3 packages as described earlier on that documentation web page.

While this could work, I don't really see how this is particularly beneficial over the resource block approach.

Maintainers are probably more familiar with resource block syntax than some pip syntax that (to my knowledge) is not frequently used
pip would have to download things during the postinstall step, which, while not forbidden, just seems like one more potential point of failure. Consider how the current solution involves staging the needed source during the install step - which would be distributed in bottles - and the postinstall step just puts everything into place.

I don't think that Homebrew installs (or should install) packages in a non-standard location.

The new postinstall block as it exists in this PR installs setuptools, pip, and wheel to HOMEBREW_PREFIX/lib/python3.9/site-packages, which is where the current (broken) pip and company already live.

I aimed to be fairly conservative in my changes here. The key change is really that we bootstrap ensurepip and use that for installation rather than setup.py install. Thus I took care to make pip install stuff into the same places as before, and it's also why we still have the resource blocks.

rgoldberg · 2020-12-18T06:30:02Z

It sounds like it's fine to keep the resource blocks and everything else as is, then.

But, just FYI, I think that if we had pip download the package updates instead of having resource blocks, pip would still update the packages in the exact same location, so we'd wind up with the same result.

carlocab · 2020-12-18T06:37:48Z

So if it produces the exact same result using code that is different from all the other formulae and less familiar to maintainers and contributors (and hence requires special attention, is more difficult to maintain, etc), why do it?

carlocab · 2020-12-18T14:27:40Z

11.0

Error: 19 failed steps!
brew test --retry --verbose asuka
brew test --retry --verbose augustus
brew test --retry --verbose cargo-edit
brew test --retry --verbose corral
brew test --retry --verbose creduce
brew fetch --retry deno
brew fetch --retry diff-pdf
brew install envoy
brew test --retry --verbose hyperkit
brew test --retry --verbose mariadb
brew test --retry --verbose [email protected]
brew test --retry --verbose [email protected]
brew test --retry --verbose [email protected]
brew test --retry --verbose [email protected]
brew test --retry --verbose [email protected]
brew test --retry --verbose networkit
brew test --retry --verbose pcl
brew test --retry --verbose pony-stable
brew test --retry --verbose ponyc

I think all the errors so far is already documented at #66450, with the exception of deno. However, the fetch failure for deno is because its bottle briefly went missing. (It's fixed now.)

Seems like this can close #59802 already.

fxcoudert · 2020-12-18T15:07:28Z

Thanks @alebcay for working on this!

BrewTestBot · 2020-12-18T15:07:35Z

@fxcoudert has triggered a merge.

rgoldberg · 2020-12-18T16:39:11Z

So if it produces the exact same result using code that is different from all the other formulae and less familiar to maintainers and contributors (and hence requires special attention, is more difficult to maintain, etc), why do it?

I already mentioned:

It sounds like it's fine to keep the resource blocks and everything else as is, then.

So I was fine with not changing the formula to download the packages via pip instead of via resource because of the Homebrew formula standardization arguments.

I was just trying to discuss how pip works, so people learn more about it (including myself, in case someone knew that my assumption about how pip would operate is wrong).

xaocon · 2020-12-18T17:31:11Z

Getting errors trying to install. I followed the instructions to try postinstall and tried the instructions from ensurepip to do pip upgrade then tried again. It looks like a fair amount of the postinstall isn't happening because of ensurepip failing. I briefly looked through some of the discussion and linked issues and saw discussion related to hashes that seem like it could be related but I don't see info on fixing. If this is expected maybe there should be some more detailed info about how to fix? If it's not, should i create a new issue for this?

❯ brew postinstall [email protected]
==> Postinstalling [email protected]
==> /usr/local/Cellar/[email protected]/3.9.1_1/bin/python3 -m ensurepip
==> /usr/local/Cellar/[email protected]/3.9.1_1/bin/pip3 install -v --global-option=--no-user-cfg --install-option=--force --install-option=--single-version-externally-managed --install-option=--record=installed.txt --upgrade --target=/usr/local
Last 15 lines from /Users/epitstick/Library/Logs/Homebrew/[email protected]/post_install.02.pip3:
  Found link https://files.pythonhosted.org/packages/fb/46/26d13ba147ba430f9cda0d0cf599a041d142a5c8b1a90ff845ebce7fba0f/pip-20.3b1-py2.py3-none-any.whl#sha256=122fcd82deac1153c1699f29815bfab3f876e5bbe018cc2df1297f9802572a97 (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.3b1
  Found link https://files.pythonhosted.org/packages/7f/61/2da3c027ad7bd4bc87a3ee7e7160c93e7500dac3536e02ff93008e9b3460/pip-20.3b1.tar.gz#sha256=819c710a5c8d8c5e44695d03e51cb23b08c070e1ae6a5d6910a89e248e0ff29c (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.3b1
  Found link https://files.pythonhosted.org/packages/55/73/bce122d1ed0217b3c1a3439ab16dfa94bbeabd0d31755fcf907493abf39b/pip-20.3-py2.py3-none-any.whl#sha256=3236fe7288d155c238bb6c85d3e784db3a8e592e827b83fea4d36d8b749ecc4b (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.3
  Found link https://files.pythonhosted.org/packages/03/41/6da553f689d530bc2c337d2c496a40dc9c0fdc6481e5df1f3ee3b8574479/pip-20.3.tar.gz#sha256=9ae7ca6656eac22d2a9b49d024fc24e00f68f4c4d4db673d2d9b525c3dea6e0e (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.3
  Found link https://files.pythonhosted.org/packages/ab/11/2dc62c5263d9eb322f2f028f7b56cd9d096bb8988fcf82d65fa2e4057afe/pip-20.3.1-py2.py3-none-any.whl#sha256=425e79b20939abbffa7633a91151a882aedc77564d9313e3584eb0416c28c558 (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.3.1
  Found link https://files.pythonhosted.org/packages/cb/5f/ae1eb8bda1cde4952bd12e468ab8a254c345a0189402bf1421457577f4f3/pip-20.3.1.tar.gz#sha256=43f7d3811f05db95809d39515a5111dd05994965d870178a4fe10d5482f9d2e2 (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.3.1
  Skipping link: yanked for reason: <none given>: https://files.pythonhosted.org/packages/3d/0c/01014c0442830eb38d6baef0932fdcb389279ce74295350ecb9fe09e048a/pip-20.3.2-py2.py3-none-any.whl#sha256=8d779b6a85770bc5f624b5c8d4d922ea2e3cd9ce6ee92aa260f12a9f072477bc (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*)
  Skipping link: yanked for reason: <none given>: https://files.pythonhosted.org/packages/51/63/86e147c44335b03055e58a27c791d94fff4baaf08d67852f925ab9b90240/pip-20.3.2.tar.gz#sha256=aa1516c1c8f6f634919cbd8a58fc81432b0fa96f421a97d05a205ee86b07c43d (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*)
  Found link https://files.pythonhosted.org/packages/54/eb/4a3642e971f404d69d4f6fa3885559d67562801b99d7592487f1ecc4e017/pip-20.3.3-py2.py3-none-any.whl#sha256=fab098c8a1758295dd9f57413c199f23571e8fde6cc39c22c78c961b4ac6286d (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.3.3
  Found link https://files.pythonhosted.org/packages/ca/1e/d91d7aae44d00cd5001957a1473e4e4b7d1d0f072d1af7c34b5899c9ccdf/pip-20.3.3.tar.gz#sha256=79c1ac8a9dccbec8752761cb5a2df833224263ca661477a2a9ed03ddf4e0e3ba (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.3.3
Given no hashes to check 155 links for project 'pip': discarding no candidates
WARNING: You are using pip version 20.2.3; however, version 20.3.3 is available.
You should consider upgrading via the '/usr/local/opt/[email protected]/bin/python3.9 -m pip install --upgrade pip' command.
Removed file:///usr/local/Cellar/python%403.9/3.9.1_1/libexec/pip from build tracker '/private/tmp/pip-req-tracker-99nq6z2f'
Removed build tracker: '/private/tmp/pip-req-tracker-99nq6z2f'
Warning: The post-install step did not complete successfully
You can try again using `brew postinstall [email protected]`

I don't see anything wrong with my system that would be responsible but I'm no expert.

❯ brew --version
Homebrew 2.6.2-113-g89e5d80
Homebrew/homebrew-core (git revision a2fee0; last commit 2020-12-18)
Homebrew/homebrew-cask (git revision f4c52; last commit 2020-12-18)

❯ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and which additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew-provided
script of the same name. We found the following "config" scripts:
  /Users/epitstick/.cargo/bin/cargo-install-update-config

fxcoudert · 2020-12-18T17:36:21Z

@xaocon please provide a full report as a new issue, with all the information requested, including full logs. Thanks!

alebcay force-pushed the python-fix-pip branch from 0d7ea57 to 2ac9b33 Compare December 16, 2020 23:57

[email protected]: bootstrap with ensurepip

e2532ff

alebcay force-pushed the python-fix-pip branch from 2ac9b33 to e2532ff Compare December 17, 2020 00:02

fxcoudert mentioned this pull request Dec 17, 2020

[email protected]: update pip, wheel #67004

Closed

5 tasks

rgoldberg reviewed Dec 18, 2020

View reviewed changes

fxcoudert approved these changes Dec 18, 2020

View reviewed changes

BrewTestBot closed this in f0ae7e1 Dec 18, 2020

alebcay mentioned this pull request Dec 18, 2020

Python3: Impossible to update pip any time a pip update is released #59802

Closed

BrewTestBot added the outdated PR was locked due to age label Jan 18, 2021

Homebrew locked as resolved and limited conversation to collaborators Jan 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[email protected]: bootstrap with ensurepip #67030

[email protected]: bootstrap with ensurepip #67030

alebcay commented Dec 16, 2020

carlocab commented Dec 18, 2020 •

edited

Loading

rgoldberg left a comment •

edited

Loading

carlocab commented Dec 18, 2020

rgoldberg commented Dec 18, 2020

carlocab commented Dec 18, 2020

gromgit commented Dec 18, 2020

rgoldberg commented Dec 18, 2020

carlocab commented Dec 18, 2020

alebcay commented Dec 18, 2020 •

edited

Loading

rgoldberg commented Dec 18, 2020

carlocab commented Dec 18, 2020 •

edited

Loading

carlocab commented Dec 18, 2020 •

edited

Loading

fxcoudert commented Dec 18, 2020

BrewTestBot commented Dec 18, 2020

rgoldberg commented Dec 18, 2020

xaocon commented Dec 18, 2020

fxcoudert commented Dec 18, 2020

[email protected]: bootstrap with ensurepip #67030

[email protected]: bootstrap with ensurepip #67030

Conversation

alebcay commented Dec 16, 2020

carlocab commented Dec 18, 2020 • edited Loading

rgoldberg left a comment • edited Loading

Choose a reason for hiding this comment

carlocab commented Dec 18, 2020

rgoldberg commented Dec 18, 2020

carlocab commented Dec 18, 2020

gromgit commented Dec 18, 2020

rgoldberg commented Dec 18, 2020

carlocab commented Dec 18, 2020

alebcay commented Dec 18, 2020 • edited Loading

rgoldberg commented Dec 18, 2020

carlocab commented Dec 18, 2020 • edited Loading

carlocab commented Dec 18, 2020 • edited Loading

fxcoudert commented Dec 18, 2020

BrewTestBot commented Dec 18, 2020

rgoldberg commented Dec 18, 2020

xaocon commented Dec 18, 2020

fxcoudert commented Dec 18, 2020

carlocab commented Dec 18, 2020 •

edited

Loading

rgoldberg left a comment •

edited

Loading

alebcay commented Dec 18, 2020 •

edited

Loading

carlocab commented Dec 18, 2020 •

edited

Loading

carlocab commented Dec 18, 2020 •

edited

Loading