Poetry uses the wrong version of Python

[yig]

Bug report

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

• ran brew update and can still reproduce the problem?
• ran brew doctor, fixed all issues and can still reproduce the problem?
• ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
• if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

What you were trying to do (and why)

I am trying to use poetry to manage a Python environment. My environment depends on a 3.8.x version of Python (because some packages I depend on don't yet install correctly with 3.9).

What happened (include command output)

Poetry erroneously uses Python 3.9, the version it depends on, instead of 3.8.x, the version I requested.

Command output

brew install poetry

poetry install

poetry run python3 --version # prints 3.9.0
  

What you expected to happen

I expected Poetry to set up a Python 3.8.x environment like my pyproject.toml specifies. Installing poetry in the recommended manner outside of Homebrew (via their get-poetry.py script) or using the default Homebrew Python pip install poetry works correctly.

The recommended installation page includes a warning about the issue I encountered: "Using alternative installation methods will make Poetry always use the Python version for which it has been installed to create virtualenvs."

So either I expect Homebrew's poetry to solve this problem or else print a caveat when installing poetry that it won't correctly select the Python interpreter.

Step-by-step reproduction instructions (by running brew install commands)

brew install poetry
mkdir poetry-python-version
cd poetry-python-version

Place the following into a file named pyproject.toml in the current directory:

[tool.poetry]
name = "testing_python_versions"
version = "0.1.0"
description = "python version specifier"
authors = []

[tool.poetry.dependencies]
python = "~3.8"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Create the environment and check the Python version:

poetry install
poetry run python --version

The output of these commands is as follows:

$ poetry install
The currently activated Python version 3.9.0 is not supported by the project (~3.8).
Trying to find and use a compatible version. 
Using python3 (3.8.6)
Creating virtualenv testing-python-versions-wZA-gtVO-py3.8 in /Users/yotam/Library/Caches/pypoetry/virtualenvs
Installing dependencies from lock file
$ poetry run python --version
Python 3.9.0

The interpreter really is the unwanted 3.9. It can run new Python 3.9 syntax:

$ poetry run python
Python 3.9.0 (default, Oct  6 2020, 04:17:54) 
[Clang 12.0.0 (clang-1200.0.32.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> d = {'spam': 1, 'eggs': 2, 'cheese': 3}
>>> e = {'cheese': 'cheddar', 'aardvark': 'Ethel'}
>>> d | e
{'spam': 1, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}

When I install poetry using their recommended get-poetry.py script outside of Homebrew, or I install using Homebrew Python's pip install poetry, the above works correctly.

[andrewcrook]

It is probably a path and environment issue rather than the Homebrew package itself.

You also probably have both macOS versions 2 and 3 and a homebrew version of python3 as a homebrew dependency.
Managing multiple versions can be a nightmare. I personally use pyenv and set a version of Python local to my project directory before running poetry.

default - macOS system version for python
[probably should be a version of 3 using pip3 but next macOS removes python2 so will re-setup upon upgrading]

>mkdir test123
> cd test123
>poetry init
This command will guide you through creating your pyproject.toml config.
Package name [test123]:  test
Version [0.1.0]:
Description []:
Author [None, n to skip]:  me
License []:
Compatible Python versions [^2.7]:  
.......
......

using pyenv - previously installed python 3.7.4 via pyenv

> mkdir test123
> cd test123
> pyenv local 3.7.4
> poetry init
This command will guide you through creating your pyproject.toml config.
Package name [test123]:
Version [0.1.0]:
Description []:
Author [None, n to skip]:  me
License []:
Compatible Python versions [^3.7]:
.....
....

with pyenv you can set and manage versions for system/global, shell and local virtual environments

You can also extend this with virtualenv using pyenv-virtualenv and have environments for each Python verson and an environment for each project and there specific dependence's. e.g environments could be 3.8.5, project1, 3.8.5/envs/project1 but that’s of scope for this ticket.

[andrewcrook]

Might also be related to issue #62911 regarding version 3.8 and 3.9 from homebrew causing issues.
Again pyenv would get around this. I recommend all Python developers go down this route.

[yig]

I don't want to use a second python environment manager to manage my python environment manager ;-). Poetry can handle this workaround itself by typing poetry env use python3.8 before poetry install. I don't like the redundancy (or potential for error), since compatible Python versions are already specified in the pyproject.toml file.

Poetry's own installer doesn't have this problem, so it would be nice if we could figure out a way to make Homebrew's work just as well. I really appreciate having all my installed software managed by Homebrew.

[andrewcrook]

poetry env use python3.8

I belive that should be

poetry env use 3.8

[yig]

Either one works. 3.8 is shorthand for python3.8 which is shorthand for an explicit path to the executable. Source: https://python-poetry.org/docs/managing-environments/#switching-between-environments

[andrewcrook]

I don't want to use a second python environment manager to manage my python environment manager ;-)

Just use pyenv without pyenv-virtualenv for the different versions of Python then and use the virtual environments with poetry with its virtualenv support. Poetry actually recommends using pyenv to manage multiple Python versions on the page you referenced. I use pyenv-virtualenv because I haven't always been using poetry.

Either one works. 3.8 is shorthand for python3.8

Ah okay

[yig]

Interesting, they do recommend pyenv. What a wild ride. If I install pyenv, then pyenv will install its own versions of Python completely independently from Homebrew python. That has some advantages. It's pretty heavyweight, though.

The Poetry managing environments page also says:

By default, Poetry will try to use the currently activated Python version to create the virtual environment for the current project.
However, for various reasons, this Python version might not be compatible with the python requirement of the project. In this case, Poetry will try to find one that is and use it. If it's unable to do so then you will be prompted to activate one explicitly, see Switching environments.

So at the very least, it's clear that Homebrew Poetry is misbehaving. poetry install finds and says it's using the correct version of Python (see my output above), but it doesn't actually use it. No error messages, no prompt, just incorrect behavior.

[andrewcrook]

I think you’re right, I am wondering if its todo with the fact that the formula has Python 3.9 as a dependency and that the installer generates a poetry launcher file where the hash-bang is hardcoded to the absolute path of the Python 3.9 dependency it also sets up some other paths.

(bin/"poetry").write <<~PYTHON
      #!#{Formula["[email protected]"].opt_bin/"python3"}
      import sys
      sys.path.insert(0, "#{site_packages}")
      sys.path.insert(0, "#{vendor_site_packages}")
      if __name__ == "__main__":
          from poetry.console import main
          main()
    PYTHON

homebrew-core/Formula/poetry.rb

Line 195 in d1b6def

#!#{Formula["[email protected]"].opt_bin/"python3"}

So poetry basically always runs under Python 3.9. However, the launcher file generated by the get-poetry.py script uses #!/usr/bin/env getting the Python path from the environment (which is changed by environment manager tools)

f.write(u("#!/usr/bin/env {}\n".format(python_executable))) 

https://github.com/python-poetry/poetry/blob/9c085a585c093a722717fec318bfc6610e93a9f1/get-poetry.py#L695

python_executable being the Python binary for whatever system e.g windows py.exe and python or python3 on nix.

so poetry runs under the Python set in the environment which it changes where as the homebrew formula version ignores this and runs poetry under python 3.9 always.

I wonder if you run the following if it proves this?

poetry run env python --version

[yig]

Inside my poetry-python-version directory mentioned above,

poetry run env python --version

prints

Python 3.9.0

I changed the shebang of /usr/local/bin/poetry to #!/usr/bin/env python3 and it worked. Clearing the environments and then running poetry run env python --version prints Python 3.8.6. However, changing the shebang to #!/usr/bin/env python does not work. That picks up a Python 2.7 interpreter, causing poetry to die with the exception ImportError: No module named typing. I find that odd since get-poetry.py itself uses python and works with version 2.7 (despite spewing warnings about how Python 2.7 is deprecated).

[BrewTestBot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[juanmirocks]

I have the same or similar problem. I want to use python 3.6 for my project, yet somehow poetry's env is "locked" to my system's Homebrew 3.9.0 python version.

Even if I try forcing the environment in poetry (with a pyenv version), the behavior is the same.

11:15:40|new$ pyenv local 3.6.12
(new-lHUWv3XF-py3.6)

11:24:13|new$ poetry env use $(pyenv which python)
Using virtualenv: /Users/juanmirocks/Library/Caches/pypoetry/virtualenvs/new-lHUWv3XF-py3.6
(new-lHUWv3XF-py3.6)

11:24:18|new$ poetry env info

Virtualenv
Python:         3.9.0
Implementation: CPython
Path:           /Users/juanmirocks/Library/Caches/pypoetry/virtualenvs/new-lHUWv3XF-py3.6
Valid:          True

System
Platform: darwin
OS:       posix
Python:   /usr/local/Cellar/[email protected]/3.9.0_2/Frameworks/Python.framework/Versions/3.9
(new-lHUWv3XF-py3.6)

11:24:23|new$ poetry run env python --version
Python 3.9.0
(new-lHUWv3XF-py3.6)

[andrewcrook]

yet somehow poetry's env is "locked" to my system's Homebrew 3.9.0 python version.

@juanmirocks

Read above it is hardcoded to use the current python version from HomeBrew which is a dependency.
With pyenv etc you don't want this, however, will making it use env Python3 break Homebrew's guidelines?

#62910 (comment)

tbh I don't use HomeBrew for pyenv and poetry at this time for all things Python I tend to use pip Python's own package manager. I have a system environment then local/project environments.

I think I can fix it but it might go against the Homebrew guidelines by taking out this dependency.

[juanmirocks]

I understand. Thank you for your clarification.

I cannot tell here with any expertise. I kinda see that Homebrew / pyenv / poetry clash somehow with each other in their assumptions.

I only know that from a user perspective... I would expect to install poetry with Homebrew... and then things "just work".

On top of that, if we keep the recipe as it is, AFAIHK poetry is unusable & broken under Homebrew (unless one always uses the latest Python version). That is, I see 2 solutions:

• either completely remove poetry from Homebrew, or otherwise

• do fix the recipe so that it works as one would expect (that is, it's compatible with Homebrew-installed pyenv).

[carlocab]

I think I can fix it but it might go against the Homebrew guidelines by taking out this dependency.

The best test for that is a PR. I'd appreciate it if you can propose a fix. I don't know if it'll be accepted, but it could be worth a shot.

Now that I've re-read the thread properly, is the fix you had in mind the change suggested here, @andrewcrook? #62910 (comment) If so, then I would be surprised if it weren't accepted. Do you want to open a PR? I'll do it if you'd rather not.

[BrewTestBot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.