Where nox should be installed

[crisfcodes]

In the second chapter of Hypermodern Python you use pip install --user --upgrade nox to install nox did you do this inside the virtualenvironment created by poetry? OR you deactivate the virtualenvironment and install globally nox?

If you did it inside poetry virtualenvironment you use pip instead of poetry so that it doesn't stay in the pyptoject.toml packages?

[cjolowicz]

The pip command installs Nox globally, into a user-specific directory.

Poetry’s virtual environment for your project is not activated in a normal shell. You use the poetry commands to modify it.

[crisfcodes]

Thank you for responding so quickly.

I have some questions, I use pylint instead flake8 and I have configured it to detect when pylint has been unable to import a module(import-error/E0401). Because nox creates a new virtual environment, pylint searches that environment instead of my development virtual environment. To solve this and not let pylint complain I have to install all the dependencies like you do for the tests session, because of this, this task becomes very slow. Is there any way to make this process more efficient or to make nox ignore the virtual environment that it creates and searches in the development virtual environment?

[cjolowicz]

Without seeing your project, two things come to mind.

The first is to re-use the virtualenv by passing --reuse-existing-virtualenvs to Nox. While poetry install is still run, this should be much faster as it will only perform updates if the versions in your environment do not match those in the lock file.

The second idea is a little more subtle. Do you use the --no-dev option of poetry install, followed by a manual installation of pylint? And is pylint listed as a development dependency? If so, Poetry will uninstall pylint on each run, which will slow down the session. For this and other reasons, I have recently switched to use only pip install, and export dependencies from Poetry to requirements.txt format on the fly. See the Hypermodern Python Cookiecutter for how to do this.

[crisfcodes]

Can You explain to me a little better what you do inside your sessions. You create in each run of your session the file requirements.txt and then install with pip -r? or do you create it once with poetry and then use it every session, and update it every time you install new modules?

PD: You were right in the second assumption, I use pylint as a development dependency and I use poetry install --no-dev. I'll leave you the session I'm trying to improve on. Every time that i run my session, this installs, upgrades, and removes packages according to your feedback my session install the no-dev dependencies, after this dev dependencias are removed. This is the output: Package operations: 1 install, 13 updates, 12 removals.

@nox.session(python="3.8")
def pylint(session: Session) -> None:
    """Lint using pylint."""

    args = session.posargs or locations
    session.run("poetry", "install", "--no-dev", external=True)
    install_with_constraints(
        session, "pylint", "pylint_django", "pylint_celery",
    )
    session.run("pylint", *args)

[cjolowicz]

You create in each run of your session the file requirements.txt and then install with pip -r? or do you create it once with poetry and then use it every session, and update it every time you install new modules?

The requirements.txt file is generated every time the session is run. It is not installed with pip install -r. Dependencies are installed with pip install <package>, but the requirements.txt file is passed as a constraints file via the --constraint option. This ensures that the versions in the constraints file are used for all dependencies and subdependencies. In other words, we selectively install packages instead of the entirety of dependencies, but honor the version pins for all dependencies.

Note that pylint and the plugins are also installed in the Poetry environment. So if you just need to quickly lint some files, you can always poetry run pylint. Use Nox to run full checks in a repeatable way in isolated environments. Use the Poetry environment to interact with your package during development, and to quickly run one-off checks and tasks.

A third option is to run pylint via pre-commit. This will check files staged for a commit, and only perform installs the first time the check runs.

[wanderrful]

Piggybacking off of this to ask some questions as well.

First, thank you very much @cjolowicz for your series of blog posts! They were super enlightening for me.

A recurring issue I'm having with those Nox sessions is that for the sessions you advised using --no-dev it will end up actually uninstalling everything in the relevant Python's virtualenv (i.e. the same env that Poetry is using). I figure that it has to do with external=True being used, though perhaps I may have missed something in your initial instructions for starting up Pyenv and Virtualenv with .bash_profile.

Have you encountered this issue yourself, and if so how what guidance would you have to deal with it?

[cjolowicz]

@wanderrful This is a known issue. Please see #111 (which has further links).