Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support for reuse_venv option #730

Merged
merged 12 commits into from
Feb 25, 2024
5 changes: 3 additions & 2 deletions docs/config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ Use of :func:`session.install()` is deprecated without a virtualenv since it mod
def tests(session):
session.run("pip", "install", "nox")

You can also specify that the virtualenv should *always* be reused instead of recreated every time:
You can also specify that the virtualenv should *always* be reused instead of recreated every time unless ``--reuse-venv=never``:

.. code-block:: python

Expand Down Expand Up @@ -432,7 +432,8 @@ The following options can be specified in the Noxfile:
* ``nox.options.tags`` is equivalent to specifying :ref:`-t or --tags <opt-sessions-pythons-and-keywords>`.
* ``nox.options.default_venv_backend`` is equivalent to specifying :ref:`-db or --default-venv-backend <opt-default-venv-backend>`.
* ``nox.options.force_venv_backend`` is equivalent to specifying :ref:`-fb or --force-venv-backend <opt-force-venv-backend>`.
* ``nox.options.reuse_existing_virtualenvs`` is equivalent to specifying :ref:`--reuse-existing-virtualenvs <opt-reuse-existing-virtualenvs>`. You can force this off by specifying ``--no-reuse-existing-virtualenvs`` during invocation.
* ``nox.options.reuse_venv`` is equivalent to specifying :ref:`--reuse-venv <opt-reuse-venv>`. Preferred over using ``nox.options.reuse_existing_virtualenvs``.
* ``nox.options.reuse_existing_virtualenvs`` is equivalent to specifying :ref:`--reuse-existing-virtualenvs <opt-reuse-existing-virtualenvs>`. You can force this off by specifying ``--no-reuse-existing-virtualenvs`` during invocation. Alias of ``nox.options.reuse_venv``.
* ``nox.options.stop_on_first_error`` is equivalent to specifying :ref:`--stop-on-first-error <opt-stop-on-first-error>`. You can force this off by specifying ``--no-stop-on-first-error`` during invocation.
* ``nox.options.error_on_missing_interpreters`` is equivalent to specifying :ref:`--error-on-missing-interpreters <opt-error-on-missing-interpreters>`. You can force this off by specifying ``--no-error-on-missing-interpreters`` during invocation.
* ``nox.options.error_on_external_run`` is equivalent to specifying :ref:`--error-on-external-run <opt-error-on-external-run>`. You can force this off by specifying ``--no-error-on-external-run`` during invocation.
Expand Down
21 changes: 17 additions & 4 deletions docs/usage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -179,34 +179,47 @@ Finally note that the ``--no-venv`` flag is a shortcut for ``--force-venv-backen
nox --no-venv

.. _opt-reuse-existing-virtualenvs:
.. _opt-reuse-venv:

Re-using virtualenvs
--------------------

By default, Nox deletes and recreates virtualenvs every time it is run. This is usually fine for most projects and continuous integration environments as `pip's caching <https://pip.pypa.io/en/stable/cli/pip_install/#caching>`_ makes re-install rather quick. However, there are some situations where it is advantageous to reuse the virtualenvs between runs. Use ``-r`` or ``--reuse-existing-virtualenvs``:
By default, Nox deletes and recreates virtualenvs every time it is run. This is
usually fine for most projects and continuous integration environments as
`pip's caching <https://pip.pypa.io/en/stable/cli/pip_install/#caching>`_ makes
re-install rather quick. However, there are some situations where it is
advantageous to reuse the virtualenvs between runs. Use ``-r`` or
``--reuse-existing-virtualenvs`` or for fine-grained control use
``--reuse-venv=yes|no|always|never``:

.. code-block:: console

nox -r
nox --reuse-existing-virtualenvs

nox --reuse-venv=yes # preferred

If the Noxfile sets ``nox.options.reuse_existing_virtualenvs``, you can override the Noxfile setting from the command line by using ``--no-reuse-existing-virtualenvs``.
Similarly you can override ``nox.options.reuse_venvs`` from the Noxfile via the command line by using ``--reuse-venv=yes|no|always|never``.

Additionally, you can skip the re-installation of packages when a virtualenv is reused. Use ``-R`` or ``--reuse-existing-virtualenvs --no-install``:
Additionally, you can skip the re-installation of packages when a virtualenv is reused.
Use ``-R`` or ``--reuse-existing-virtualenvs --no-install`` or ``--reuse-venv=yes --no-install``:

.. code-block:: console

nox -R
nox --reuse-existing-virtualenvs --no-install
nox --reuse-venv=yes --no-install

The ``--no-install`` option causes the following session methods to return early:

- :func:`session.install <nox.sessions.Session.install>`
- :func:`session.conda_install <nox.sessions.Session.conda_install>`
- :func:`session.run_install <nox.sessions.Session.run_install>`

This option has no effect if the virtualenv is not being reused.
The ``never`` and ``always`` options in ``--reuse-venv`` gives you more fine-grained control
as it ignores when a ``@nox.session`` has ``reuse_venv=True|False`` defined.

These options have no effect if the virtualenv is not being reused.

.. _opt-running-extra-pythons:

Expand Down
58 changes: 56 additions & 2 deletions nox/_options.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,13 @@
from nox import _option_set
from nox.tasks import discover_manifest, filter_manifest, load_nox_module

if sys.version_info < (3, 8): # pragma: no cover
from typing_extensions import Literal
else: # pragma: no cover
from typing import Literal
samypr100 marked this conversation as resolved.
Show resolved Hide resolved

ReuseVenvType = Literal["no", "yes", "never", "always"]

"""All of Nox's configuration options."""

options = _option_set.OptionSet(
Expand Down Expand Up @@ -148,6 +155,30 @@ def _envdir_merge_func(
return command_args.envdir or noxfile_args.envdir or ".nox"


def _reuse_venv_merge_func(
samypr100 marked this conversation as resolved.
Show resolved Hide resolved
command_args: argparse.Namespace, noxfile_args: argparse.Namespace
) -> ReuseVenvType:
"""Merge reuse_venv from command args and Noxfile while maintaining
backwards compatibility with reuse_existing_virtualenvs. Default is "no".

Args:
command_args (_option_set.Namespace): The options specified on the
command-line.
noxfile_Args (_option_set.Namespace): The options specified in the
Noxfile.
"""
# back-compat scenario with no_reuse_existing_virtualenvs/reuse_existing_virtualenvs
if command_args.no_reuse_existing_virtualenvs:
return "no"
if (
command_args.reuse_existing_virtualenvs
or noxfile_args.reuse_existing_virtualenvs
):
return "yes"
# regular option behavior
return command_args.reuse_venv or noxfile_args.reuse_venv or "no"


def default_env_var_list_factory(env_var: str) -> Callable[[], list[str] | None]:
"""Looks at the env var to set the default value for a list of env vars.

Expand Down Expand Up @@ -199,12 +230,22 @@ def _force_pythons_finalizer(


def _R_finalizer(value: bool, args: argparse.Namespace) -> bool:
"""Propagate -R to --reuse-existing-virtualenvs and --no-install."""
"""Propagate -R to --reuse-existing-virtualenvs and --no-install and --reuse-venv=yes."""
if value:
args.reuse_venv = "yes"
args.reuse_existing_virtualenvs = args.no_install = value
return value


def _reuse_existing_virtualenvs_finalizer(
value: bool, args: argparse.Namespace
) -> bool:
"""Propagate --reuse-existing-virtualenvs to --reuse-venv=yes."""
if value:
args.reuse_venv = "yes"
return value


def _posargs_finalizer(
value: Sequence[Any], args: argparse.Namespace
) -> Sequence[Any] | list[Any]:
Expand Down Expand Up @@ -414,6 +455,18 @@ def _tag_completer(
" creating a venv. This is an alias for '--force-venv-backend none'."
),
),
_option_set.Option(
"reuse_venv",
"--reuse-venv",
group=options.groups["environment"],
noxfile=True,
merge_func=_reuse_venv_merge_func,
help=(
"Controls existing virtualenvs recreation. This is ``'no'`` by"
" default, but any of ``('yes', 'no', 'always', 'never')`` are accepted."
),
choices=["yes", "no", "always", "never"],
),
*_option_set.make_flag_pair(
"reuse_existing_virtualenvs",
("-r", "--reuse-existing-virtualenvs"),
Expand All @@ -422,7 +475,8 @@ def _tag_completer(
"--no-reuse-existing-virtualenvs",
),
group=options.groups["environment"],
help="Reuse existing virtualenvs instead of recreating them.",
help="This is an alias for '--reuse-venv=yes|no'.",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd love to see this called out in the docs as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be in the docs, near the end of this statement in config.rst:

* ``nox.options.reuse_existing_virtualenvs`` is equivalent to specifying :ref:--reuse-existing-virtualenvs . You can force this off by specifying ``--no-reuse-existing-virtualenvs`` during invocation. Alias of ``nox.options.reuse_venv``.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs should state explicitly that --reuse-venv=yes is equivalent to --reuse-existing-virtualenvs.

I'm happy for this to land as-is though. We probably want to deprecate the longer version anyway.

(OT: It might also be a nice improvement to allow specifying --reuse-venv without a value, defaulting to yes.)

Copy link
Contributor Author

@samypr100 samypr100 Feb 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, I didn't explicitly deprecate it on this PR but it can be done so in a follow up one. When that happens I think it makes sense to augment --reuse-venv (or add --reuse/--no-reuse like @henryiii prefers) to default to yes when specified. Note -r is a valid shorthand for --reuse-venv=yes since it's a shorcut to --reuse-existing-virtualenvs .

Copy link
Contributor Author

@samypr100 samypr100 Feb 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs should state explicitly that --reuse-venv=yes is equivalent to --reuse-existing-virtualenvs.

Done in fdf876c and 88d11b7

finalizer_func=_reuse_existing_virtualenvs_finalizer,
),
_option_set.Option(
"R",
Expand Down
18 changes: 15 additions & 3 deletions nox/sessions.py
Original file line number Diff line number Diff line change
Expand Up @@ -767,9 +767,7 @@ def _create_venv(self) -> None:
self.venv = PassthroughEnv()
return

reuse_existing = (
self.func.reuse_venv or self.global_config.reuse_existing_virtualenvs
)
reuse_existing = self.reuse_existing_venv()

if backend is None or backend in {"virtualenv", "venv", "uv"}:
self.venv = VirtualEnv(
Expand All @@ -795,6 +793,20 @@ def _create_venv(self) -> None:

self.venv.create()

def reuse_existing_venv(self) -> bool:
return any(
(
# forces every session to reuse its env
self.global_config.reuse_venv == "always",
# sessions marked True will always be reused unless never is specified
self.func.reuse_venv is True
and self.global_config.reuse_venv != "never",
# session marked False will never be reused unless always is specified
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment doesn't relate to the code below

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to clarify this function with even more documentation here 715bc9a, let me know if it's better.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is super helpful

self.func.reuse_venv is not False
and self.global_config.reuse_venv == "yes",
)
)

def execute(self) -> Result:
logger.warning(f"Running session {self.friendly_name}")

Expand Down
46 changes: 46 additions & 0 deletions tests/conftest.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,23 @@
# Copyright 2023 Alethea Katherine Flowers
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from __future__ import annotations

import re
from pathlib import Path
from string import Template
from typing import Callable

import pytest


Expand All @@ -6,3 +26,29 @@ def reset_color_envvars(monkeypatch):
"""Remove color-related envvars to fix test output"""
monkeypatch.delenv("FORCE_COLOR", raising=False)
monkeypatch.delenv("NO_COLOR", raising=False)


RESOURCES = Path(__file__).parent.joinpath("resources")


@pytest.fixture
def generate_noxfile_options(tmp_path: Path) -> Callable[..., str]:
"""Generate noxfile.py with test and templated options.

The options are enabled (if disabled) and the values are applied
if a matching format string is encountered with the option name.
"""

def generate_noxfile(**option_mapping: str | bool) -> str:
path = Path(RESOURCES) / "noxfile_options.py"
text = path.read_text(encoding="utf8")
if option_mapping:
for opt, _val in option_mapping.items():
# "uncomment" options with values provided
text = re.sub(rf"(# )?nox.options.{opt}", f"nox.options.{opt}", text)
text = Template(text).safe_substitute(**option_mapping)
path = tmp_path / "noxfile.py"
path.write_text(text)
return str(path)

return generate_noxfile
5 changes: 3 additions & 2 deletions tests/resources/noxfile_options.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,9 @@

import nox

nox.options.reuse_existing_virtualenvs = True
# nox.options.error_on_missing_interpreters = {error_on_missing_interpreters} # used by tests
# nox.options.reuse_existing_virtualenvs = ${reuse_existing_virtualenvs}
# nox.options.reuse_venv = "${reuse_venv}"
# nox.options.error_on_missing_interpreters = ${error_on_missing_interpreters}
nox.options.sessions = ["test"]


Expand Down
Loading
Loading