Custom test grouping and test group order logic

README.rst

@@ -80,20 +80,133 @@ that worker and report the failure as usual. You can use the
 ``--max-worker-restart`` option to limit the number of workers that can
 be restarted, or disable restarting altogether using ``--max-worker-restart=0``.
 
-By default, the ``-n`` option will send pending tests to any worker that is available, without
-any guaranteed order, but you can control this with these options:
+Dividing tests up
+^^^^^^^^^^^^^^^^^
+
+In order to divide the tests up amongst the workers, ``pytest-xdist`` first puts sets of
+them into "test groups". The tests within a test group are all run together in one shot,
+so fixtures of larger scopes won't be run once for every single test. Instead, they'll
+be run as many times as they need to for the tests within that test group. But, once
+that test group is finished, it should be assumed that all cached fixture values from
+that test group's execution are destroyed.
+
+By default, there is no grouping logic and every individual test is placed in its own
+test group, so using the ``-n`` option will send pending tests to any worker that is
+available, without any guaranteed order. It should be assumed that when using this
+approach, every single test is run entirely in isolation from the others, meaning the
+tests can't rely on cached fixture values from larger-scoped fixtures.
+
+Provided test grouping options
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+By default, ``pytest-xdist`` doesn't group any tests together, but it provides some
+grouping options, based on simple criteria about a test's nodeid. so you can gunarantee
+that certain tests are run in the same process. When they're run in the same process,
+you gunarantee that larger-scoped fixtures are only executed as many times as would
+normally be expected for the tests in the test group. But, once that test group is
+finished, it should be assumed that all cached fixture values from that test group's
+execution are destroyed.
+
+Here's the options that are built in:
+
+* ``--dist=loadscope``: tests will be grouped by **module** shown in each test's node
+  for *test functions* and by the **class** shown in each test's nodeid for *test
+  methods*. This feature was added in version ``1.19``.
+
+* ``--dist=loadfile``: tests will be grouped by the **module** shown in each test's
+  nodeid. This feature was added in version ``1.21``.
+
+Defining custom load distribution logic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``pytest-xdist`` iterates over the entire list of collected tests and usually determines
+what group to put them in based off of their nodeid. There is no set number of test
+groups, as it creates a new groups as needed. You can tap into this system to define
+your own grouping logic by using the ``pytest_xdist_set_test_group_from_nodeid``.
+
+If you define your own copy of that hook, it will be called once for every test, and the
+nodeid for each test will be passed in. Whatever it returns is the test group for that
+test. If a test group doesn't already exist with that name, then it will be created, so
+anything can be used.
+
+For example, let's say you have the following tests::
+
+    test/test_something.py::test_form_upload[image-chrome]
+    test/test_something.py::test_form_upload[image-firefox]
+    test/test_something.py::test_form_upload[video-chrome]
+    test/test_something.py::test_form_upload[video-firefox]
+    test/test_something_else.py::test_form_upload[image-chrome]
+    test/test_something_else.py::test_form_upload[image-firefox]
+    test/test_something_else.py::test_form_upload[video-chrome]
+    test/test_something_else.py::test_form_upload[video-firefox]
+
+In order to have the ``chrome`` related tests run together and the ``firefox`` tests run
+together, but allow them to be separated by file, this could be done:
 
-* ``--dist=loadscope``: tests will be grouped by **module** for *test functions* and
-  by **class** for *test methods*, then each group will be sent to an available worker,
-  guaranteeing that all tests in a group run in the same process. This can be useful if you have
-  expensive module-level or class-level fixtures. Currently the groupings can't be customized,
-  with grouping by class takes priority over grouping by module.
-  This feature was added in version ``1.19``.
+.. code-block:: python
+
+    def pytest_xdist_set_test_group_from_nodeid(nodeid):
+        browser_names = ['chrome', 'firefox']
+        nodeid_params = nodeid.split('[', 1)[-1].rstrip(']').split('-')
+        for name in browser_names:
+            if name in nodeid_params:
+                return "{test_file}[{browser_name}]".format(
+                    test_file=nodeid.split("::", 1)[0],
+                    browser_name=name,
+                )
+
+The tests would then be divided into these test groups:
+
+.. code-block:: python
+
+    {
+        "test/test_something.py::test_form_upload[chrome]" : [
+            "test/test_something.py::test_form_upload[image-chrome]",
+            "test/test_something.py::test_form_upload[video-chrome]"
+        ],
+        "test/test_something.py::test_form_upload[firefox]": [
+            "test/test_something.py::test_form_upload[image-firefox]",
+            "test/test_something.py::test_form_upload[video-firefox]"
+        ],
+        "test/test_something_else.py::test_form_upload[firefox]": [
+            "test/test_something_else.py::test_form_upload[image-firefox]",
+            "test/test_something_else.py::test_form_upload[video-firefox]"
+        ],
+        "test/test_something_else.py::test_form_upload[chrome]": [
+            "test/test_something_else.py::test_form_upload[image-chrome]",
+            "test/test_something_else.py::test_form_upload[video-chrome]"
+        ]
+    }
+
+You can also fall back on one of the default load distribution mechanism by passing the
+arguments for them listed above when you call pytest. Because this example returns
+``None`` if the nodeid doesn't meet any of the criteria, it will defer to whichever
+mechanism you chose. So if you passed ``--dist=loadfile``, tests would otherwise be
+divided up by file name.
+
+Keep in mind, this is a means of optimization, not a means for determinism.
+
+Controlling test group execution order
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Sometimes you may want to have certain test groups start before or after others. Once
+the test groups have been determined, the ``OrderedDict`` they are stored in can have
+its order modified through the ``pytest_xdist_order_test_groups`` hook. For example, in
+order to move the test group named ``"groupA"`` to the end of the queue, this can be
+done:
+
+.. code-block:: python
+
+    def pytest_xdist_order_test_groups(workqueue):
+        workqueue.move_to_end("groupA")
 
-* ``--dist=loadfile``: tests will be grouped by file name, and then will be sent to an available
-  worker, guaranteeing that all tests in a group run in the same worker. This feature was added
-  in version ``1.21``.
+Keep in mind, this is a means of optimization, not a means for determinism or filtering.
+Removing test groups from this ``OrderedDict``, or adding new ones in after the fact can
+have unforseen consequences.
 
+If you want to filter out which tests get run, it is recommended to either rely on test
+suite structure (so you can target the tests in specific locations), or by using marks
+(so you can select or filter out based on specific marks with the ``-m`` flag).
 
 Making session-scoped fixtures execute only once
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

changelog/18.feature.rst

@@ -0,0 +1 @@
+Allow defining of custom logic for test distribution among test groups, and changing the order in which test groups are passed out to workers.

src/xdist/newhooks.py

@@ -55,3 +55,55 @@ def pytest_xdist_node_collection_finished(node, ids):
 @pytest.mark.firstresult
 def pytest_xdist_make_scheduler(config, log):
     """ return a node scheduler implementation """
+
+
+@pytest.mark.trylast
+def pytest_xdist_set_test_group_from_nodeid(nodeid):
+    """Set the test group of a test using its nodeid.
+
+    This will determine which tests are grouped up together and distributed to
+    workers at the same time. This will be called for every test, and whatever
+    is returned will be the name of the test group that test belongs to. In
+    order to have tests be grouped together, this function must return the same
+    value for each nodeid for each test.
+
+    For example, given the following nodeids::
+
+        test/test_something.py::test_form_upload[image-chrome]
+        test/test_something.py::test_form_upload[image-firefox]
+        test/test_something.py::test_form_upload[video-chrome]
+        test/test_something.py::test_form_upload[video-firefox]
+        test/test_something_else.py::test_form_upload[image-chrome]
+        test/test_something_else.py::test_form_upload[image-firefox]
+        test/test_something_else.py::test_form_upload[video-chrome]
+        test/test_something_else.py::test_form_upload[video-firefox]
+
+    In order to have the ``chrome`` related tests run together and the
+    ``firefox`` tests run together, but allow them to be separated by file,
+    this could be done::
+
+        def pytest_xdist_set_test_group_from_nodeid(nodeid):
+            browser_names = ['chrome', 'firefox']
+            nodeid_params = nodeid.split('[', 1)[-1].rstrip(']').split('-')
+            for name in browser_names:
+                if name in nodeid_params:
+                    return "{test_file}[{browser_name}]".format(
+                        test_file=nodeid.split("::", 1)[0],
+                        browser_name=name,
+                    )
+
+    This would then defer to the default distribution logic for any tests this
+    can't apply to (i.e. if this would return ``None`` for a given ``nodeid``).
+    """
+
+@pytest.mark.trylast
+def pytest_xdist_order_test_groups(workqueue):
+    """Sort the queue of test groups to determine the order they will be executed in.
+
+    The ``workqueue`` is an ``OrderedDict`` containing all of the test groups in the
+    order they will be handed out to the workers. Groups that are listed first will be
+    handed out to workers first. The ``workqueue`` only needs to be modified and doesn't
+    need to be returned.
+
+    This can be useful when you want to run longer tests first.
+    """