Nested parallelism with pytest-xdist and MPI/OpenMP

[gnikit]

I have a test suite containing N tests, meant to run some long and computationally intensive programs, each test class spins up M processes (the number of processes varies from class to class). I was wondering if there was a way to use xdist and some custom weights to load balance the work in the scheduler. Any thoughts?

MWE

This is a dummy example using MPI.

from subprocess import run


class TestParallelMyApp1:
    CORES = 4

    def test_main(self):
        run(["mpiexec", "-n", str(self.CORES), "python3", "-c", "print('App1')"])


class TestParallelMyApp2:
    CORES = 14

    def test_main(self):
        run(["mpiexec", "-n", str(self.CORES), "python3", "-c", "print('App2')"])


class TestParallelMyApp3:
    CORES = 10

    def test_main(self):
        run(["mpiexec", "-n", str(self.CORES), "python3", "-c", "print('App3')"])

Desired Outcome

On a machine with 14 cores, running pytest -n auto would run:

• TestParallelMyApp1 and TestParallelMyApp3
• and then TestParallelMyApp2