Please enable deterministic scheduling in the pytest plugin under Trio

[joelb123]

Things to check first

• I have searched the existing issues and didn't find my feature already requested there

Feature description

I request a pytest plugin feature to enable deterministic testing, similar to what is done under pytest-trio. I do not see any easy way to do this for the asyncio backend.

I have made a feature request in trio to enable keywording of this as a backend feature, which is a far preferable way of implementing the same thing.

Use case

Deterministic scheduling is very handy in many test scenarios. Trio allows somewhat-deterministic scheduling through monkey-patching its event loop generator to set _ALLOW_DETERMINISTIC_SCHEDULING and replacing its random-value generator with a seeded one. Both of which are done by pytest-trio, but not by the AnyIO pytest plugin. I say "somewhat-deterministic" because differences in thread timing based on CPU utilization can still introduce stochastic elements, even in fully mocked-up code.

AFAIK, it's not similarly easy to achieve deterministic-ish scheduling with asyncio, although I have seen code that injects additional jobs into the event loop to make a state machine of sorts. One thing that attracted me to AnyIO is to do deterministic (and non-performant) testing with trio and then use asyncio+uvloop in production.

[agronholm]

I could add that to AnyIO 4.0. As for the asyncio backend, it already schedules everything deterministically.
A PR would ensure that this gets in.

[joelb123]

Thanks for the prompt response. I'll try to get you a PR.

I'm intrigued by the statement--which you are quite knowledgeable about--that asyncio schedules everything deterministically. Do you mean that in the sense of "it's a deterministic algorithm, but you can't control the random choices made"? I looked into the python-side asyncio code but I couldn't find any way to seed asyncio, and simply using random.seed(1234) doesn't accomplish anything for my code.

Can you point me to an example of code doing deterministic scheduling with asyncio? The only things I have seen involve injection into the event loop that bypasses scheduling altogether.

[joelb123]

After some testing, I have found a simple workaround, which is to simply install pytest-trio. The plugin gets loaded at pytest startup, and the monkey-patching gets done at that time. Patching happens regardless of the setting of "trio_mode" in pytest.ini.

I think it's simpler and better to include pytest-trio to the trio dependencies. If you prefer not to do that, you could simply put something along the lines of "if you want deterministic testing using pytest under trio, then include pytest-trio as a dependency" to the documentation.

[agronholm]

When I say "deterministic scheduling" is that the asyncio event loop runs callbacks (including tasks) in the order in which they were added or rescheduled. This is in contrast to trio which can arbitrarily rearrange the order in which tasks are woken up.