Fixture resolution inside a plugin

[knyazer]

Hey, guys,

I was looking into writing a plugin to do mutation testing with pytest: mutation testing is effectively just rerunning the test suite a ton of times, while slightly modifying the source code under tests. While the mutation part itself is fine and done, the part about integrating it with pytest is not so much. I had a few ideas how to implement it, and I have tried them, but it feels very wrong.

1. Clone items a bunch of times in pytest_collection_modifyitems hook: sounds great, until I discovered that this leads to tons of issues with fixtures. I have tried to fix that, but failed, since it requires some manipulations with internal pytest. Also, there was a discussion somewhere about this approach not being recommended, so I abandoned it.
2. Merge all the items into one "test all" item, and then just generate a ton of these items. This approach is inspired by Sybil, that has a very neat piece of code that uses some pytest internals and generates an Item based on the given piece of code. However, the problem is that seemingly the magic-monkeypatching they do doesn't update the fixtures, meaning fixture is not getting torn down after execution of an item that requests it, even if the scope of the fixture is "function". While this is fixable, with even more monkeypatching, I really don't like doing that. But this approach is promising, since I got through 1/3 tests in my test suite for the plugin :) Currently, to run an item inside another item, I do this:

def merged_test():
    for item in items:
        item.funcargs = {}
        session._setupstate.stack[item] = ([item.teardown], None)
        item._request = fixtures.TopRequest(pyfuncitem=item, _ispytest=True)
        item._request._fillfixtures()
        item.runtest()

3. Define my own item collector, and then just recollect every file a bunch of times (number of mutants number of times, e.g. something in the thousands). This is wonderful, but slow. The whole point of the plugin I'm making is to be faster, and not recollecting everything is one of the few reasons to make such a plugin. However, I am almost giving up, so I am probably going to resort to doing it this way, and will see how it goes.

---

The main point of this post is to ask: are you maybe aware of a better method to do what I want? Something that would not involve hacking into pytest internals?
The second point of this post is to document the thing I discover, so that later users won't have to repeat my mistakes of trying to force fixtures to work :)

[bluetech]

What some plugins do, e.g. pytest-rerunfailures, is to run the same item's "runtest protocol" multiple times. I wouldn't say this is sanctioned by pytest's core stability promise, but we're at least aware that this is being done and try not to break it too bad, so it's another thing you can try if your other options don't work out.

[knyazer]

Oh, thanks! That's very useful, but only partly because of what you said: they also have a function that cleans up fixtures,
_remove_cached_results_from_failed_fixtures, that seems to solve my issue with (2)!

I will take a look tomorrow, but you probably saved me a ton of work :)

[knyazer]

Alright, so I spent last week writing stuff: and now I am stuck again! I went with the approach 2, and use the remove_cache... function to removed the cached fixture results:

def merged_test():
    for item in items:
        item.funcargs = {}
        session._setupstate.stack[item] = ([item.teardown], None)
        item._request = fixtures.TopRequest(pyfuncitem=item, _ispytest=True)
        item._request._fillfixtures()
        item.runtest()
        # slightly modified remove_cached_results_from_failed_fixtures
        cached_result = "cached_result"
        fixture_info = getattr(item, "_fixtureinfo", None)
        for fixture_def_str in getattr(fixture_info, "name2fixturedefs", ()):
            fixture_defs = fixture_info.name2fixturedefs[fixture_def_str]
            for fixture_def in fixture_defs:
                setattr(fixture_def, cached_result, None)

While this works wonderfully most of the time, there are some weird edge cases when this just gets absolutely destroyed. The case I encounter is related to the fact that I want to update the __global__ of all the fixtures, and trying to do this just absolutely destroys everything, for some reason. I found a temporal solution, where I update the _request fixtures with

fixturedefs = item._request._arg2fixturedefs
for name in item.fixturenames:
    fixture = fixturedefs[name]
    for i in range(len(fixture)):
        setattr(
            fixture[i],
            "func",
            item.function.__globals__[
                fixture[i].func.__name__
            ].__wrapped__, # wrapped because pytest wraps fixtures into some preprocessor, but we want to modify the original one
        )

This is not all! In addition to that I need to resample all the fixtures (which I have not figured out how to do without ultra ugly hacks), and then it starts to work, somewhat.

However, this leads to even more obscure bugs, that are not even a problem of pytest! Like, some fancy functionality with mocking just stops working, because if at some point you stored the value of the fixture in a closure, thus preserving its original __global__, the closure-available fixture becomes different from the one that is available from the fixture cache everywhere else, and you stop being able to do anything meaningful.

I am again searching for some meaningful links with examples about how to trigger fixture reload from the pytest: something like, if you know the sources changed, how to force the fixtures to update accordingly. I am slowly giving up again....

The two routes to move forward, as I see them, are either:

• Spent a few days figuring out how pytest internals work, and then just make a PR in pytest that would just expose the logic I need :]
• Continue with monkeypatching everything until it magically starts working. Current target is to update the globals (which are readonly :) ) of the fixtures instead of overwriting the whole fixture (which causes "obscure problems" i mentioned), but I am so dissappointed in life that I don't believe this will work :) at least, not quickly.

Anyway, I will keep this discussion up to date until I either give up, or succeed. Since I spent the whole of today figuring out what is wrong, I wanted to feel at least a little bit accomplished, so I wrote this update log.

[knyazer]

Actually, the issues I'm having might be due to the pydantic (at least, there is a significant correlation between using pydantic and the type of failure I'm describing occuring). Gonna look into it tomorrow ig....