Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pytest-asyncio Hangs near finish using GPT-3.5 inference fixture #1084

Open
junuMoon opened this issue May 18, 2024 · 0 comments
Open

pytest-asyncio Hangs near finish using GPT-3.5 inference fixture #1084

junuMoon opened this issue May 18, 2024 · 0 comments

Comments

@junuMoon
Copy link

Issue Description

When running pytest-asyncio with pytest-xdist, my tests consistently hang between 90-100% completion. I suspect there might be an issue with resource acquisition not being properly managed at the end of the test suite.

I am using a fixture for running GPT-3.5 inference.

It accepts --models as an argument and uses it to parameterize to create a test suite.

def pytest_generate_tests(metafunc):
    """Generate tests based on custom command-line options."""
    if "model" in metafunc.fixturenames and metafunc.config.option.models:
        models = metafunc.config.option.models
        if models == "all":
            models = list(ModelType)
        elif models == "served":
            models = list(ServedModelType)

        model_fixtures = []
        ids = []
        for model in models:
            if model not in ModelType:
                raise ValueError(f"Unknown model: {model}")
            model_fixtures.append(model)
            ids.append(model)

        metafunc.parametrize("model", model_fixtures, ids=ids, scope="session")

@pytest.mark.asyncio
async def test(model, agenerate_message):
    response = await agenerate_message(model=model)
    passed = "passed" in response
    assert passed

I added a session-scoped fixture to manage the event loop and ensure clean-up, the problem persists.

@pytest.fixture(scope="session")
def event_loop():
    loop = asyncio.get_event_loop()
    yield loop

    pending = asyncio.tasks.all_tasks(loop)
    loop.run_until_complete(asyncio.gather(*pending))
    loop.run_until_complete(asyncio.sleep(1))
    loop.close()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant