[Python 3.13.0b2] test_pdb_used_outside_test and test_pdb_used_in_generate_tests are failing with Python 3.13

[hrnciar]

Hello, test_pdb_used_outside_test and test_pdb_used_in_generate_tests are failing with Python 3.13 beta 2 on Fedora Rawhide. The same error was observed also with pytest 7.4.3.

+ /builddir/build/BUILD/pytest-8.2.2-build/BUILDROOT/usr/bin/pytest testing --timeout=30 -n auto -rs
============================= test session starts ==============================
platform linux -- Python 3.13.0b2, pytest-8.2.2, pluggy-1.5.0
rootdir: /builddir/build/BUILD/pytest-8.2.2-build/pytest-8.2.2
configfile: pyproject.toml
plugins: timeout-2.3.1, xdist-3.6.1, hypothesis-6.98.8
timeout: 30.0s
timeout method: signal
timeout func_only: False
created: 2/2 workers
2 workers [3663 items]

........................................................................ [  1%]
......................................................ss................ [  3%]
............s...................................................x....... [  5%]
........................................................................ [  7%]
....................................Xs.................................. [  9%]
...sssssss............................x................................. [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 23%]
.............s.......................................................... [ 25%]
........................................................................ [ 27%]
...........................................s............................ [ 29%]
........................................................................ [ 31%]
..........................s............................................. [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
.x.............................................s........................ [ 43%]
.........sssss.ssss..................................................... [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
......................................................................... [ 53%]
.x........................................................x............. [ 55%]
......................................................................x. [ 57%]
........................................................................ [ 58%]
..........................s............................................s [ 60%]
................................................................x....... [ 62%]
........................................................................ [ 64%]
.................................s.x.................................... [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
.................................................x...................... [ 74%]
........................................................................ [ 76%]
.............................................................s.......... [ 78%]
...............................................s.......................x [ 80%]
........................x............................................... [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 90%]
......................................................................... [ 92%]
............................s........................................... [ 94%]
..sss...........................x......................................s [ 96%]
......X....XX..s...................................x.................... [ 98%]
.............................................xX....FF........            [100%]
=================================== FAILURES ===================================
______________________ TestPDB.test_pdb_used_outside_test ______________________
[gw1] linux -- Python 3.13.0 /usr/bin/python3

self = <test_debugging.TestPDB object at 0x7f9fbabf6d50>
pytester = <Pytester PosixPath('/tmp/pytest-of-mockbuild/pytest-0/popen-gw1/test_pdb_used_outside_test0')>

    def test_pdb_used_outside_test(self, pytester: Pytester) -> None:
        p1 = pytester.makepyfile(
            """
            import pytest
            pytest.set_trace()
            x = 5
        """
        )
        child = pytester.spawn(f"{sys.executable} {p1}")
>       child.expect("x = 5")

/builddir/build/BUILD/pytest-8.2.2-build/pytest-8.2.2/testing/test_debugging.py:771: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.13/site-packages/pexpect/spawnbase.py:354: in expect
    return self.expect_list(compiled_pattern_list,
/usr/lib/python3.13/site-packages/pexpect/spawnbase.py:383: in expect_list
    return exp.expect_loop(timeout)
/usr/lib/python3.13/site-packages/pexpect/expect.py:181: in expect_loop
    return self.timeout(e)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pexpect.expect.Expecter object at 0x7f9fa1526510>
err = TIMEOUT('Timeout exceeded.')

    def timeout(self, err=None):
        spawn = self.spawn
    
        spawn.before = spawn._before.getvalue()
        spawn.after = TIMEOUT
        index = self.searcher.timeout_index
        if index >= 0:
            spawn.match = TIMEOUT
            spawn.match_index = index
            return index
        else:
            spawn.match = None
            spawn.match_index = None
            msg = str(spawn)
            msg += '\nsearcher: %s' % self.searcher
            if err is not None:
                msg = str(err) + '\n' + msg
    
            exc = TIMEOUT(msg)
            exc.__cause__ = None    # in Python 3.x we can use "raise exc from None"
>           raise exc
E           pexpect.exceptions.TIMEOUT: Timeout exceeded.
E           <pexpect.pty_spawn.spawn object at 0x7f9fa1526900>
E           command: /usr/bin/python3
E           args: ['/usr/bin/python3', '/tmp/pytest-of-mockbuild/pytest-0/popen-gw1/test_pdb_used_outside_test0/test_pdb_used_outside_test.py']
E           buffer (last 100 chars): b'est_pdb_used_outside_test0/test_pdb_used_outside_test.py(2)<module>()\r\n-> pytest.set_trace()\r\n(Pdb) '
E           before (last 100 chars): b'est_pdb_used_outside_test0/test_pdb_used_outside_test.py(2)<module>()\r\n-> pytest.set_trace()\r\n(Pdb) '
E           after: <class 'pexpect.exceptions.TIMEOUT'>
E           match: None
E           match_index: None
E           exitstatus: None
E           flag_eof: False
E           pid: 1483
E           child_fd: 18
E           closed: False
E           timeout: 10.0
E           delimiter: <class 'pexpect.exceptions.EOF'>
E           logfile: <_io.BufferedWriter name='/tmp/pytest-of-mockbuild/pytest-0/popen-gw1/test_pdb_used_outside_test0/spawn.out'>
E           logfile_read: None
E           logfile_send: None
E           maxread: 2000
E           ignorecase: False
E           searchwindowsize: None
E           delaybeforesend: 0.05
E           delayafterclose: 0.1
E           delayafterterminate: 0.1
E           searcher: searcher_re:
E               0: re.compile(b'x = 5')

/usr/lib/python3.13/site-packages/pexpect/expect.py:144: TIMEOUT
___________________ TestPDB.test_pdb_used_in_generate_tests ____________________
[gw1] linux -- Python 3.13.0 /usr/bin/python3

self = <test_debugging.TestPDB object at 0x7f9fbabf6df0>
pytester = <Pytester PosixPath('/tmp/pytest-of-mockbuild/pytest-0/popen-gw1/test_pdb_used_in_generate_tests0')>

    def test_pdb_used_in_generate_tests(self, pytester: Pytester) -> None:
        p1 = pytester.makepyfile(
            """
            import pytest
            def pytest_generate_tests(metafunc):
                pytest.set_trace()
                x = 5
            def test_foo(a):
                pass
        """
        )
        child = pytester.spawn_pytest(str(p1))
>       child.expect("x = 5")

/builddir/build/BUILD/pytest-8.2.2-build/pytest-8.2.2/testing/test_debugging.py:788: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.13/site-packages/pexpect/spawnbase.py:354: in expect
    return self.expect_list(compiled_pattern_list,
/usr/lib/python3.13/site-packages/pexpect/spawnbase.py:383: in expect_list
    return exp.expect_loop(timeout)
/usr/lib/python3.13/site-packages/pexpect/expect.py:181: in expect_loop
    return self.timeout(e)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pexpect.expect.Expecter object at 0x7f9fa135d010>
err = TIMEOUT('Timeout exceeded.')

    def timeout(self, err=None):
        spawn = self.spawn
    
        spawn.before = spawn._before.getvalue()
        spawn.after = TIMEOUT
        index = self.searcher.timeout_index
        if index >= 0:
            spawn.match = TIMEOUT
            spawn.match_index = index
            return index
        else:
            spawn.match = None
            spawn.match_index = None
            msg = str(spawn)
            msg += '\nsearcher: %s' % self.searcher
            if err is not None:
                msg = str(err) + '\n' + msg
    
            exc = TIMEOUT(msg)
            exc.__cause__ = None    # in Python 3.x we can use "raise exc from None"
>           raise exc
E           pexpect.exceptions.TIMEOUT: Timeout exceeded.
E           <pexpect.pty_spawn.spawn object at 0x7f9fa135ecf0>
E           command: /usr/bin/python3
E           args: ['/usr/bin/python3', '-mpytest', '--basetemp=/tmp/pytest-of-mockbuild/pytest-0/popen-gw1/test_pdb_used_in_generate_tests0/temp-pexpect', '/tmp/pytest-of-mockbuild/pytest-0/popen-gw1/test_pdb_used_in_generate_tests0/test_pdb_used_in_generate_tests.py']
E           buffer (last 100 chars): b'e_tests0/test_pdb_used_in_generate_tests.py(3)pytest_generate_tests()\r\n-> pytest.set_trace()\r\n(Pdb) '
E           before (last 100 chars): b'e_tests0/test_pdb_used_in_generate_tests.py(3)pytest_generate_tests()\r\n-> pytest.set_trace()\r\n(Pdb) '
E           after: <class 'pexpect.exceptions.TIMEOUT'>
E           match: None
E           match_index: None
E           exitstatus: None
E           flag_eof: False
E           pid: 1484
E           child_fd: 19
E           closed: False
E           timeout: 10.0
E           delimiter: <class 'pexpect.exceptions.EOF'>
E           logfile: <_io.BufferedWriter name='/tmp/pytest-of-mockbuild/pytest-0/popen-gw1/test_pdb_used_in_generate_tests0/spawn.out'>
E           logfile_read: None
E           logfile_send: None
E           maxread: 2000
E           ignorecase: False
E           searchwindowsize: None
E           delaybeforesend: 0.05
E           delayafterclose: 0.1
E           delayafterterminate: 0.1
E           searcher: searcher_re:
E               0: re.compile(b'x = 5')

/usr/lib/python3.13/site-packages/pexpect/expect.py:144: TIMEOUT
=========================== short test summary info ============================
SKIPPED [1] testing/_py/test_local.py:464: sys.version_info < (3,6)
SKIPPED [1] testing/_py/test_local.py:469: sys.version_info < (3,6)
SKIPPED [1] testing/_py/test_local.py:669: case-insensitive only on windows
SKIPPED [1] testing/_py/test_local.py:890: condition: not (sys.platform == 'win32' or getattr(os, '_name', None) == 'nt')
SKIPPED [1] testing/_py/test_local.py:1242: condition: not (sys.platform == 'win32' or getattr(os, '_name', None) == 'nt')
SKIPPED [1] testing/_py/test_local.py:1248: condition: not (sys.platform == 'win32' or getattr(os, '_name', None) == 'nt')
SKIPPED [1] testing/_py/test_local.py:1260: condition: not (sys.platform == 'win32' or getattr(os, '_name', None) == 'nt')
SKIPPED [1] testing/_py/test_local.py:1266: condition: not (sys.platform == 'win32' or getattr(os, '_name', None) == 'nt')
SKIPPED [1] testing/_py/test_local.py:1271: condition: not (sys.platform == 'win32' or getattr(os, '_name', None) == 'nt')
SKIPPED [1] testing/_py/test_local.py:1279: condition: not (sys.platform == 'win32' or getattr(os, '_name', None) == 'nt')
SKIPPED [1] testing/_py/test_local.py:1286: condition: not (sys.platform == 'win32' or getattr(os, '_name', None) == 'nt')
SKIPPED [1] testing/test_tmpdir.py:359: win only
SKIPPED [1] testing/test_capture.py:1435: only on windows
SKIPPED [1] testing/test_findpaths.py:142: condition: sys.platform != 'win32'
SKIPPED [1] testing/acceptance_test.py:1424: Test is not isolated
SKIPPED [9] testing/code/test_excinfo.py:1678: could not import 'exceptiongroup': No module named 'exceptiongroup'
SKIPPED [1] ../BUILDROOT/usr/lib/python3.13/site-packages/_pytest/pathlib.py:473: symlinks not supported: [Errno 17] File exists: '/tmp/pytest-of-mockbuild/pytest-0/popen-gw1/test_collect_symlink_dir0/symlink_dir' -> '/tmp/pytest-of-mockbuild/pytest-0/popen-gw1/test_collect_symlink_dir0/dir'
SKIPPED [1] testing/test_collection.py:1784: Windows only
SKIPPED [1] testing/test_conftest.py:397: only relevant for case-insensitive file systems
SKIPPED [1] testing/test_pathlib.py:547: Windows only
SKIPPED [1] testing/test_pluginmanager.py:108: requires a case-insensitive file system
SKIPPED [1] testing/test_unittest.py:1326: could not import 'asynctest': No module named 'asynctest'
SKIPPED [3] testing/test_warnings.py:518: not relevant until pytest 9.0
SKIPPED [1] testing/test_compat.py:97: coroutine removed
SKIPPED [1] testing/test_faulthandler.py:72: sometimes crashes on CI (#7022)
= 2 failed, 3607 passed, 35 skipped, 14 xfailed, 5 xpassed in 115.37s (0:01:55) =

[webknjaz]

Interestingly, this isn't failing in pytest's own CI — what's the difference with the Ubuntu env?

There's a new deprecation warning in the third beta, though: #12552.

[hrnciar]

With pytest 8.3.1 and Python 3.13.0b4, the tests mentioned above are still failing, there is also one new failing test -test_pdb_and_capsys.

EDIT: With multiple runs the test_pdb_and_capsys is passing, so the only two tests from the original report are problematic.

[jaraco]

Is this related to the issue I'm having, where the pdb prompt is unresponsive to me? Under Python 3.13rc2, I run a failing test with --pdb and it stalls:

 draft 🐚 cat > test_fail.py
def test_fail():
  raise ValueError()
 draft 🐚 pipx run pytest
⚠️ Found a space in the pipx home path. We heavily discourage this, due to multiple incompatibilities. Please check our docs for more information
    on this, as well as some pointers on how to migrate to a different home path.
=============================================================== test session starts ===============================================================
platform darwin -- Python 3.13.0rc2+, pytest-8.3.3, pluggy-1.5.0
rootdir: /Users/jaraco/draft
collected 1 item                                                                                                                                  

test_fail.py F                                                                                                                              [100%]

==================================================================== FAILURES =====================================================================
____________________________________________________________________ test_fail ____________________________________________________________________

    def test_fail():
>     raise ValueError()
E     ValueError

test_fail.py:2: ValueError
============================================================= short test summary info =============================================================
FAILED test_fail.py::test_fail - ValueError
================================================================ 1 failed in 0.01s ================================================================
 draft [1] 🐚 pipx run pytest --pdb
⚠️ Found a space in the pipx home path. We heavily discourage this, due to multiple incompatibilities. Please check our docs for more information
    on this, as well as some pointers on how to migrate to a different home path.
=============================================================== test session starts ===============================================================
platform darwin -- Python 3.13.0rc2+, pytest-8.3.3, pluggy-1.5.0
rootdir: /Users/jaraco/draft
collected 1 item                                                                                                                                  

test_fail.py F
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

    def test_fail():
>     raise ValueError()
E     ValueError

test_fail.py:2: ValueError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/jaraco/draft/test_fail.py(2)test_fail()
-> raise ValueError()
 <--- no prompt, unresponsive

Edit: I'm unable to replicate the issue in Python 3.13rc2 as found in deadsnakes on Ubuntu, so the issue is probably with my manual build.

[bluthej]

@jaraco I'm having the same issue you mentioned with the official 3.13.0 release and pytest 8.3.3. I also just tested it with the current pytest main branch (8.4.0.dev109+gf37397470) and it's the same thing: if I pytest hits a breakpoint I don't have the pdb prompt and it's totally unresponsive (even ctrl+c cannot get me out of there).

But I don't know if it's related to the original issue or if this should be a separate one.

[The-Compiler]

I believe the failed tests in the OP are due to pdb now showing the breakpoint line instead of the next line after it.

With

def test_pdb():
    breakpoint()
    x = 5

compare Python 3.12

============================= test session starts ==============================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
[...]

test_pdb.py 
>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>
> [...]/test_pdb.py(3)test_pdb()
-> x = 5     # [!!!]
(Pdb) q

!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: Quitting debugger !!!!!!!!!!!!!!!!!!!
============================ no tests ran in 9.02s =============================

with 3.13:

============================= test session starts ==============================
platform linux -- Python 3.13.0, pytest-8.4.0.dev109+gf37397470, pluggy-1.5.0
[...]

test_pdb.py 
>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>
> [...]/test_pdb.py(2)test_pdb()
-> breakpoint()   # [!!!]
(Pdb) q

!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: Quitting debugger !!!!!!!!!!!!!!!!!!!
============================ no tests ran in 5.95s =============================

And the tests just seem to be expecting the former rather than the latter.

As you can see, however, I cannot reproduce any issues with using pdb interactively, which seems to be a separate issue. @jaraco @bluthej I suggest you open an issue. @bluthej if you're on macOS as well, it might be something specific to that.

[bluthej]

@The-Compiler I'm actually on Linux, but I realize that I should have mentioned that I installed Python 3.13 with uv, which means that I'm actually using a version from python-build-standalone so it might be due to that... I'll check with the official install and I guess if I can't reproduce it it's a python-build-standalone issue.

[The-Compiler]

Interestingly, this isn't failing in pytest's own CI — what's the difference with the Ubuntu env?

We skip almost all test_debugging.py tests on CI except on Python 3.8... See #12884.

I'm actually on Linux, but I realize that I should have mentioned that I installed Python 3.13 with uv, which means that I'm actually using a version from python-build-standalone so it might be due to that... I'll check with the official install and I guess if I can't reproduce it it's a python-build-standalone issue.

Seems like a good thing to test, but please open an issue even if not. I think it's still worth investigating whether the issue is on pytest's side, and if so, we should see what we can do about it. Even if it only shows with that build, it might still not be their fault (unless they do some sort of patching there).