BUG: dangling pytest ioc processes if test suite segfaults

[tangkong]

Current Behavior

Currently test IOCs in the test suite are spun up in a separate Multiprocessing.Process. If the context and test suite exits normally, this process is cleaned up just fine. If something in the test suite breaks, this process can get left behind. This appears as a pytest process, and you can also caget the test PVs in question even after regaining terminal control.

The most immediate symptom of this is that the ioc test will never pass while one of these dangling processes exists, presumably due to PV name collisions.

To fix this you have to manually kill the process

Expected Behavior

We should find some way to either ensure these are always cleaned up, clean them up faster, or document this carefully so we make sure to avoid it in the future

Context / environment

Discovered while working on some gui tests, where I spawn threads and need to wait for them to get cleaned up.

Steps to Reproduce (for bugs)

1. Write some bad test code (make a thread that tries to access a resource, destroy the resource, or something similar)
2. Run python -m pytest and watch it die

Details

================================================================================ test session starts ================================================================================
platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
PyQt5 5.15.9 -- Qt runtime 5.15.8 -- Qt compiled 5.15.8
rootdir: /cds/home/r/roberttk/devrepos/superscore
configfile: pyproject.toml
plugins: xdist-3.5.0, jaxtyping-0.2.21, anyio-3.7.1, benchmark-4.0.0, timeout-2.1.0, asyncio-0.23.5, typeguard-2.13.3, cov-4.1.0, repeat-0.9.2, qt-4.4.0
asyncio: mode=auto
collected 51 items

superscore/tests/test_backend.py ........                                                                                                                                     [ 15%]
superscore/tests/test_cl.py .....                                                                                                                                             [ 25%]
superscore/tests/test_client.py .......                                                                                                                                       [ 39%]
superscore/tests/test_ioc.py F                                                                                                                                                [ 41%]
superscore/tests/test_model.py .........                                                                                                                                      [ 58%]
superscore/tests/test_page.py ...Fatal Python error: Aborted

Thread 0x00007ff968c4d700 (most recent call first):
  File "/cds/home/r/roberttk/devrepos/superscore/superscore/widgets/views.py", line 874 in run

Current thread 0x00007ff98a46e740 (most recent call first):
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/fixtures.py", line 1043 in finish
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/runner.py", line 544 in teardown_exact
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/runner.py", line 187 in pytest_runtest_teardown
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/runner.py", line 240 in <lambda>
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/runner.py", line 340 in from_call
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/runner.py", line 239 in call_and_report
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/runner.py", line 135 in runtestprotocol
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/runner.py", line 115 in pytest_runtest_protocol
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/main.py", line 364 in pytest_runtestloop
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/main.py", line 339 in _main
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/main.py", line 285 in wrap_session
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/pluggy/_callers.py", line 102 in _multicall
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/pluggy/_manager.py", line 119 in _hookexec
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/pluggy/_hooks.py", line 501 in __call__
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/config/__init__.py", line 174 in main
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/_pytest/config/__init__.py", line 197 in console_main
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/site-packages/pytest/__main__.py", line 7 in <module>
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/runpy.py", line 87 in _run_code
  File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.8.4/lib/python3.9/runpy.py", line 197 in _run_module_as_main
Aborted

3. `ps aux | grep pytest`

Details

roberttk 29576 10.0  0.1 528104 77548 ?        S    12:36   0:03 python -m pytest

4. caget shows test PVs, subsequent tests fail

Details

(pcds-5.9.0+sscore)roberttk@psbuild-rhel7-01:~/devrepos/superscore(enh_coll_builder +)$ python -m pytest
================================================================================ test session starts ================================================================================
platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
PyQt5 5.15.9 -- Qt runtime 5.15.8 -- Qt compiled 5.15.8
rootdir: /cds/home/r/roberttk/devrepos/superscore
configfile: pyproject.toml
plugins: xdist-3.5.0, jaxtyping-0.2.21, anyio-3.7.1, benchmark-4.0.0, timeout-2.1.0, asyncio-0.23.5, typeguard-2.13.3, cov-4.1.0, repeat-0.9.2, qt-4.4.0
asyncio: mode=auto
collected 52 items

superscore/tests/test_backend.py ........                                                                                                                                     [ 15%]
superscore/tests/test_cl.py .....                                                                                                                                             [ 25%]
superscore/tests/test_client.py .......                                                                                                                                       [ 38%]
superscore/tests/test_ioc.py CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "SCORETEST:MGNT:GUNB:TEST0", Connecting to: psbuild-rhel7-01.slac.stanford.edu:5064, Ignored: psbuild-rhel7-01.slac.stanford.edu:52856"
    Source File: modules/ca/src/client/cac.cpp line 1320
    Current Time: Fri Sep 13 2024 13:12:17.240715481
..................................................................
F

(pcds-5.9.0+sscore)roberttk@psbuild-rhel7-01:~/devrepos/superscore(enh_coll_builder +)$ caget SCORETEST:MGNT:GUNB:TEST0
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "SCORETEST:MGNT:GUNB:TEST0", Connecting to: 134.79.165.97:52856, Ignored: psbuild-rhel7-01.slac.stanford.edu:5064"
    Source File: ../cac.cpp line 1320
    Current Time: Fri Sep 13 2024 13:12:40.486027569
..................................................................
SCORETEST:MGNT:GUNB:TEST0      On