Why does pytest not execute the entire loop

[tejendrapsingh]

[graingert]

pytest with xdist doesn't print anything out when using -s pytest-dev/pytest-xdist#354

Probably this issue is in where you catch all exceptions and suppress them https://github.com/tejendrapsingh/for-help/blob/main/using-pytest-for-parallelization.py#L900

Instead you should only catch the exceptions that you expect

If you want to catch multiple exceptions and continue a for loop you should use an exception group:

exceptions = []
for x in y:
    try:
        ...
    except Exception as e:
        e.add_note("failed processing {x}")
        exceptions.append(e)
if exceptions:
    raise ExceptionGroup("multiple errors processing", exceptions)

These bare except statements are also bad, and incompatible with a signal handler and BaseException timeout approach: https://github.com/tejendrapsingh/for-help/blob/main/using-pytest-for-parallelization.py#L92

[graingert]

Also this is not the correct way to add metadata to an exception:

except Exception as e:
        print("*** print_exception:")
        pytest.fail(f"Test failed for {data} {year_val} with error: {str(e)}")

You should instead do:

except Exception as e:
    e.add_note(f"Test failed for {data} {year_val}")
    raise

[tejendrapsingh]

@graingert Thanks for these suggestions. I will edit the code with what you suggested and see if it makes a difference. Thank you very much for taking the time to go through the code. I am not a native programmer. So, always trying to find the easiest way to make things work. Pardon my ``dirty'' code writing.

[The-Compiler]

I'd suggest you don't use pytest for this at all, as you don't actually seem to be testing anything. If you use Selenium directly from Python, and then use something like the concurrent.futures module to parallelize that, you'll have much more control over what's happening.

I also agree that your approach to exception handling likely hides bugs in your code - see e.g. The Most Diabolical Python Antipattern – Real Python.