Pytest: stop on any failure in tests

[k4nar]

When I’m debugging a test with VSCode, I’m expecting the debugger to automatically add a breakpoint at the line where the test is failing.

This is the behavior of pytest with the option --pdb, so I think it’s reasonable to expect the same when using it with VSCode.

In the FAQ the recommended solution is to use "User Uncaught Exceptions". However this is not practical with any "real world" codebase: breakpoints are added in a lot of exceptions handled by third parties or Python itself (see #1248, #1102, #1155, …). In my case I would have to skip dozens of breakpoints each time, and those exceptions are legitimate.

Note: this feature has been requested before (#722, #723, #1103), but the suggested solutions did not work for me.

[k4nar]

I’ve tried to re-use what neotest-python is doing, which was done by @wookayin based on this recommendation: #723 (comment) .

Here is the code I’ve added in my conftest.py:

def pytest_exception_interact(
    node: "pytest.Item | pytest.Collector",
    call: "pytest.CallInfo",
    report: "pytest.CollectReport | pytest.TestReport",
):
    # Reference: https://github.com/microsoft/debugpy/issues/723
    import threading

    try:
        import pydevd
    except ImportError:
        return  # debugpy or pydevd not available, do nothing

    py_db = pydevd.get_global_debugger()
    if py_db is None:
        return

    excinfo = call.excinfo._excinfo

    thread = threading.current_thread()
    additional_info = py_db.set_additional_thread_info(thread)
    additional_info.is_tracing += 1
    try:
        py_db.stop_on_unhandled_exception(py_db, thread, additional_info, excinfo)
    finally:
        additional_info.is_tracing -= 1

The debugger stops if I have an error in my test, but at the line with py_db.stop_on_unhandled_exception, not at the actual line in my test!

I’ve tried to change the frames of the exception to "trick" pydevd, but I’m not familiar enough with the project to make it work.

[k4nar]

Another popular recommendation has been made in StackOverflow (link), but the solution does not work anymore either.

If I add this in my conftest.py:

def is_debugging():
    return "debugpy" in sys.modules

if is_debugging():

    @pytest.hookimpl(tryfirst=True)
    def pytest_exception_interact(call):
        raise call.excinfo.value

    @pytest.hookimpl(tryfirst=True)
    def pytest_internalerror(excinfo):
        raise excinfo.value

Then the tests fail with error, and then the whole testsuite is launched unexpectedly:

[...]
tests/test_api.py:99 test_api_get - assert Fa…
Results (1.53s):
         1 failed
Error[vscode-pytest]: unable to read testIds from temp fileassert False is True
============================= test session starts ==============================
[...]

[rchiodo]

I assume pytest failures throw a specific error. I think one way this could potentially be handled, would be to allow break on specific exceptions. Like Visual Studio does for debugging .NET code. You would say specify the exception types to break on in your launch.json

You might also be able to implement breaking during pytest exceptions by writing a pydevd plugin. Pydevd plugins have a exception_break handler and a has_exception_breaks function that should allow you to determine when to break on an exception.

[k4nar]

I think that would work for AssertionError yes. I’ve seen devs doing this with PyCharm for instance (but it’s not possible in VSCode at the moment AFAICT).

But actually what pytest does is that it catches any unhandled exception, and deal with them to make only this particular test fail (not the whole testsuite) and provide a meaningful output.

But what it allows to do is to write hooks such as pytest_exception_interact to allow custom code to interact with the exception. From there we have access to the Traceback and the frames, but I don’t know how to tell debugpy (or pydevd) to break on a given frame.

[rchiodo]

But what it allows to do is to write hooks such as pytest_exception_interact to allow custom code to interact with the exception. From there we have access to the Traceback and the frames, but I don’t know how to tell debugpy (or pydevd) to break on a given frame.

Putting breakpoint() in that code would likely cause the debugger to stop

[k4nar]

Yes, but it stops in the hook, not where the error was.

[rchiodo]

I don't believe there's a way to get debugpy to break on the specific exception without modifying debugpy's code. Short of modifying debugpy, I think you could write a pydevd plugin, or it sounds like you can break in your handler and then look at the stack in the exception info.

The modifications to debugpy would likely require special logic in the _unwind_event function to break on specific exceptions.