PyTest is running app code outside of designated test functions

[gh-andre]

Normally PyTest scans the code for functions like test_* and *_test and classes Test* and calls only functions named to match test patterns. What I'm finding is that PyTest will actually run application code if there is a dictionary field access error, which caught me by surprise in that before I figured out what's going on, it tied up the CI pipeline trying to run the code it's not supposed to touch.

Here's a small test case that demonstrates this behavior. It will also happen if this code is just in the global scope and without asyncio, but this is closer to how my actual code uses it.

import asyncio

async def some_fn() -> None:
    x = {"x": 123}
    print(x["y"])


async def main() -> None:
    print("*** Running app code!!! ***")

    await some_fn()

asyncio.run(main())

Running with this:

pytest abc.py

, generates this output, which shows that the non-test code is running:

=========================================== test session starts ============================================
platform linux -- Python 3.10.8, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/user/tmp
collected 0 items / 1 error

================================================== ERRORS ==================================================
________________________________________ ERROR collecting async.py _________________________________________
async.py:13: in <module>
    asyncio.run(main())
../miniconda3/lib/python3.10/asyncio/runners.py:44: in run
    return loop.run_until_complete(main)
../miniconda3/lib/python3.10/asyncio/base_events.py:649: in run_until_complete
    return future.result()
async.py:11: in main
    await some_fn()
async.py:5: in some_fn
    print(x["y"])
E   KeyError: 'y'
--------------------------------------------- Captured stdout ----------------------------------------------
*** Running app code!!! ***
========================================= short test summary info ==========================================
ERROR async.py - KeyError: 'y'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================= 1 error in 0.08s =============================================

So, the question is - is it the field access error that triggers this behavior or PyTest may routinely run application code outside of any test functions?

Thanks

[RonnyPfannschmidt]

You run main on import, pytest must import to find functions

Protect your call to main by checking if __name__ == "__main__" :

[gh-andre]

This doesn't happen if there is not exception, though. If it was indeed just running main on import, wouldn't it do it regardless of whether the code raises an exception or not?

[RonnyPfannschmidt]

If there is no exception, it just runs without error and pytest does the search

[gh-andre]

If it ran indeed without the exception, I would see the printed output and I do not. Would I not? I also can observe the same behavior in the actual app that takes longer to execute if it runs app code and PyTest just scans it and exits without running any app code.

[RonnyPfannschmidt]

Pytest captures. Output run pytest -s to see the full output

[gh-andre]

I see. I never used introspection and incorrectly thought that PyTest could find test functions/classes and would invoke them without running any app code. Thanks for explaining.

[The-Compiler]

Note it does not explicitly run anything - it just imports the module, and your code is written in a way that it does things when the module is imported (which is considered bad practice).

You will see the same issue if you run python3 -c "import async" - as a rule of thumb, importing a module should not run anything significant.

[gh-andre]

@The-Compiler

I do realize how import works and my misunderstanding was based on the incorrect thinking that finding test functions wouldn't result in importing modules and running global code in those module. What I learned is that introspection uses import, which means that tools using introspection will run global code, even though they don't invoke any of app functions explicitly.