Pytest mocks spill into python library cache

[IvanovCosmin]

Summary:
Old unit test started failing on the CI with cryptic errors 100% of the time, but it was extremely hard to reproduce locally.

The following code is the minimal example to make the bug happen.

from pathlib import Path
import pytest
import pytest_mock


@pytest.mark.asyncio
async def test_repro(
    mocker: pytest_mock.MockerFixture
):
    mocker.patch.object(Path, "exists", return_value=True, autospec=True)
    mocker.patch.object(Path, "open", mocker.mock_open(read_data=b"helloworld"))
    import anyio._backends

To run the code and reproduce the problem you must use the following command
find <path_to_dependecies_folder> -name '*.pyc' -delete && pytest tests/test_example.py
In my case it is find ~/miniconda3/envs/unitary/ -name '*.pyc' -delete && pytest tests/test_example.py

It is important to delete the cache because it is going to fail only once

What happens is that rewrite.py will somehow use the mocks instead of the real pathlib.Path to write and read the relevant .pyc files. I don't fully understand why rewrite.py writes dependency caches.

The error is something related to:

platform linux -- Python 3.10.9, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/ubuntu/unitarybot
plugins: mock-3.10.0, anyio-3.6.2, asyncio-0.21.0
asyncio: mode=strict
collected 1 item                                                                                                                        

tests/test_example.py F                                                                                                           [100%]

=============================================================== FAILURES ================================================================
______________________________________________________________ test_repro _______________________________________________________________

mocker = <pytest_mock.plugin.MockerFixture object at 0x7fc3847f2680>

    @pytest.mark.asyncio
    async def test_repro(
        mocker: pytest_mock.MockerFixture
    ):
    
        # file upload using file
        mocker.patch.object(Path, "exists", return_value=True, autospec=True)
        mocker.patch.object(Path, "open", mocker.mock_open(read_data=b"helloworld"))
>       import anyio._backends

tests/test_example.py:14: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:1006: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:688: in _load_unlocked
    ???
../miniconda3/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:172: in exec_module
    exec(co, module.__dict__)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   NameError: name 'helloworld' is not defined

../miniconda3/lib/python3.10/site-packages/anyio/_backends/__init__.py:1: NameError
======================================================== short test summary info ========================================================
FAILED tests/test_example.py::test_repro - NameError: name 'helloworld' is not defined
=========================================================== 1 failed in 0.05s =========

This probably makes sense right now, but you must take into account that the original error happened with another third party dependency and it was something similar to this

from pathlib import Path
import pytest
import pytest_mock

import httpx

@pytest.mark.asyncio
async def test_repro(
    mocker: pytest_mock.MockerFixture
):
    
    # file upload using file
    mocker.patch.object(Path, "exists", return_value=True, autospec=True)
    mocker.patch.object(Path, "open", mocker.mock_open(read_data=b"helloworld"))
    
   #heavely simplified version of the test
    async with httpx.AsyncClient() as client:
        pass

I am not using pip, but conda:
Conda list output:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
anyio                     3.6.2                    pypi_0    pypi
brotlipy                  0.7.0           py310h7f8727e_1002  
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2023.01.10           h06a4308_0  
certifi                   2022.12.7       py310h06a4308_0  
cffi                      1.15.1          py310h5eee18b_3  
charset-normalizer        2.0.4              pyhd3eb1b0_0  
conda                     23.1.0          py310h06a4308_0  
conda-content-trust       0.1.3           py310h06a4308_0  
conda-package-handling    2.0.2           py310h06a4308_0  
conda-package-streaming   0.7.0           py310h06a4308_0  
cryptography              38.0.4          py310h9ce1e76_0  
exceptiongroup            1.1.1                    pypi_0    pypi
idna                      3.4             py310h06a4308_0  
iniconfig                 2.0.0                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.4.2                h6a678d5_6  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libstdcxx-ng              11.2.0               h1234567_1  
libuuid                   1.41.5               h5eee18b_0  
ncurses                   6.4                  h6a678d5_0  
openssl                   1.1.1s               h7f8727e_0  
packaging                 23.1                     pypi_0    pypi
pip                       22.3.1          py310h06a4308_0  
pluggy                    1.0.0           py310h06a4308_1  
pycosat                   0.6.4           py310h5eee18b_0  
pycparser                 2.21               pyhd3eb1b0_0  
pyopenssl                 22.0.0             pyhd3eb1b0_0  
pysocks                   1.7.1           py310h06a4308_0  
pytest                    7.3.1                    pypi_0    pypi
pytest-asyncio            0.21.0                   pypi_0    pypi
pytest-mock               3.10.0                   pypi_0    pypi
python                    3.10.9               h7a1cb2a_0  
readline                  8.2                  h5eee18b_0  
requests                  2.28.1          py310h06a4308_0  
ruamel.yaml               0.17.21         py310h5eee18b_0  
ruamel.yaml.clib          0.2.6           py310h5eee18b_1  
setuptools                65.6.3          py310h06a4308_0  
six                       1.16.0             pyhd3eb1b0_1  
sniffio                   1.3.0                    pypi_0    pypi
sqlite                    3.40.1               h5082296_0  
tk                        8.6.12               h1ccaba5_0  
tomli                     2.0.1                    pypi_0    pypi
toolz                     0.12.0          py310h06a4308_0  
tqdm                      4.64.1          py310h06a4308_0  
tzdata                    2022g                h04d1e81_0  
urllib3                   1.26.14         py310h06a4308_0  
wheel                     0.37.1             pyhd3eb1b0_0  
xz                        5.2.10               h5eee18b_1  
zlib                      1.2.13               h5eee18b_0  
zstandard                 0.18.0          py310h5eee18b_0  

This also spills into the pyc file, unsure if this is what is supposed to happen.

Let me know what other info I can get for you!

[RonnyPfannschmidt]

Breaking parts of the stdlib breaks parts of pytest

Monkeypatching pathlib itself changes it for everything, not just the code under test

It's recommend to replace the imported instances of stdlib bits if necessary

[nicoddemus]

Yeah unfortunately pytest depends on the stdlib, and as @RonnyPfannschmidt said this might break pytest itself.

Not sure if there is anything actionable that can be done here.

[RonnyPfannschmidt]

we could add a warning + document recommended/possible patterns

a posible pattern is instead of patching the stdlib, replacing the imported references with patched objects
(with al lthe minefield style fragility that comes of it

the other pattern would be to add testability instrumentation into libraries, and then one can pass the dependencies instead of breaking

[nicoddemus]

we could add a warning + document recommended/possible patterns

That's an idea. I suggest we close this issue then, and create a new one more descriptive of the action.

[IvanovCosmin]

The weird part is this wasn't an issue for close to 1 year. We never had any issues in tests with mocked Paths. What changed?

[RonnyPfannschmidt]

more parts of pytest got ported to pathlib

[nicoddemus]

Closing this in favor of #11022