Test failures in test_assert_called_args_with_introspection

[ghisvail]

Whilst attempting to integrate pytest-mock to the Debian CI infrastructure, I encountered the following issue when running the tests against the installed version from the Debian package:

=================================== FAILURES ===================================
__________________ test_assert_called_args_with_introspection __________________

mocker = <pytest_mock.MockFixture object at 0x7f46460fce10>

    def test_assert_called_args_with_introspection(mocker):
        stub = mocker.stub()
    
        complex_args = ('a', 1, set(['test']))
        wrong_args = ('b', 2, set(['jest']))
    
        stub(*complex_args)
        stub.assert_called_with(*complex_args)
        stub.assert_called_once_with(*complex_args)
    
        with assert_argument_introspection(complex_args, wrong_args):
            stub.assert_called_with(*wrong_args)
>           stub.assert_called_once_with(*wrong_args)

test_pytest_mock.py:390: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python2.7/contextlib.py:35: in __exit__
    self.gen.throw(type, value, traceback)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

left = ('a', 1, set(['test'])), right = ('b', 2, set(['jest']))

    @contextmanager
    def assert_argument_introspection(left, right):
        """
        Assert detailed argument introspection is used
        """
        try:
            yield
        except AssertionError as e:
            # this may be a bit too assuming, but seems nicer then hard-coding
            import _pytest.assertion.util as util
            # NOTE: we assert with either verbose or not, depending on how our own
            #       test was run by examining sys.argv
            verbose = any(a.startswith('-v') for a in sys.argv)
            expected = '\n  '.join(util._compare_eq_iterable(left, right, verbose))
>           assert expected in str(e)
E           assert "Full diff:\n  - ('a', 1, set(['test']))\n  ?   ^   ^        ^\n  + ('b', 2, set(['jest']))\n  ?   ^   ^        ^" in "Expected call: mock('b', 2, set(['jest']))\nActual call: mock('a', 1, set(['test']))\n\npytest introspection follows:\n\nArgs:\n"
E            +  where "Expected call: mock('b', 2, set(['jest']))\nActual call: mock('a', 1, set(['test']))\n\npytest introspection follows:\n\nArgs:\n" = str(AssertionError(u"Expected call: mock('b', 2, set(['jest']))\nActual call: mock('a', 1, set(['test']))\n\npytest introspection follows:\n\nArgs:\n",))

test_pytest_mock.py:346: AssertionError
_________________ test_assert_called_kwargs_with_introspection _________________

mocker = <pytest_mock.MockFixture object at 0x7f464627c150>

    def test_assert_called_kwargs_with_introspection(mocker):
        stub = mocker.stub()
    
        complex_kwargs = dict(foo={'bar': 1, 'baz': 'spam'})
        wrong_kwargs = dict(foo={'goo': 1, 'baz': 'bran'})
    
        stub(**complex_kwargs)
        stub.assert_called_with(**complex_kwargs)
        stub.assert_called_once_with(**complex_kwargs)
    
        with assert_argument_introspection(complex_kwargs, wrong_kwargs):
            stub.assert_called_with(**wrong_kwargs)
>           stub.assert_called_once_with(**wrong_kwargs)

test_pytest_mock.py:405: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python2.7/contextlib.py:35: in __exit__
    self.gen.throw(type, value, traceback)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

left = {'foo': {'bar': 1, 'baz': 'spam'}}
right = {'foo': {'baz': 'bran', 'goo': 1}}

    @contextmanager
    def assert_argument_introspection(left, right):
        """
        Assert detailed argument introspection is used
        """
        try:
            yield
        except AssertionError as e:
            # this may be a bit too assuming, but seems nicer then hard-coding
            import _pytest.assertion.util as util
            # NOTE: we assert with either verbose or not, depending on how our own
            #       test was run by examining sys.argv
            verbose = any(a.startswith('-v') for a in sys.argv)
            expected = '\n  '.join(util._compare_eq_iterable(left, right, verbose))
>           assert expected in str(e)
E           assert "Full diff:\n  - {'foo': {'bar': 1, 'baz': 'spam'}}\n  + {'foo': {'baz': 'bran', 'goo': 1}}" in "Expected call: mock(foo={'goo': 1, 'baz': 'bran'})\nActual call: mock(foo={'baz': 'spam', 'bar': 1})\n\npytest introspection follows:\n\nKwargs:\n"
E            +  where "Expected call: mock(foo={'goo': 1, 'baz': 'bran'})\nActual call: mock(foo={'baz': 'spam', 'bar': 1})\n\npytest introspection follows:\n\nKwargs:\n" = str(AssertionError(u"Expected call: mock(foo={'goo': 1, 'baz': 'bran'})\nActual ca...foo={'baz': 'spam', 'bar': 1})\n\npytest introspection follows:\n\nKwargs:\n",))

test_pytest_mock.py:346: AssertionError
_________________________ test_detailed_introspection __________________________

testdir = <Testdir local('/tmp/pytest-of-root/pytest-0/testdir/test_detailed_introspection0')>

    def test_detailed_introspection(testdir):
        """Check that the "mock_use_standalone" is being used.
        """
        testdir.makepyfile("""
            def test(mocker):
                m = mocker.Mock()
                m('fo')
                m.assert_called_once_with('', bar=4)
        """)
        result = testdir.runpytest('-s')
        result.stdout.fnmatch_lines([
            "*AssertionError: Expected call: mock('', bar=4)*",
            "*Actual call: mock('fo')*",
            "*pytest introspection follows:*",
            '*Args:',
            "*assert ('fo',) == ('',)",
            "*At index 0 diff: 'fo' != ''*",
            "*Use -v to get the full diff*",
            "*Kwargs:*",
            "*assert {} == {'bar': 4}*",
            "*Right contains more items:*",
            "*{'bar': 4}*",
>           "*Use -v to get the full diff*",
        ])
E       Failed: nomatch: "*AssertionError: Expected call: mock('', bar=4)*"
E           and: u'============================= test session starts =============================='
E           and: u'platform linux2 -- Python 2.7.14+, pytest-3.2.5, py-1.4.34, pluggy-0.4.0'
E           and: u'rootdir: /tmp/pytest-of-root/pytest-0/testdir/test_detailed_introspection0, inifile:'
E           and: u'plugins: mock-1.6.3'
E           and: u'collected 1 item'
E           and: u''
E           and: u'test_detailed_introspection.py F'
E           and: u''
E           and: u'=================================== FAILURES ==================================='
E           and: u'_____________________________________ test _____________________________________'
E           and: u''
E           and: u'mocker = <pytest_mock.MockFixture object at 0x7f4646277c90>'
E           and: u''
E           and: u'    def test(mocker):'
E           and: u'        m = mocker.Mock()'
E           and: u"        m('fo')"
E           and: u">       m.assert_called_once_with('', bar=4)"
E       fnmatch: "*AssertionError: Expected call: mock('', bar=4)*"
E          with: u"E       AssertionError: Expected call: mock('', bar=4)"
E       fnmatch: "*Actual call: mock('fo')*"
E          with: u"E       Actual call: mock('fo')"
E       nomatch: '*pytest introspection follows:*'
E           and: u'E       '
E       fnmatch: '*pytest introspection follows:*'
E          with: u'E       pytest introspection follows:'
E       nomatch: '*Args:'
E           and: u'E       '
E       fnmatch: '*Args:'
E          with: u'E       Args:'
E       nomatch: "*assert ('fo',) == ('',)"
E           and: u'E       '
E           and: u'E       Kwargs:'
E           and: u''
E           and: u'test_detailed_introspection.py:4: AssertionError'
E           and: u'=========================== 1 failed in 0.01 seconds ==========================='
E           and: u''
E       remains unmatched: "*assert ('fo',) == ('',)"

/tmp/autopkgtest.31sHEi/autopkgtest_tmp/test_pytest_mock.py:535: Failed
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux2 -- Python 2.7.14+, pytest-3.2.5, py-1.4.34, pluggy-0.4.0
rootdir: /tmp/pytest-of-root/pytest-0/testdir/test_detailed_introspection0, inifile:
plugins: mock-1.6.3
collected 1 item

test_detailed_introspection.py F

=================================== FAILURES ===================================
_____________________________________ test _____________________________________

mocker = <pytest_mock.MockFixture object at 0x7f4646277c90>

    def test(mocker):
        m = mocker.Mock()
        m('fo')
>       m.assert_called_once_with('', bar=4)
E       AssertionError: Expected call: mock('', bar=4)
E       Actual call: mock('fo')
E       
E       pytest introspection follows:
E       
E       Args:
E       
E       Kwargs:

test_detailed_introspection.py:4: AssertionError
=========================== 1 failed in 0.01 seconds ===========================
========== 3 failed, 38 passed, 1 skipped, 2 xfailed in 0.69 seconds ===========

The plugin seems to be correctly registered:

============================= test session starts ==============================
platform linux2 -- Python 2.7.14+, pytest-3.2.5, py-1.4.34, pluggy-0.4.0 -- /usr/bin/python2.7
cachedir: .cache
rootdir: /tmp/autopkgtest.31sHEi/autopkgtest_tmp, inifile:
plugins: mock-1.6.3
collecting ... collected 44 items

The same thing happens for both Python 2 and Python 3.

[nicoddemus]

Hi @ghisvail,

Those tests expect pytest's assertion rewrite to be enabled to work, and indeed I can reproduce the problem if I execute the tests with --assert=plain.

Pytest assertion rewrite is enabled by default, can you check your command-line or PYTEST_ environment variables for that setting?

Either way I think the proper solution is to skip those tests when executed with assert=plain.

[nicoddemus]

The tests which require assertion rewrite are now skipped with --assert=plain.