AttributeError: 'flake8' not a registered marker with --strict --flake8

[alex-dr]

I don't know enough about the codebase to understand this, but 3.1 introduces a regression that prevents --strict and --flake8 flags from being passed simultaneously. This broke many builds across a few repositories until we restricted our pytest dependencies to <3.1.

Here is the error:

~/workspace/flake-fail$ py.test --strict --flake8
======================================== test session starts ========================================
platform linux2 -- Python 2.7.13, pytest-3.1.1, py-1.4.33, pluggy-0.4.0
rootdir: /home/alex/workspace/flake-fail, inifile: setup.cfg
plugins: flake8-0.8.1
collected 0 items / 1 errors 

============================================== ERRORS ===============================================
_________________________________________ ERROR collecting  _________________________________________
../../.virtualenvs/flail/local/lib/python2.7/site-packages/pytest_flake8.py:71: in pytest_collect_file
    statistics=config._flake8statistics)
../../.virtualenvs/flail/local/lib/python2.7/site-packages/pytest_flake8.py:89: in __init__
    self.add_marker("flake8")
../../.virtualenvs/flail/local/lib/python2.7/site-packages/_pytest/main.py:383: in add_marker
    marker = getattr(MARK_GEN, marker)
../../.virtualenvs/flail/local/lib/python2.7/site-packages/_pytest/mark.py:251: in __getattr__
    self._check(name)
../../.virtualenvs/flail/local/lib/python2.7/site-packages/_pytest/mark.py:266: in _check
    raise AttributeError("%r not a registered marker" % (name,))
E   AttributeError: 'flake8' not a registered marker
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================== 1 error in 0.13 seconds ======================================

Minimal reproduction example:

# File: setup.py
from setuptools import setup


install_requirements = [
    'pytest-flake8',
    'pytest>=3.1',
    'six',
]

setup(name='flail',
      version='0.0.0',
      install_requires=install_requirements,
      )

mkvirtualenv flail
pip install .
py.test  # okay
py.test --strict  # okay
py.test --flake8  # okay
py.test --strict --flake8  # fail
pip install 'pytest<3.1'
py.test --strict --flake8  # okay

pip list:
configparser (3.5.0)
enum34 (1.1.6)
flail (0.0.0)
flake8 (3.3.0)
mccabe (0.6.1)
pip (9.0.1)
pkg-resources (0.0.0)
py (1.4.33)
pycodestyle (2.3.1)
pyflakes (1.5.0)
pytest (3.1.1)
pytest-flake8 (0.8.1)
setuptools (36.0.0)
six (1.10.0)
wheel (0.29.0)

Operating System: Ubuntu 17.04 (also observed on earlier versions)
Bug present in: 3.1.0, 3.1.1

• Include a detailed description of the bug or suggestion
• pip list of the virtual environment you are using
• pytest and operating system versions
• Minimal example if possible

[alex-dr]

Figured out some workarounds:

1: (Preferred) Change --strict for -W error. Same functionality, no problems with any plugins.
2: Add markers explicitly in your setup.cfg/setup.ini:

[tools:pytest]
markers = 
    flake8
    mccabe
    ...etc

[alex-dr]

Actually, -W error seems a bit too aggressive for my tastes. Every Python runtime warning causes the test to abort. In some cases this seems to affect libraries outside of our control, so that's not going to work out.

[The-Compiler]

I did a quick bisect, and this appeared with 10094a3 (cc @RonnyPfannschmidt).

However, it seems like the behavior is now more correct - the pytest-flake8 plugin should register its marker properly, so I think you should open an issue there instead. Unfortunately, it's pretty dead it seems...

Also, note that --strict and -W error are totally different. Currently, the only thing --strict does is validating that markers are valid, which can be very useful to catch typos (which would be silent otherwise) in marks. I'd highly recommend turning --strict on. Its description is unfortunately misleading, see #2444.

As for -W error, what I personally do is adding filters for messages outside of my control and then opening a bug for affected libraries. If nobody does that, the libraries will only figure out about their stuff being broken when things actually get removed 😉

Finally, I'm not sure I'd still recommend doing linting via pytest, even though I'm the author of pytest-mccabe. I got happier with just running flake8 (and pytest) separately via tox. Less moving parts, less stuff you need to wait for when flake8 updates, and more functionality.

[RonnyPfannschmidt]

i reported an issue on the plugin and will close this issue as there is literally nothing pytest can do about it in terms of code

[alex-dr]

Thanks for the quick, detailed response!

I'll note that this also seems to affect latest pytest-mccabe and pytest-pep8 in the same manner, and quite likely some others. Shall I open issues for those repos as well?

[The-Compiler]

Please do! Or if you have the time, pull requests 😉 What needs to be done is something like this:

def pytest_configure(config):
    config.addinivalue_line('markers', "flake8: Tests which run flake8.")

(or something similar in an existing pytest_configure hook, if any)

You might also want to add a test running the plugin with --strict. Usually there should be existing tests using the pytester fixture which you can adopt.

[alex-dr]

Cool. Pushed some PR's against repos I use that are affected.