pytest plugin

[njsmith]

Extract what we've got into a proper standalone plugin

(Or alternatively make the regular trio package include an entry point? But probably it is better to decouple them a bit, in case we ever have to make an emergency release due to pytest breaking something -> don't want to force people to upgrade trio itself for that.)

[njsmith]

Another nice thing would be if this plugin could somehow teach pytest about MultiErrors. (It's not unusable right now, because pytest AssertionErrors do contain the crucial information in their repr, and MultiError shows the repr of all the embedded exceptions. But it would be better if the traceback pytest machinery could play along properly.)

I'm not sure if this is possible with pytest as it currently stands, though. Maybe something involving a custom collector with a custom repr_failure method?

[njsmith]

(Once ths is done then our tests could use some cleanup to take advantage... they're a bit gross right now. There's some duplication between our different conftest.py's, and they're packaged with the source, but can't actually be run by pytest --pyargs trio, ...)

[njsmith]

Another idea: async fixtures, for when you need to await or otherwise access the trio context during fixture setup/teardown

this mighhhht be really hard to implement, not sure

[njsmith]

Possibly useful link: https://github.com/pytest-dev/cookiecutter-pytest-plugin

[njsmith]

pytest-asyncio has async fixture support: https://github.com/pytest-dev/pytest-asyncio/blob/master/pytest_asyncio/plugin.py

I guess the general trick would be to stash these fixtures on the side somewhere, and then unpack them when entering the test itself.

Maybe convert an async fixture into a sync fixture that yields an object of a special type that the trio runner recognizes and can call some methods on.

The problem would be nested fixtures. I guess this might be solved by having the fixture objects memoize themselves internally and use the same argument pre-processing code as the tests themselves?

[njsmith]

Here's the pytest docs with all the wacky things fixtures can do: https://docs.pytest.org/en/latest/fixture.html

Module and session scoped async fixtures don't make any sense, since we tear down the run loop on every test. Maybe the simplest thing is to skip the sneaky hook methods and define our own pytest_trio.async_fixture decorator.

Not sure what to do about addfinalizer, maybe it just doesn't support async finalizers and that's ok.

Everything else looks like it should work ok with the approach above.

Probably this whole discussion of fixtures should be its own issue, because this certainly isn't a prerequisite to having a useful pytest-trio. But pytest-trio doesn't have an issue tracker yet :-)

[Tinche]

Not sure what exactly you have in mind for this, but I could just port pytest-asyncio to trio for you. That would probably be easy and it'd get you something, at least.

[njsmith]

@Tinche: Oh, hello! Thanks for the offer! If you want to work on this then it'd be much appreciated for sure. Not having a good pytest plugin is pretty embarrassing for a project that prides itself on testing :-).

I'm not sure how much of the details of pytest-asyncio will carry over, because trio doesn't have the concept of an explicit loop object. OTOH trio does consider testing to be a first-class consideration, so it provides a @trio.testing.trio_test decorator out-of-the-box that's suitable to use with any testing framework (currently a bit rudimentary but it'd be nice to improve it further).

I was imagining that a v0.0.1 version of pytest-trio would basically just be this code pulled out into a standalone plugin, plus some configuration options added. (Probably: require that either a pytest.ini config variable be set to enable it globally, or else a pytest.mark.trio be applied, the latter intended for projects that might have some tests that use trio, some that use asyncio, etc.)

That would already be enough to let trio start dogfooding the package ourselves, and also be super helpful to others...

Then fancier things would be fixture support -- this is a bit tricky without explicit loop support, because you have to delay the actual fixture initialization until you start executing the test, but I think it's doable -- timeouts, helpers to detect things like ResourceWarning and "coroutine was never awaited", and so forth...... but we can figure that out later :-). What do you think?