Put together a Python 3.7 wishlist

[njsmith]

[this was originally #79, but that thread got taken over by the discussion of whether we can provide better diagnostics on missing await, so I'm moving the umbrella issue here]

This is currently very rough and needs expanding into something we can actually bring to the Python core team, but so I don't forget:

• __iterclose__ (PEP 533) – probably the single most important from a user perspective. Unfortunately appears to be stalled; definitely won't happen for 3.7.

• can we do anything about how easy it is to forget an await? In retrospect async functions shouldn't implement __call__ IMO... but probably too late to fix that. Still, it kinda sucks that one of first things in our tutorial is a giant warning about this wart, so worth at least checking if anyone has any ideas... [see Can we make forgetting an await be an error? #79 for discussion]

• possibly Result should actually be builtin? I think it would actually really simplify CPython's generator API and implementation. (in particular, unifying send and throw could dramatically simplify the implementation of yield from while fixing some of the weird intractable throw bugs that trio currently has to work around)

• speaking of which, straightforward bugs that affect us:

  • issue29600 (this is the worst; it actually blocks us from implementing some useful API, like an async version of MultiError.catch) (now fixed, thanks @1st1!)
  • issue29590 – fully worked around inside Trio; still affects other libraries like asyncio, but crossing it off our list
  • issue29587 – fully worked around inside Trio; still affects other libraries like asyncio, but crossing it off our list
  • issue29728 (fix merged, thanks @Mariatta!)
  • issue29515 – worked around inside Trio
  • issue29988
  • issue30038 (makes our test suite flaky on windows – see control-C sometimes getting missed on Windows #119. PR submitted and awaiting review) merged, thanks @Haypo!
  • issue30039 (PR submitted and awaiting review) merged, thanks @1st1!
  • issue30050 - PR submitted merged, thanks @1st1!
  • bpo-30579 - PR submitted for the most important part merged!
  • bpo-32359
  • bpo-32561

• better ergonomics for MultiErrors (catching, printing, rethrowing...). Fundamentally the issue here is that in trio, it's effectively possible to call multiple functions in parallel, so we need a way to handle multiple errors raised in parallel. Some clever hacks let us get a long way, but currently this is really stretching the limits of the assumptions baked into Python's exception handling machinery. There are a number of pieces to this; I'm not sure all of them. But:

  • no brainer: making traceback objects instantiable or mutable (both we and jinja2 are carrying disgusting code to work around this) – this is bpo-30579

  • would be nice: attaching attributes to tracebacks (probably: subclassing them)

    • one use case here is to hide/de-emphasize parts of the traceback that are in trio's guts; I think Nick had a similar use case for wanting a way to hide tracebacks inside the import machinery guts?

  • better control over implicit exception chaining. here's an example where implicit exception chaining corrupts our exception reporting and there's currently nothing we can do about it:

    v = ValueError()
    v.__context__ = KeyError()
    def discard_NameError(exc):
        if isinstance(exc, NameError):
            return None
        return exc
    with pytest.raises(ValueError) as excinfo:
        with MultiError.catch(discard_NameError):
            raise MultiError([v, NameError()])
    assert isinstance(excinfo.value.__context__, KeyError)
    assert not excinfo.value.__suppress_context__

    because Python will overwrite the ValueError's __context__ in the catch's __exit__, even though it's already set. There's no way to stop it. [Well... I guess we could exploit issue29587. But that seems a bit evil?] [Update! I actually did find an effective countermeasure – see Correctly preserve exception __context__ in MultiError.catch #165. It's a little bit gross but it's certainly not the worst thing in the multierror code; not sure whether it's worth trying to get a better solution upstream or not.]

  • better hooks to control exception printing? This is the biggie – overriding sys.excepthook is really tacky, and has major limitations (e.g. ipython and pytest have their own exception printing code). But it's also the vaguest, because I'm not sure what this would look like

• better support for safe KeyboardInterrupt management (in particular for __(a)exit__ blocks, with their currently unfixable race condition). This is complicated and gets into the code of the bytecode interpreter. Some notes:

  • for sync context managers, SETUP_WITH atomically calls __enter__ + sets up the finally block
  • for async context managers, the async setup stuff is split over multiple bytecodes, so we lose this guarantee -- it's possible for a KI to arrive in between calling __aenter__ and the SETUP_ASYNC_WITH that pushes the finally block
  • for sync context managers, WITH_CLEANUP_START atomically calls __exit__
  • for async context managers, there again are multiple bytecodes (WITH_CLEANUP_START calls __aenter__ (i.e., instantiates the coroutine object), then GET_AWAITABLE calls __await__, then LOAD_CONST to get the None to use as the initial send, then YIELD_FROM to actually run the __aenter__ body
  • and, of course, even once we enter the __(a)exit__, there's currently no way for that entry to atomically enable KI protection.
  • so, proposals, I think?:
    • make entering an __aenter__ or __aexit__ a single bytecode, or otherwise atomic WRT KeyboardInterrupt delivery (now filed as issue29988)
    • add a field to the stack frame that points to the function object (if any). This avoids circular references (unlike putting it on the code object), would make entry/exit atomic, is much more elegant than the way we currently have to hack everyone's locals dicts, would help other projects like pytest with it's __tracebackhide__ hack, and would enable better introspection in general (right now you can't even print __qualname__ in tracebacks!)

[njsmith]

There are some signal handling issues to add to the list, see #109.

The main thing is that it would be good if there were a way to use set_wakeup_fd but have it ignore EWOULDBLOCK errors instead of printing a warning to stderr. (From a quick spot check, tornado and twisted also use set_wakeup_fd and would also benefit from this change – set_wakeup_fd currently assumes that you want to actually reuse the wakeup fd as a communications channel, but asyncio and curio are the only async libraries that I can find that attempt to (unreliably) use their wakeup fd like this. I also checked libuv.)

[njsmith]

Closing in favor of #495