Python 3.8 wishlist

[njsmith]

3.7 is in beta now, so I'm closing the old 3.7 wishlist (#103) and starting a new one :-). This isn't complete or fully-fleshed out, but to get us started, here's some stuff off the top of my head:

• Better MultiError support: the main two pain points are catching them, and printing them (in particular we can override sys.excepthook, but that doesn't help with other code that wants to print or capture tracebacks like IPython, pytest, raven, ...)

• Whatever we decide we need to improve handling of cancel scope/nursery interaction with generators. PEP 568 is one approach (Implement a strategy for handling (async) generator cleanup #265). This is complicated because cancel scopes and nurseries have somewhat different needs. Or maybe we just want to disallow yielding from inside them... but that would also need some kind of help from the interpreter. Details: [discussion] What to do about cancel scopes / nurseries inside generators? #264 (comment)

• Motion on bpo-32561 would be neat (better performance for async file operations)

• Making KeyboardInterrupt protection more reliable:

  • bpo-29988
  • some way to make @enable_ki_protection atomic; for example, bpo-12857

• Try proposing a hook that runs on bare calls to async functions (so e.g. await f() doesn't call the hook, but f() does). Trio would make this an error (Can we make forgetting an await be an error? #79), asyncio would make it call ensure_future, twisted would call asDeferred, etc.

  • There was a longish discussion of this starting here

• Adding a signal.raise_ wrapper (bpo-35568, see also Our current mechanism for accessing the C raise() function is broken on Windows if using the debug CRT #816)

[oremanj]

Regarding bpo-12857 for KI atomicity, could we just use a flag in the code object co_flags? The decorator could reconstruct the code object with the new flags, like @types.coroutine does. And frames already point to the running code object.

I guess if we're to stop munging frame objects at runtime to make PyPy happy, it would need to be two flags, one meaning "enable KI protection" and one "disable KI protection". The same CO_* flags namespace is used for __future__ flags that only affect parsing, not runtime; setting these on a code object should be safe, and they're very unlikely to be reused for more flags that do affect runtime behavior in the (ha) future. We could pick, say, CO_FUTURE_UNICODE_LITERALS and CO_FUTURE_DIVISION. Or, in a nod to PEP 401, CO_FUTURE_BARRY_AS_FLUFL.

PyPy seems to define the same CO_FUTURE_* flags as CPython does, with the same values.

[njsmith]

Future flags can be consulted at runtime in case you need to call eval, for example Patsy does this. Reusing future flags that don't mean anything anymore is clever though... However, I'm nervous that reconstructing code objects on the fly will actually make pypy even crankier? It's fine (for some value of fine :-)) for functions statically decorated with @ki_*_protection, but we also apply it dynamically, e.g. when starting a child task or in run_sync_in_worker_thread. Anyway, we should probably move this to its own issue :-). Also, there isn't much point worrying about tiny race conditions here unless we're going to fix all of them, which definitely does require upstream fixes (bpo-29988), so not sure how energy it's worth putting into sneaky solutions like this.