`async fn` tracking issue

[davidhewitt]

This issue is a placeholder issue which I'd like to use to keep track of what we need to do to support async fn for #[pyfunction] (and #[pymethods]).

While this would be a great feature to have, I think anything which we merge into PyO3 core should be simple to use, performant, and (ideally) async-runtime agnostic.

Users have started researching the design space of how possible cross-language implementations could be written. The most advanced candidate is pyo3-async-runtimes, (formerly pyo3-asyncio). See the guide as a great example of how to use it: https://pyo3.rs/latest/ecosystem/async-await.html

At the moment I suggest we allow some time for that and other crates to mature, and once we understand performance trade-offs / difficulties etc. we can consider upstreaming something here.

As this is supported by third-party crates I think we can afford patience for now.

[awestlake87]

pyo3-asyncio still needs some time and development to mature. There are currently a few things that I think need to be addressed:

• Documentation and testing for pyo3-asyncio within native modules - seems like many users need this to get started
• Cleaner initialization - better asyncio configuration, potentially some AlreadyInitialized errors, etc.
• Stream <-> async generator conversions - my first draft had some performance concerns
• Cleaner error handling - no complaints yet, but I have a few nitpicks here and there

I've been taking a bit of a passive approach to this library lately to see what users need it for, and it seems like these are the main concerns right now.

[JohnScience]

For the time being, using async conditionally can be enabled with qualifier_attr.

[wyfo]

I've just released pyo3-async (see reddit).
Its approach is completely different from pyo3-asyncio as it is runtime-agnostic on Rust side (on Python side, it needs to know the runtime to yield the correct objects, but the runtime don't need to run to instantiate coroutines/async generators).
I've also added the support for trio, as it's quite popular (and supporting it was simple enough, contrary to curio for example).
Last but not least, it supports async generators (simply reusing its coroutine implementation for __anext__), and cancellation.

Adding a macro for async fn is on my TODO list, but I think I have to discuss about it here first, whether or not the implementation could be good enough to be quoted in the documentation.

EDIT: This crate is just a one week POC (actually the first time I use PyO3 ever). It lacks advanced tests, documentation examples, benchmarks, etc. I just wanted to explore the event loops interoperation, and because it worked fine, I've written this crate to submit it to your feedback.

[wyfo]

I've run some quick benchmarks to compare both pyo3-async and pyo3-asyncio. Results are as expected:

• for wrapping of already ready future, performances are not comparable (more than 10x faster, but can be even worse if event loops are busy). This is because pyo3-async doesn't spawn task both on Python and Rust side, so successful polling is done synchronously on Python await operation, which become synchronous too. On the other side, pyo3-async will always be asynchronous.
• you can also wrap spawned tokio task in pyo3-async coroutine, so you can emulate pyo3-asyncio behavior, and thus you get comparable performances.

Here is a simple example illustrating the first point:

#[pymodule]
fn example(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
    m.add_class::<Sender>()?;
    Ok(())
}

#[derive(Clone)]
#[pyclass]
struct Sender(tokio::sync::mpsc::Sender<()>);

impl Sender {
    async fn future(self) -> PyResult<()> {
        self.0.send(()).await.ok();
        Ok(())
    }
}

#[pymethods]
impl Sender {
    #[new]
    fn __new__(capacity: usize) -> Self {
        let (tx, mut rx) = tokio::sync::mpsc::channel(capacity);
        pyo3_asyncio::tokio::get_runtime().spawn(async move { while rx.recv().await.is_some() {} });
        Self(tx)
    }

    fn send_async(&self) -> asyncio::Coroutine {
        asyncio::Coroutine::from_future(self.clone().future())
    }

    fn send_asyncio<'py>(&self, py: Python<'py>) -> PyResult<&'py PyAny> {
        pyo3_asyncio::tokio::future_into_py(py, self.clone().future())
    }
}

import asyncio
import example  # built with maturin
import time

CAPA = 100
N = 10000

async def send_async():
    sender = example.Sender(CAPA)
    start = time.perf_counter()
    for _ in range(N):
        await sender.send_async()
    print("pyo3-async", time.perf_counter() - start)

async def send_asyncio():
    sender = example.Sender(CAPA)
    start = time.perf_counter()
    for _ in range(N):
        await sender.send_asyncio()
    print("pyo3-asyncio", time.perf_counter() - start)
    
for _ in range(3):
    asyncio.run(send_async())
    asyncio.run(send_asyncio())
    print("===============")

P.S. I'm currently implementing a pyasync macro, and I've some interesting issues about GIL-bound parameters, but I think I've found an elegant solution (with a bit of unsafe behind).

[wyfo]

I've publish a new release of pyo3-async with #[pyfunction]/#[pymethods]!

@davidhewitt Would you mind taking a look at this crate? Could it be worth to mention it in PyO3 documentation?

[davidhewitt]

@wyfo sorry for the slow reply, this looks like a very interesting development! I'm reading through the code today and will let you know what I think ASAP!

[davidhewitt]

Overall I think it looks like a great crate, though I would suggest you add some CI to test with multiple Python / OS versions. Hopefully our testing in CI here means that you don't have any platform-specific quirks, but I wouldn't guarantee that.

This would definitely be welcome of a mention in the PyO3 documentation. I also suspect there are common parts which can be shared with pyo3-asyncio (and both projects may benefit from sharing what they can for interoperability's sake).

At the moment I think the key API reason which would prevent pyo3-async from being brought into the main PyO3's package would be that you have a couple of #[pyclass] types to represent various coroutines and generators on the Python side. The downside of this is that these end up as unique types in each Python extension module compiled using pyo3-async. Apart from PanicException, which exists out of necessity (and we have discussed a lot, we have avoided adding #[pyclass] types into PyO3 itself.

This prompted me to ask the CPython core devs in the Python discourse how they might want us to deal with these #[pyclass] types. Let's see what they say.

A couple of more specific thoughts:

• It's very nice to see a runtime-agnostic solution to how to bind Rust and Python futures.
• The fact you need to re-wrap the PyO3 macros is unfortunate though also understandable. I wonder if there's something which can be done to avoid the need for you to do this? At least from a cursory scan of the implementation you're not overly coupled to the PyO3 macro implementations.

[wyfo]

CI is indeed on the TODO list, my first goal was write the code to illustrate the potential of this approach.

I don't think however that there are common parts with pyo3-asyncio, because both crates don't operate on the same level, as pyo3 fully relies on runtimes. But both return Python object or trait implementation, so nothing should prevent to work with both crates at the same time.

Actually, I did not know about this #[pyclass] issue, and that's indeed unfortunate. Funny enough, pyo3-asyncio has the same issue, but on Python-to-Rust side, while my crate is concerned on Rust-to-Python side.
Well, the answers to your post on Python forum are interesting, particularly the Cython hack. It may indeed be a thing for PyO3 coroutine implementation (but I'm still hoping that CPython will fix the coroutine ABI in the future).

For the macro part, I've hesitated to do a unique #[pyasync] macro to decorate both PyO3 #[pyfunction] and #[pymethods], but I've finally preferred re-wrapping them. But I'm not really coupled with their implementation, the only thing I need is to know if there is a #[pyo3(name = "...")] attribute, because PyO3 doesn't support having two of them.
In fact, another design decision was to generate an additional function returning the coroutine (to be decorated with #[pyfunction], or inside #[pymethods] impl) and keeping the original async one, or to modify the function in place. Because I don't like to modify code in macro, I've choosen the first solution, but the generated function needs then to have a #[pyo3(name = "...")] to match the original (if it doesn't already have one).
Also, the goal was to showcase a possible integration in PyO3, so re-wrapping was more suited. But a #[pyasync] thing would be ok for me too.

[davidhewitt]

Yes, it sounds like if Cython has set a precedent with its Coroutine types it seems reasonable to me for us to do something similar. @adamreichold and I have talked a lot about this kind of problem in the past, see #3073. At the time we were leaning towards a pyo3_runtime package, but maybe there's strengths of both approaches for different parts of the problem.

Maybe if we can sketch out a summary of all the pieces that we'd need to add to PyO3 to support async fn, and also what the limitations of our support would be, we could make a decision on whether we now have enough design knowledge to be confident in adding this feature.

Main concerns I'd have would be on:

• interactions between the GIL and async code
• any performance limitations which we know about in this design
• how to ensure PyO3 core remains async runtime agnostic
• how applications already using pyo3-asyncio can build upon PyO3's async fn

If it all fits together well, maybe with this latest development we get to a point where async fn sugar is supported in PyO3 and pyo3-asyncio can build upon that to add advanced helpers and async runtime integrations? (And we also perhaps bring pyo3-asyncio under the pyo3 org, if @awestlake87 wants that.)

@wyfo would you be interested in collecting that together, as this is freshest in your mind? (And if we think it makes sense, implementing? 👀)

[wyfo]

After a second reflexion about this #[pyclass] issue, I've changed my mind quite a bit. Actually, several libraries defines their own exceptions, so why not consider PanicException as a per-library defined exception, that should be exported with the rest of the Python objects? I will open an new issue to discuss this particular topic.

Regarding coroutines, what would be the issue of having one coroutine type per extension? (You have this https://github.com/python/cpython/blob/84b7e9e3fa67fb9b92088d17839d8235f1cec62e/Lib/asyncio/coroutines.py#L38, but it's not really an issue). Coroutines are only meant to be awaited, not used with == or isinstance, and you can still use isinstance(..., collections.abc.Coroutine) or asyncio.iscoroutine (which use isinstance), or even inspect.isawaitable. So I quite agree with PetrViktorin.

About async fn support:

• Python async ecosystem is not normalized, contrary to Rust's one with the Future trait. There are multiple async backends, with incompatible implementation; asyncio is of course the main one, because standard, but there is also trio, which is quite popular now, especially with the anyio project, allowing him to be supported in trendy projects like Starlette/FastAPI. Should PyO3 only support asyncio, or multiple backends (with asyncio as default, of course)?
  In pyo3-async, I've chosen to support both asyncio and trio, but also a lazy specialized implementation using sniffio, following the anyio trend. (I did not found a way to support curio, and didn't look to other backends)
• Python coroutines are similar to Rust futures, both are suspendable computations. With asyncio, a Rust Future can "simply" be wrapped into a Python awaitable, which will yield asyncio.Future instantiated on demand, and using the asyncio.Future.set_result as wake method for a custom std::task::Waker. (That's what is implemented in *pyo3-async).
  However, because Python async backends are single-threaded, wake operation must use thread-safe method, e.g. call_soon_threadsafe. (I've added a quick optimization in pyo3-async, I store the thread id at the future creation and check the current thread id in wake method to determine if call_soon_threadsafe must be called)
  The direct binding between Python and Rust async executions is necessary, at least to have decent performance, because it allows synchronous execution of futures already ready (see async fn tracking issue #1632 (comment))
• There is still an important difference between Rust and Python async world: cancellation. In Python, cancellation is an exception that can be caught and even ignored, there can be asynchronous cleanup, while in Rust, you can only drop your future an cleanup with synchronous Drop. That's why Python async cancellation is not equivalent to just dropping a Rust future. A good design should give a way to properly handle Python cancellation.
  In pyo3-async, coroutine wrapper has an optional throw callback, allowing to react to cancellation exception (for example using a mpsc::UnboundedSender). This is the best design I've found, because simple and not tied to a particular library.
• Python awaitable doesn't require to be a coroutine, it just need to implement __await__ (i.e. to be awaitable). However, in pyo3-async, I've chosen to implement directly the coroutine protocol (which imply the awaitable one) because async backends like asyncio or trio don't support raw awaitables (asyncio automatically wraps them in a dummy coroutine), so it seemed to me that it should be more performant, but I've not tested this. It's also possible to implement the generator protocol without implementing __await__, by using types.coroutine wrapper, but I suspect this call could have performance impact; I still have to test it.
• With direct binding between Rust and Python asynchronous execution (see second point), Rust Future is always polled with GIL acquired. However, to be embedded in a Python object, the future must be Send + 'static, so it's not possible for it to capture pyo3::marker::Python or any &pyo3::types::PyAny (#[pyfunction] parameters are thus also concerned), and to hold them across an await boundary. However, because the GIL is held, Python::with_gil is only one thread-local check, so it seems to be an acceptable safety cost, with unsafe Python::assume_gil_acquired for the zero-cost alternative.
  By the way, having the GIL held for Future polling allows to use it to convert the Future result to a Python type, so it doesn't have to be in the future code, allowing them to return a Result<T, E> where T: IntoPy<PyObject>, PyErr: From<E>.
  In pyo3-async, this is materialized by the PyFuture trait and its blanket implementation.

I don't know any performance limitation about this design for now. It is also completely agnostic of any Rust async runtime, and I don't see a reason to change that. In fact, it may be more efficient to spawn a big future into a tokio task, to have it polled exclusively on Rust side, and to wrap the task handle into a Python coroutine, but this optimization is just few lines of code and doesn't require first-class support IMO. But PyO3 needs to expose the Coroutine wrapper to allow this kind of use case (and cancellation support).

Regarding pyo3-asyncio, async fn support/Coroutine wrapper should completely replace future_into_py, while still allowing them to wrap their futures in a tokio/async-std task as mentioned above.

Here is what I can say for now. For the implementation, I would pretty much copy-paste the content of pyo3-async 😇