Issue switching between synchronous and asynchronous mode with Client

[pierreglaser]

Hi,
I played around with distributed to solve joblib/joblib#852, especially the ability of Client to switch from sync to async mode.

The documentation seems to state that it is possible:

distributed/docs/source/asynchronous.rst

Lines 47 to 58 in e5ec8da

	If you want to reuse the same client in asynchronous and synchronous
	environments you can apply the ``asynchronous=True`` keyword at each method
	call.
	.. code-block:: python
	client = Client() # normal blocking client
	async def f():
	futures = client.map(func, L)
	results = await client.gather(futures, asynchronous=True)
	return results

But the following script just hangs forever without outputing anything.

from tornado.ioloop import IOLoop
from distributed import Client


async def f():
    results = client.map(lambda x: x, [(1,), (2,)])
    results = await client.gather(results, asynchronous=True)
    return results


if __name__ == "__main__":
    client = Client()
    res = IOLoop.current().run_sync(f)
    print("final result: {}".format(res))

When using Client(asynchronous=True) instead, the scripts runs fine.
Is it expected behavior?

Using python3.7, distributed master, tornado 6.0.3

[mrocklin]

You need to run f in the IOLoop that is created by the Client. Tornado's IOLoop.current function returns the IOLoop for the current thread, while the Client, when running in synchronous mode has to make an IOLoop on a different thread.

Probably you want the following:

from tornado.ioloop import IOLoop
from distributed import Client


async def f():
    results = client.map(lambda x: x, [(1,), (2,)])
    results = await client.gather(results, asynchronous=True)
    return results


if __name__ == "__main__":
    client = Client()
    res = client.sync(f)  # <<---- changed line
    print("final result: {}".format(res))

[pierreglaser]

Thanks a lot -- sorry for the noise. Closing.

[mrocklin]

It was a good question, and it's nice to have a public record of it. Thanks.

…

On Thu, Jul 18, 2019 at 9:09 AM Pierre Glaser ***@***.***> wrote: Closed #2853 <#2853>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2853?email_source=notifications&email_token=AACKZTE2KOGFPUQE4YG6G6LQAB2TFA5CNFSM4IE2GAZKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOSSK4YXY#event-2492845151>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACKZTCBI7PX4Q6COUQ6P2DQAB2TFANCNFSM4IE2GAZA> .

[dhirschfeld]

Perhaps #2293 might have helped avoid some confusion?