Python & Async Simplified

Last updated 1 year ago by Andrew Godwin

python

As some of you may be aware, I have spent many of the last months rewriting Channels to be entirely based on Python 3 and its asynchronous features (asyncio).

Python's async framework is actually relatively simple when you treat it at face value, but a lot of tutorials and documentation discuss it in minute implementation detail, so I wanted to make a higher-level overview that deliberately ignores some of the small facts and focuses on the practicalities of writing projects that mix both kinds of code.

Two Worlds

That first part is the most critical thing to understand - Python code can now basically run in one of two "worlds", either synchronous or asynchronous. You should think of them as relatively separate, having different libraries and calling styles but sharing variables and syntax.

You can't use a synchronous database library like mysql-python directly from async code; similarly, you can't use an async Redis library like aioredis directly from sync code. There are ways to do both, but you need to explicitly cross into the other world to run code from it.

In the synchronous world, the Python that's been around for decades, you call functions directly and everything gets processed as it's written on screen. Your only built-in option for running code in parallel in the same process is threads.

In the asynchronous world, things change around a bit. Everything runs on a central event loop, which is a bit of core code that lets you run several coroutines at once. Coroutines run synchronously until they hit an await and then they pause, give up control to the event loop, and something else can happen.

This means that synchronous and asynchronous functions/callables are different types - you can't just mix and match them. Try to await a sync function and you'll see Python complain, forget to await an async function and you'll get back a coroutine object rather than the result you wanted.

You actually don't need to know the fine details of how it all works to use it - just know that coroutines have to explicitly give up control via an await. This is different to threads or greenlets, which can context-switch at any time.

If you block a coroutine synchronously - maybe you use time.sleep(10) rather than await asyncio.sleep(10) - you don't return control to the event loop, and you'll hold up the entire process and nothing else can happen. On the plus side, nothing else can run while your code is moving through from one await call to the next, making race conditions harder.

This is called cooperative multitasking, and while it has many upsides, this silent failure mode is its main design issue. If you use a blocking synchronous call by mistake, nothing will explicitly fail, but things will just run mysteriously slowly. Python has a debug mode that will warn you about things blocking for too long, along with other common errors, but be aware of one thing - writing explicitly asynchronous code is harder than writing synchronous code.

This is one of the reasons Channels gives you the choice, rather than forcing you into always writing async; sync code is easier to write, generally safer, and has many more libraries to choose from.

You should think of your codebase as comprised of pieces of either sync code or async code - anything inside an async def is async code, anything else (including the main body of a Python file or class) is synchronous code. Notably, init must always be synchronous even if all the class' methods are asynchronous.

Read full Article