“The async/await syntax is a major improvement in Python, allowing asynchronous code to look and feel just like synchronous code, making it much more readable and understandable.”
– Guido van Rossum (Creator of Python)
I always found it an arduous task to synchronously scrape from multiple websites. Requesting a server, then waiting for the data, and then doing it again and again for each website. Even someone who does not have much knowledge about programming can point out that this is a tedious, slow, and inefficient way of doing things.
But now, with more popularity of asynchronous programming in Python, this arduous task has turned into a piece of cake. One test found that asynchronous programming handled 37% more requests than synchronous programming in the same time frame. (Medium: Sync vs Async – Which is Better?) Sound nothing less than a cheat code, right?
Let’s understand the concept of asynchronous programming better, following those steps by Kirill Yurovskiy, and learn how it makes various operations more efficient and scalable in modern software development.
Understanding Synchronous vs Asynchronous Code
Before diving into asynchronous programming, let me tell you how synchronous and asynchronous code differ.
In synchronous programming, operations happen one after another. There is no option but to wait for a preceding task to finish before moving on to the next task, hence I/O-bound operations may introduce noticeable delays.
Example:
import time
def task_1():
time.sleep(2) # Simulates a delay
print(“Task 1 completed”)
def task_2():
print(“Task 2 completed”)
task_1()
task_2()
Output:
Task 1 completed
Task 2 completed
Here, task_2 cannot execute until task_1 finishes, resulting in wasted time.
On the other hand, with asynchronous programming, you can run a number of tasks in parallel without waiting for each other. Instead of blocking, tasks yield control back to the event loop when idle, allowing other tasks to execute. Python does this using the asyncio library and the async/await syntax.
FUN FACT Python wasn’t named after a snake! It was actually named after Monty Python’s Flying Circus, a British comedy show. Creator Guido van Rossum was a fan of the show and wanted to give his new language a fun name.
The Event Loop: The Heart of Async Programming
Some of you might be confused about how it works, so to answer your doubts, Python’s asynchronous programming centers around an event loop. It is responsible for executing tasks in parallel by handling coroutines. Schematically, you can consider an event loop as a sort of scheduler: it decides which tasks to continue executing.
In Python, asyncio implements an event loop that works in the following form:
The event loop feeds in and executes coroutines.
An await keyword inside a coroutine is used to suspend its execution and return control back to the event loop.
When the awaited operation is completed, the event loop will continue the coroutine.
Here is a very simple example:
import asyncio
async def task_1():
print(“Task 1 started”)
await asyncio.sleep(2) # Simulating a non-blocking delay
print(“Task 1 completed”)
async def task_2():
print(“Task 2 started”)
print(“Task 2 completed”)
async def main():
await asyncio.gather(task_1(), task_2()) # Run tasks concurrently
asyncio.run(main())
Output:
Task 1 started
Task 2 started
Task 2 completed
Task 1 completed
Here, you can see how task_1 and task_2 execute concurrently. While task_1 is waiting during await asyncio.sleep(2), the event loop schedules and completes task_2.
Coroutines and the async/await Syntax
For those of you who are not aware, coroutines are Python functions defined by the async def syntax. They are considered the building blocks for asynchronous programming, and they represent operations that may be paused and resumed. It’s only possible to call a coroutine with await, which transfers control to the event loop to run the coroutine.
Example:
import asyncio
async def say_hello():
await asyncio.sleep(1)
print(“Hello, World!”)
asyncio.run(say_hello())
In this example, await asyncio.sleep(1) suspends execution, allowing the event loop to attend to other duties. When the delay finishes, say_hello resumes.
Tasks and Futures: Managing Concurrent Operations
Moving on to another major element, in asyncio, the notions of Tasks and Futures revolve around the management and scheduling of coroutines. A Task wraps a coroutine and allows it to run concurrently. A Future is the result of an asynchronously conducted operation that may or may not have occurred.
You can create tasks using asyncio.create_task().
Example:
import asyncio
async def task_1():
await asyncio.sleep(2)
print(“Task 1 finished”)
async def task_2():
print(“Task 2 finished”)
async def main():
t1 = asyncio.create_task(task_1())
t2 = asyncio.create_task(task_2())
await t1
await t2
asyncio.run(main())
Here, both task_1 and task_2 run concurrently because they had been wrapped as tasks.
DO YOU KNOW? Asynchronous programming significantly outperforms synchronous programming in terms of non-blocking I/O operations and improved responsiveness, as depicted in the graph below.
Common Async Patterns and Best Practices
The common async patterns and best practices I’ve mentioned below will help you enhance the efficiency of your programs.
Use asyncio.gather() for Concurrency: This can allow you to run several coroutines concurrently.
Use Concurrency Limits with Semaphores: When dealing with hundreds of tasks, use asyncio.Semaphore() in order to bind the number of running coroutines at any given time.
Avoid Blocking Calls: Make sure all calls to functions like time. Sleep () that will take for a while or block are made non-blocking-that is, await asyncio.sleep()
Error handling for asynchronous code is not much different than that for synchronous code. You have to wrap any coroutines in a try-except block.
Example:
import asyncio
async def faulty_task():
await asyncio.sleep(1)
raise ValueError(“Something went wrong”)
async def main():
try:
await faulty_task()
except ValueError as e:
print(f”Caught exception: {e}”)
asyncio.run(main())
Output:
Caught exception: Something went wrong
Working with Async Context Managers
Let me give you an example to help you understand how Python supports asynchronous context managers by async.
Example Working with files asynchronously:
import aiofiles
import asyncio
async def write_file():
async with aiofiles.open(“example.txt”, mode=”w”) as file:
await file.write(“Hello, Async World!”)
asyncio.run(write_file())
Database Operations with asyncio
When working with asynchronous database libraries like aiomysql or asyncpg for PostgreSQL, asyncio is considered a great option. These allow for non-blocking access to databases, hence enhancing the performance of I/ O-bound programs.
rows = await conn.fetch(“SELECT * FROM table_name”)
await conn.close()
print(rows)
asyncio.run(fetch_data())
Practical Example: Writing an Async Web Scraper
Like I hinted in the introduction, asynchronous programming proves particularly helpful when you need to make multiple network requests, which happens quite frequently in web scraping:
import aiohttp
import asyncio
async def fetch_url(session, url):
async with session.get(url) as response:
print(f”Read {response.content_length} bytes from {url}”)
Here, all URLs are fetched concurrently using aiohttp and asyncio.gather().
Testing Async Code: How and What
Testing of async functions requires testing libraries compatible with async – for instance, pytest with the pytest-asyncio plugin.
Example:
import asyncio
import pytest
async def sample_function():
await asyncio.sleep(1)
return 42
@pytest.mark.asyncio
async def test_sample_function():
result = await sample_function()
assert result == 42
Common Pitfalls and How to Avoid Them
There are some common pitfalls that I’ve seen developers struggling with, so to give you an idea and help you out with those, I have listed some common ones below.
Mixing Blocking Code with Async: The blocking operations halt the event loop. Use non-blocking alternatives.
Overloading the Event Loop: Avoid scheduling too many tasks simultaneously. Use semaphores to avoid concurrency.
For production-grade asynchronous applications, I would suggest you consider scaling with tools like:
FastAPI: A high-performance web framework using asyncio.
Unicorn: An ASGI server that can handle concurrency efficiently.
Task Queues: Use libraries like Celery for background tasks.
After all that we have discussed, we can conclude that with asyncio, asynchronous programming in Python is a very active and effective way to improve the performance of any I/O-bound application.
Learning about the event loop, coroutines, and best practices I shared in this article, will let developers build scalable, efficient, and responsive systems. Mastering asynchronous programming can open a whole new dimension of performance in Python for developers, helping them build web scrapers, handle database queries, or create API servers.
