Python Generators and Coroutines: A Deep Dive

Python Internals: Generators & Coroutines

The Python function ‘get_trades’ can cause high latency and massive RAM usage. Generators can be used to optimize this function, allowing for lazy evaluation and reduced memory usage.

Why This Matters

In real-world applications, the ‘eager’ pattern of doing all the work and collecting all the results before handing them over can lead to high latency, massive RAM usage, and fragility. Generators, on the other hand, allow for lazy evaluation, where data is only pulled through as fast as it is consumed, making them ideal for processing large datasets or infinite streams.

Key Insights

• Generators can be used to implement lazy evaluation, reducing memory usage and improving performance (Python 3.x)
• The ‘yield’ keyword can be used to create generators, allowing for pause and resume functionality (Python documentation)
• Coroutines can be used to create stateful processing units, enabling the creation of complex pipelines and workflows (Real Python)

Working Examples

A simple generator function that counts down from a given number

def countdown(n):
    while n > 0:
        yield n
        n -= 1

A generator function that simulates an infinite stream of market ticks

def market_ticker():
    import random, itertools
    symbols = ['AAPL', 'GOOG', 'MSFT', 'AMZN']
    for i in itertools.count():
        yield {
            'symbol': random.choice(symbols),
            'price': round(random.uniform(100, 300), 2),
            'volume': random.randint(100, 10000),
        }

Practical Applications

• Use case: Processing large log files with generators (Company: Google, Behavior: Reduced memory usage)
• Pitfall: Using eager evaluation for large datasets (Anti-pattern: Loading entire dataset into memory, Consequence: High latency and memory usage)

References:

• https://dev.to/aykhlf_yassir/python-internals-generators-coroutines-11j2