Python Iterators and Generators

Iteration — processing a sequence of values one at a time — is one of the most common patterns in programming, and Python formalizes it through the iterator protocol. Understanding this protocol explains what actually happens under the hood every time you write a "for" loop, and it opens the door to generators, which let you produce sequences of values lazily instead of building them all in memory up front.

This distinction matters enormously for performance: a generator that produces a billion values one at a time uses a small, constant amount of memory, while a list holding a billion values does not.

Why it matters

Every for loop in Python uses an iterator, and generators are the easy way to write one. A generator produces values one at a time and keeps nothing in memory, so it can process a file of any size or an endless stream. They are used throughout the standard library and are a favourite interview topic.

The Iterator Protocol: __iter__ and __next__

An "iterable" is any object you can loop over — it implements __iter__, which returns an "iterator." An iterator implements __next__, which returns the next value each time it's called, and raises the built-in StopIteration exception once there are no more values left. A "for" loop is essentially syntactic sugar: it calls iter(obj) once to get an iterator, then repeatedly calls next() on it until StopIteration is raised, at which point the loop ends cleanly.

Generator Functions and yield

Writing a full class with __iter__ and __next__ just to produce a sequence of values is verbose. A generator function is a much simpler shortcut: any function containing the yield keyword automatically becomes a generator function, and calling it returns a generator object (which already satisfies the iterator protocol) rather than running the function body immediately. Each call to next() resumes the function from exactly where it last paused at a yield, runs until the next yield (or the function ends), and returns that yielded value.

Memory Efficiency: Generators vs Lists

A list comprehension like [x * x for x in range(10_000_000)] builds the entire list of ten million values in memory before you can use any of them. A generator expression, written with parentheses instead of brackets — (x * x for x in range(10_000_000)) — or an equivalent generator function, computes each value only when it's requested and never holds the whole sequence in memory at once. This "lazy evaluation" is exactly why functions like range() in Python 3, and many standard library iteration tools, are designed around the iterator protocol rather than returning full lists.

Examples

class CountDown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

for number in CountDown(3):
    print(number)

3
2
1

def countdown_gen(start):
    current = start
    while current > 0:
        yield current
        current -= 1

gen = countdown_gen(3)
print(next(gen))
print(next(gen))
print(list(gen))       # exhausts the remaining values

import sys
list_version = [x for x in range(100000)]
gen_version = (x for x in range(100000))
print(sys.getsizeof(list_version) > sys.getsizeof(gen_version))

3
2
[1]
True

Exercise

Write two generator functions. evens(limit) yields the even numbers below the limit, and countdown(n) yields n, n-1, ... down to 1. The program prints the evens below 10, then takes one value from a countdown with next, then prints what is left.

Expected output: [0, 2, 4, 6, 8], 3, [2, 1] (one per line)

def evens(limit):
    # TODO: yield each even number below limit
    pass


def countdown(n):
    # TODO: yield n, n - 1, ... 1
    pass


print(list(evens(10)))

timer = countdown(3)
print(next(timer))
print(list(timer))

def evens(limit):
    for number in range(0, limit, 2):
        yield number


def countdown(n):
    while n > 0:
        yield n
        n -= 1


print(list(evens(10)))

timer = countdown(3)
print(next(timer))
print(list(timer))

Quick quiz

Interview questions