Multithreading in Python

Many programs spend most of their time waiting: for web pages to download, for API responses, for database queries, for files to be read. Threads let one program wait on many things at once, so twenty slow downloads take about as long as the slowest one instead of the sum of all of them.

This lesson covers the threading module, the high-level concurrent.futures.ThreadPoolExecutor, race conditions and locks, thread-safe queues for producer/consumer designs, and the Global Interpreter Lock (GIL) — including Python's new free-threaded builds.

Threads and the GIL

A thread is an independent flow of execution within a process; threads share memory. In standard CPython, the Global Interpreter Lock lets only one thread execute Python bytecode at a time, so threads do not speed up CPU-heavy pure-Python code — but the GIL is released while waiting for I/O, so threads are excellent for I/O-bound work. Python 3.13+ offers an optional free-threaded build without the GIL; for CPU-bound work today, use multiprocessing.

Creating Threads

threading.Thread(target=func, args=(...)) creates a thread, start() runs it and join() waits for it to finish. Daemon threads (daemon=True) are killed when the main program exits. In most code, prefer ThreadPoolExecutor: it reuses a pool of threads, returns results through futures, and propagates exceptions.

Race Conditions and Locks

When threads modify shared data, operations like count += 1 (read, add, write) can interleave and lose updates. Protect shared state with threading.Lock in a with block, or avoid sharing entirely by returning results and using thread-safe queue.Queue. Other primitives include RLock, Semaphore (limit concurrency), Event (signal between threads) and Condition.

Threads vs asyncio vs Processes

Use threads for I/O-bound work with blocking libraries (requests, database drivers). Use asyncio for very high numbers of concurrent I/O operations with async libraries. Use processes for CPU-bound work.

Examples

Starting and joining threads

Python

import threading
import time

def download(name, seconds):
    time.sleep(seconds)                     # simulated network wait
    print(f"{name} done")                   # finishing order can vary between runs

start = time.perf_counter()
threads = [threading.Thread(target=download, args=(f"file{i}", 0.3)) for i in range(1, 4)]
for t in threads:
    t.start()
for t in threads:
    t.join()
elapsed = time.perf_counter() - start
print(f"3 downloads in ~{elapsed:.1f}s (sequential would take ~0.9s)")

Output

file1 done
file2 done
file3 done
3 downloads in ~0.3s (sequential would take ~0.9s)

ThreadPoolExecutor with results and exceptions

Python

from concurrent.futures import ThreadPoolExecutor, as_completed
import time

def fetch_price(product):
    time.sleep(0.2)
    if product == "broken":
        raise ValueError("price service error")
    return product, len(product) * 100

products = ["pen", "notebook", "broken", "bag"]
with ThreadPoolExecutor(max_workers=4) as pool:
    futures = {pool.submit(fetch_price, p): p for p in products}
    results = {}
    for future in as_completed(futures):
        name = futures[future]
        try:
            product, price = future.result()
            results[product] = price
        except ValueError as e:
            results[name] = f"failed: {e}"

print(dict(sorted(results.items())))

with ThreadPoolExecutor() as pool:
    print(list(pool.map(str.upper, ["a", "b", "c"])))

Output

{'bag': 300, 'broken': 'failed: price service error', 'notebook': 800, 'pen': 300}
['A', 'B', 'C']

A race condition and fixing it with a Lock

Python

import threading

counter = 0
lock = threading.Lock()

def unsafe_increment(n):
    global counter
    for _ in range(n):
        value = counter
        value += 1
        counter = value

def safe_increment(n):
    global counter
    for _ in range(n):
        with lock:
            counter += 1

for worker in (unsafe_increment, safe_increment):
    counter = 0
    threads = [threading.Thread(target=worker, args=(200_000,)) for _ in range(4)]
    [t.start() for t in threads]
    [t.join() for t in threads]
    print(f"{worker.__name__}: {counter} (expected 800000)")

# The unsafe total varies from run to run and can even come out right by luck —
# that unpredictability is exactly what makes race conditions dangerous.

Output

unsafe_increment: 312489 (expected 800000)
safe_increment: 800000 (expected 800000)

Producer/consumer with queue.Queue

Python

import queue
import threading

tasks = queue.Queue()
results = []

def worker(worker_id):
    while True:
        item = tasks.get()
        if item is None:              # sentinel: stop
            tasks.task_done()
            break
        results.append((item, item ** 2))
        tasks.task_done()

workers = [threading.Thread(target=worker, args=(i,)) for i in range(3)]
for w in workers:
    w.start()
for n in range(1, 7):
    tasks.put(n)
for _ in workers:
    tasks.put(None)
tasks.join()
print(sorted(results))

Output

[(1, 1), (2, 4), (3, 9), (4, 16), (5, 25), (6, 36)]

Common Mistakes

Using threads to speed up CPU-bound pure-Python code (the GIL prevents it) — use processes.
Modifying shared variables from several threads without a lock.
Forgetting join(), so the main program continues before workers finish.
Swallowing exceptions in threads; ThreadPoolExecutor futures re-raise them with result().

Key Points to Remember

Threads suit I/O-bound work; the GIL limits CPU-bound parallelism in standard CPython.
ThreadPoolExecutor with submit/map/as_completed is the easiest API.
Protect shared state with Lock, or avoid sharing and use queue.Queue.
Choose threads, asyncio or processes based on the workload.

Practice the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

Use your local project environment for these examples. Codelab currently runs Python and HTML/CSS/JavaScript; framework examples may need project dependencies.