Multiprocessing in Python

For CPU-heavy work — image processing, number crunching, parsing huge files, training simple models — threads do not help in standard Python because of the GIL. Processes do: each process has its own Python interpreter and memory, so several processes run truly in parallel on multiple CPU cores.

This lesson covers the multiprocessing module, ProcessPoolExecutor, why the if __name__ == "__main__" guard is mandatory, passing data between processes, shared state, and the costs to be aware of.

Processes and Pools

multiprocessing.Process(target=func, args=...) starts a separate process; Pool and the higher-level concurrent.futures.ProcessPoolExecutor distribute work across a pool of worker processes (by default one per CPU core, see os.cpu_count()). The API mirrors ThreadPoolExecutor: submit(), map() and futures.

The __main__ Guard

On Windows and macOS, child processes are started by importing your main module again ("spawn"). Without if __name__ == "__main__": around the code that creates processes, each child would start new children endlessly. Functions sent to workers must be defined at module top level so they can be pickled.

Communication and Shared State

Processes do not share memory by default; arguments and results are pickled and sent between them. multiprocessing.Queue and Pipe exchange messages; Value, Array and Manager provide shared state with locks. Prefer passing inputs and returning results over sharing state.

Costs and Alternatives

Starting processes and pickling data takes time and memory, so parallelism pays off only when each task does substantial work. Send chunks of work (chunksize) rather than tiny tasks. For numeric work, NumPy's vectorised operations often beat manual multiprocessing; for large-scale data, tools like Dask or PySpark distribute work across machines.

Examples

# primes.py  — run with: python primes.py
import math
import os
import time
from concurrent.futures import ProcessPoolExecutor

def count_primes(limit):
    count = 0
    for n in range(2, limit):
        if all(n % d for d in range(2, math.isqrt(n) + 1)):
            count += 1
    return count

if __name__ == "__main__":            # required for multiprocessing
    jobs = [150_000] * 8
    start = time.perf_counter()
    sequential = [count_primes(j) for j in jobs]
    t_seq = time.perf_counter() - start

    start = time.perf_counter()
    with ProcessPoolExecutor() as pool:
        parallel = list(pool.map(count_primes, jobs))
    t_par = time.perf_counter() - start

    print("cores:", os.cpu_count(), "same results:", sequential == parallel)
    print(f"sequential {t_seq:.1f}s, parallel {t_par:.1f}s")

cores: 8 same results: True
sequential 6.4s, parallel 1.3s

from multiprocessing import Process, Queue, Pool

def producer(q):
    for i in range(3):
        q.put(f"message {i}")
    q.put(None)

def area(width, height):
    return width * height

if __name__ == "__main__":
    q = Queue()
    p = Process(target=producer, args=(q,))
    p.start()
    while (msg := q.get()) is not None:
        print("received", msg)
    p.join()

    with Pool(processes=4) as pool:
        print(pool.map(abs, [-1, -2, 3]))
        print(pool.starmap(area, [(2, 3), (4, 5), (6, 7)]))

received message 0
received message 1
received message 2
[1, 2, 3]
[6, 20, 42]

from multiprocessing import Process, Value

def add_many(total, n):
    for _ in range(n):
        with total.get_lock():
            total.value += 1

if __name__ == "__main__":
    total = Value("i", 0)
    workers = [Process(target=add_many, args=(total, 10_000)) for _ in range(4)]
    for w in workers:
        w.start()
    for w in workers:
        w.join()
    print(total.value)

40000