Course topics

By WebNest Studio Engineering Team

Python Tutorial

Multiprocessing in Python

For CPU-heavy work — image processing, number crunching, parsing huge files, training simple models — threads do not help in standard Python because of the GIL. Processes do: each process has its own Python interpreter and memory, so several processes run truly in parallel on multiple CPU cores.

This lesson covers the multiprocessing module, ProcessPoolExecutor, why the if __name__ == "__main__" guard is mandatory, passing data between processes, shared state, and the costs to be aware of.

Why it matters

When the work is calculation, not waiting — resizing images, analysing data, running simulations — threads do not help in Python, because of the GIL. Separate processes each have their own interpreter and can use every core of the machine. Knowing when to choose processes over threads is a standard interview question.

Processes and Pools

multiprocessing.Process(target=func, args=...) starts a separate process; Pool and the higher-level concurrent.futures.ProcessPoolExecutor distribute work across a pool of worker processes (by default one per CPU core, see os.cpu_count()). The API mirrors ThreadPoolExecutor: submit(), map() and futures.

The __main__ Guard

On Windows and macOS, child processes are started by importing your main module again ("spawn"). Without if __name__ == "__main__": around the code that creates processes, each child would start new children endlessly. Functions sent to workers must be defined at module top level so they can be pickled.

Communication and Shared State

Processes do not share memory by default; arguments and results are pickled and sent between them. multiprocessing.Queue and Pipe exchange messages; Value, Array and Manager provide shared state with locks. Prefer passing inputs and returning results over sharing state.

Costs and Alternatives

Starting processes and pickling data takes time and memory, so parallelism pays off only when each task does substantial work. Send chunks of work (chunksize) rather than tiny tasks. For numeric work, NumPy's vectorised operations often beat manual multiprocessing; for large-scale data, tools like Dask or PySpark distribute work across machines.

Examples

Parallel CPU-bound work with ProcessPoolExecutor

Python
# primes.py  — run with: python primes.py
import math
import os
import time
from concurrent.futures import ProcessPoolExecutor

def count_primes(limit):
    count = 0
    for n in range(2, limit):
        if all(n % d for d in range(2, math.isqrt(n) + 1)):
            count += 1
    return count

if __name__ == "__main__":            # required for multiprocessing
    jobs = [150_000] * 8
    start = time.perf_counter()
    sequential = [count_primes(j) for j in jobs]
    t_seq = time.perf_counter() - start

    start = time.perf_counter()
    with ProcessPoolExecutor() as pool:
        parallel = list(pool.map(count_primes, jobs))
    t_par = time.perf_counter() - start

    print("cores:", os.cpu_count(), "same results:", sequential == parallel)
    print(f"sequential {t_seq:.1f}s, parallel {t_par:.1f}s")
Output
cores: 8 same results: True
sequential 6.4s, parallel 1.3s

Process, Queue and a Pool with starmap

Python
from multiprocessing import Process, Queue, Pool

def producer(q):
    for i in range(3):
        q.put(f"message {i}")
    q.put(None)

def area(width, height):
    return width * height

if __name__ == "__main__":
    q = Queue()
    p = Process(target=producer, args=(q,))
    p.start()
    while (msg := q.get()) is not None:
        print("received", msg)
    p.join()

    with Pool(processes=4) as pool:
        print(pool.map(abs, [-1, -2, 3]))
        print(pool.starmap(area, [(2, 3), (4, 5), (6, 7)]))
Output
received message 0
received message 1
received message 2
[1, 2, 3]
[6, 20, 42]

Shared counter with Value and a Lock

Python
from multiprocessing import Process, Value

def add_many(total, n):
    for _ in range(n):
        with total.get_lock():
            total.value += 1

if __name__ == "__main__":
    total = Value("i", 0)
    workers = [Process(target=add_many, args=(total, 10_000)) for _ in range(4)]
    for w in workers:
        w.start()
    for w in workers:
        w.join()
    print(total.value)
Output
40000

Common Mistakes

  • Omitting the if __name__ == "__main__" guard, causing errors or endless process creation on Windows/macOS.
  • Parallelising tiny tasks where process start-up and pickling costs outweigh the benefit.
  • Passing lambdas or nested functions to process pools (they cannot be pickled).
  • Expecting global variables changed in a child process to change in the parent.

Exercise

Use a ProcessPoolExecutor to square the numbers 1 to 4 in separate processes and print the results. Put the code that starts the processes under the __main__ guard.

Expected output: [1, 4, 9, 16]

Python · starter code
from concurrent.futures import ProcessPoolExecutor


def square(number):
    return number * number


# TODO: under the __main__ guard, create a ProcessPoolExecutor,
#       map square over [1, 2, 3, 4] and print the results as a list
Show hints
  • The guard is if __name__ == "__main__":. Without it, on Windows and macOS each new process would start more processes.
  • executor.map returns the results in the order of the inputs.
Show solution
Python · solution
from concurrent.futures import ProcessPoolExecutor


def square(number):
    return number * number


if __name__ == "__main__":
    with ProcessPoolExecutor() as executor:
        print(list(executor.map(square, [1, 2, 3, 4])))

Quick quiz

1. Why does multiprocessing code need the if __name__ == "__main__": guard?

  • A. It makes the code faster
  • B. New processes may import the main module, and without the guard they would start processes of their own
  • C. It is only a style rule
  • D. It enables the GIL
Show answer

B. This applies where processes are started by spawning, as on Windows and macOS.

2. Do separate processes share ordinary Python variables?

  • A. Yes
  • B. Only global variables
  • C. No; each process has its own memory
  • D. Only lists
Show answer

C. Data is passed between them through queues, pipes or special shared objects.

3. Which kind of work benefits most from multiprocessing?

  • A. Waiting for network responses
  • B. Printing to the screen
  • C. Reading a small file
  • D. CPU-bound calculations
Show answer

D. Each process runs on its own core, unaffected by the GIL of the others.

Interview questions

When would you use multiprocessing instead of multithreading?

Show answer

For CPU-bound work, where the program is busy calculating. Because of the GIL, threads in CPython cannot run Python code in parallel, but each process has its own interpreter and its own GIL, so processes use several cores at once. For I/O-bound work, threads or asyncio are better, since processes cost more to start and to communicate with.

How do processes communicate with each other?

Show answer

They do not share memory, so data is sent between them. A multiprocessing.Queue or Pipe passes objects, which are serialised with pickle on the way; the arguments and results of an executor's map travel the same way. For simple shared values there are Value and Array, used with a lock. Everything sent must be picklable.

Key Points to Remember

  • Processes bypass the GIL and run CPU-bound work in parallel on multiple cores.
  • ProcessPoolExecutor / Pool distribute work; APIs mirror thread pools.
  • Always guard process creation with if __name__ == "__main__".
  • Data is pickled between processes; use Queue/Pipe/Value/Manager for communication.
  • Parallelise substantial tasks; consider NumPy, Dask or PySpark for heavy data work.

Practice in the playground

Write and run your own code for this topic, or start from the exercise or an example. Runs Python in your browser. Your code for each topic is kept in this browser.

Run the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

These examples need a project environment on your computer, so they can be copied but not run here. Codelab runs Python, FastAPI backends with SQLite, plain Java 17 and HTML/CSS/JavaScript.

WebNest Studio

Written by

WebNest Studio Engineering Team

Practical lessons from developers working with Java, Python, React, databases and production systems.

Led by Vansh Duggal, Founder, WebNest Studio · Full-stack & software engineering