Course topics

By WebNest Studio Engineering Team

Python Tutorial

Data Preprocessing and Pipelines

Models need numbers without gaps, but real data has missing values, text categories and features on wildly different scales. Preprocessing fixes this: imputing missing values, encoding categories, scaling numbers. scikit-learn's Pipeline and ColumnTransformer chain these steps with the model into one object that is trained, evaluated, tuned and saved as a unit.

Pipelines also prevent data leakage — accidentally letting information from the test set influence training — one of the most common reasons models look great in development and fail in production.

Why it matters

Models need numbers on comparable scales with no gaps, and real data has text categories, missing values and columns that differ by orders of magnitude. Preprocessing fixes this, and a pipeline makes sure the same steps are applied in training and in prediction. Getting this wrong causes data leakage, the most common reason a model that looked excellent fails in production.

Transformers

  • SimpleImputer(strategy="median") — fill missing numbers; strategy="most_frequent" for categories.
  • StandardScaler — rescale to mean 0 and standard deviation 1; MinMaxScaler — rescale to 0–1.
  • OneHotEncoder(handle_unknown="ignore") — one 0/1 column per category; unseen categories at prediction time are safely ignored.
  • OrdinalEncoder — for ordered categories such as small < medium < large.
  • Transformers learn from data with fit (e.g. the median) and apply it with transform.

Pipelines and Data Leakage

If you scale or impute using statistics of the whole dataset before splitting, the test set has leaked into training and your score is optimistic. A Pipeline fits every step only on the training data and reuses what it learned on the test data and new data. ColumnTransformer applies different steps to different columns (numeric vs categorical). The whole pipeline behaves like one estimator: fit, predict, score, cross-validation and grid search all work on it.

Examples

Encoding, scaling and imputing by hand (to see what happens)

Python
import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.DataFrame({"age": [25, 32, np.nan, 51], "plan": ["basic", "pro", "basic", "enterprise"]})

imputer = SimpleImputer(strategy="median")
ages = imputer.fit_transform(df[["age"]])
print("median learned:", imputer.statistics_, "->", ages.ravel())

scaler = StandardScaler()
print("scaled:", scaler.fit_transform(ages).ravel().round(2))

encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
print(encoder.fit_transform(df[["plan"]]))
print(encoder.get_feature_names_out())
print(encoder.transform(pd.DataFrame({"plan": ["premium"]})))     # unseen category -> all zeros
Output
median learned: [32.] -> [25. 32. 32. 51.]
scaled: [-1.03 -0.31 -0.31  1.65]
[[1. 0. 0.]
 [0. 0. 1.]
 [1. 0. 0.]
 [0. 1. 0.]]
['plan_basic' 'plan_enterprise' 'plan_pro']
[[0. 0. 0.]]

A full ColumnTransformer + Pipeline for customer churn

Python
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(5)
n = 800
customers = pd.DataFrame({
    "tenure_months": rng.integers(1, 72, n).astype(float),
    "monthly_fee": rng.uniform(199, 1499, n).round(0),
    "support_calls": rng.poisson(2, n),
    "plan": rng.choice(["basic", "pro", "enterprise"], n, p=[0.5, 0.35, 0.15]),
    "payment": rng.choice(["card", "upi", "invoice"], n),
})
risk = (-0.05 * customers["tenure_months"] + 0.4 * customers["support_calls"]
        + 0.001 * customers["monthly_fee"] + (customers["plan"] == "basic") * 0.8)
customers["churned"] = (risk + rng.normal(0, 0.8, n) > 0.9).astype(int)
customers.loc[rng.choice(n, 40, replace=False), "tenure_months"] = np.nan   # missing values
print(customers.head(3))
print("churn rate:", customers["churned"].mean().round(3))

X = customers.drop(columns="churned")
y = customers["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)

numeric = ["tenure_months", "monthly_fee", "support_calls"]
categorical = ["plan", "payment"]
preprocess = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())]), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])

for name, model in [("logistic", LogisticRegression(max_iter=1000)),
                    ("forest", RandomForestClassifier(n_estimators=300, random_state=0))]:
    pipeline = Pipeline([("prep", preprocess), ("model", model)])
    pipeline.fit(X_train, y_train)                    # imputer/scaler learn from training data only
    print(f"{name:<9} test accuracy = {pipeline.score(X_test, y_test):.3f}")

new_customer = pd.DataFrame([{"tenure_months": np.nan, "monthly_fee": 999, "support_calls": 6,
                              "plan": "basic", "payment": "invoice"}])
print("churn probability:", pipeline.predict_proba(new_customer)[0, 1].round(2))
print(list(pipeline.named_steps["prep"].get_feature_names_out())[:5])
Output
   tenure_months  monthly_fee  support_calls   plan payment  churned
0           48.0        361.0              1    pro     upi        0
1           58.0        659.0              2    pro    card        0
2            2.0        622.0              1  basic    card        1
churn rate: 0.344
logistic  test accuracy = 0.850
forest    test accuracy = 0.815
churn probability: 0.77
['num__tenure_months', 'num__monthly_fee', 'num__support_calls', 'cat__plan_basic', 'cat__plan_enterprise']

Common Mistakes

  • Scaling or imputing the whole dataset before train_test_split (data leakage).
  • Label-encoding unordered categories as 0, 1, 2, implying an order that does not exist.
  • Crashing in production on a new category because the encoder was not created with handle_unknown="ignore".
  • Applying different preprocessing code in training and in production — save and reuse the pipeline.
  • Scaling features for tree-based models, which do not need it (harmless, but unnecessary).

Exercise

Standardise the three values with StandardScaler and print them rounded to 2 decimals. One-hot encode the three colours and print the result. Then build a pipeline of a scaler and a 1-nearest-neighbour classifier, fit it, and print its predictions for 2 and 11.

Expected output: [-1.22, 0.0, 1.22], [[0.0, 1.0], [1.0, 0.0], [0.0, 1.0]], [0, 1] (one per line)

Python · starter code
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

values = [[1.0], [2.0], [3.0]]
# TODO: scale the values and print them as a flat list rounded to 2 decimals

colours = [["red"], ["blue"], ["red"]]
# TODO: one-hot encode the colours (dense output) and print the result as a list

x = [[1], [2], [3], [10], [11], [12]]
y = [0, 0, 0, 1, 1, 1]
# TODO: make a pipeline of StandardScaler and KNeighborsClassifier(n_neighbors=1),
#       fit it and print the predictions for [[2], [11]] as a list
Show hints
  • fit_transform learns the parameters and applies them. ravel() flattens a two-dimensional array.
  • Pass sparse_output=False to OneHotEncoder to get an ordinary array. Categories are ordered alphabetically, so blue comes first.
Show solution
Python · solution
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

values = [[1.0], [2.0], [3.0]]
scaled = StandardScaler().fit_transform(values)
print([round(float(value), 2) for value in scaled.ravel()])

colours = [["red"], ["blue"], ["red"]]
encoded = OneHotEncoder(sparse_output=False).fit_transform(colours)
print(encoded.tolist())

x = [[1], [2], [3], [10], [11], [12]]
y = [0, 0, 0, 1, 1, 1]
pipeline = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=1))
pipeline.fit(x, y)
print(pipeline.predict([[2], [11]]).tolist())

Quick quiz

1. What does StandardScaler do to a feature?

  • A. Scales it to between 0 and 1
  • B. Removes outliers
  • C. Rescales it to a mean of 0 and a standard deviation of 1
  • D. Converts it to text
Show answer

C. MinMaxScaler is the one that scales to a fixed range such as 0 to 1.

2. Why is one-hot encoding used for a category such as colour?

  • A. To save memory
  • B. To sort the data
  • C. To remove rare values
  • D. To turn categories into numbers without implying an order between them
Show answer

D. Coding red, green and blue as 1, 2 and 3 would suggest that blue is greater than red.

3. On which data should a scaler be fitted?

  • A. The training set only
  • B. The whole dataset
  • C. The test set only
  • D. It does not matter
Show answer

A. Fitting on all the data lets information from the test set leak into training.

Interview questions

What is data leakage?

Show answer

Leakage is when information that would not be available at prediction time is used in training, so the model's measured performance is better than it will be in real use. Common causes are fitting a scaler or imputer on the whole dataset before splitting, including a feature that is derived from the target, and letting future data into a time-based model. Splitting first and putting preprocessing inside a pipeline prevents the first kind.

Why use a scikit-learn Pipeline?

Show answer

A pipeline chains the preprocessing steps and the model into one object with a single fit and predict. The transformers are fitted only on the data passed to fit, so cross-validation and grid search apply them correctly within each fold and nothing leaks. The same steps are applied identically at prediction time, and the whole thing can be saved and deployed as one file.

Key Points to Remember

  • Impute missing values, one-hot encode categories and scale numbers.
  • fit learns from training data; transform applies it anywhere.
  • ColumnTransformer sends each column group through its own steps.
  • A Pipeline bundles preprocessing and model into one estimator and prevents leakage.
  • The same fitted pipeline is used for evaluation and production predictions.

Practice in the playground

Write and run your own code for this topic, or start from the exercise or an example. Runs Python in your browser. Your code for each topic is kept in this browser.

Run the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

These examples need a project environment on your computer, so they can be copied but not run here. Codelab runs Python, FastAPI backends with SQLite, plain Java 17 and HTML/CSS/JavaScript.

WebNest Studio

Written by

WebNest Studio Engineering Team

Practical lessons from developers working with Java, Python, React, databases and production systems.

Led by Vansh Duggal, Founder, WebNest Studio · Full-stack & software engineering