Data Preprocessing and Pipelines

Models need numbers without gaps, but real data has missing values, text categories and features on wildly different scales. Preprocessing fixes this: imputing missing values, encoding categories, scaling numbers. scikit-learn's Pipeline and ColumnTransformer chain these steps with the model into one object that is trained, evaluated, tuned and saved as a unit.

Pipelines also prevent data leakage — accidentally letting information from the test set influence training — one of the most common reasons models look great in development and fail in production.

Transformers

SimpleImputer(strategy="median") — fill missing numbers; strategy="most_frequent" for categories.
StandardScaler — rescale to mean 0 and standard deviation 1; MinMaxScaler — rescale to 0–1.
OneHotEncoder(handle_unknown="ignore") — one 0/1 column per category; unseen categories at prediction time are safely ignored.
OrdinalEncoder — for ordered categories such as small < medium < large.
Transformers learn from data with fit (e.g. the median) and apply it with transform.

Pipelines and Data Leakage

If you scale or impute using statistics of the whole dataset before splitting, the test set has leaked into training and your score is optimistic. A Pipeline fits every step only on the training data and reuses what it learned on the test data and new data. ColumnTransformer applies different steps to different columns (numeric vs categorical). The whole pipeline behaves like one estimator: fit, predict, score, cross-validation and grid search all work on it.

Examples

Encoding, scaling and imputing by hand (to see what happens)

Python

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.DataFrame({"age": [25, 32, np.nan, 51], "plan": ["basic", "pro", "basic", "enterprise"]})

imputer = SimpleImputer(strategy="median")
ages = imputer.fit_transform(df[["age"]])
print("median learned:", imputer.statistics_, "->", ages.ravel())

scaler = StandardScaler()
print("scaled:", scaler.fit_transform(ages).ravel().round(2))

encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
print(encoder.fit_transform(df[["plan"]]))
print(encoder.get_feature_names_out())
print(encoder.transform(pd.DataFrame({"plan": ["premium"]})))     # unseen category -> all zeros

Output

median learned: [32.] -> [25. 32. 32. 51.]
scaled: [-1.03 -0.31 -0.31  1.65]
[[1. 0. 0.]
 [0. 0. 1.]
 [1. 0. 0.]
 [0. 1. 0.]]
['plan_basic' 'plan_enterprise' 'plan_pro']
[[0. 0. 0.]]

A full ColumnTransformer + Pipeline for customer churn

Python

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(5)
n = 800
customers = pd.DataFrame({
    "tenure_months": rng.integers(1, 72, n).astype(float),
    "monthly_fee": rng.uniform(199, 1499, n).round(0),
    "support_calls": rng.poisson(2, n),
    "plan": rng.choice(["basic", "pro", "enterprise"], n, p=[0.5, 0.35, 0.15]),
    "payment": rng.choice(["card", "upi", "invoice"], n),
})
risk = (-0.05 * customers["tenure_months"] + 0.4 * customers["support_calls"]
        + 0.001 * customers["monthly_fee"] + (customers["plan"] == "basic") * 0.8)
customers["churned"] = (risk + rng.normal(0, 0.8, n) > 0.9).astype(int)
customers.loc[rng.choice(n, 40, replace=False), "tenure_months"] = np.nan   # missing values
print(customers.head(3))
print("churn rate:", customers["churned"].mean().round(3))

X = customers.drop(columns="churned")
y = customers["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)

numeric = ["tenure_months", "monthly_fee", "support_calls"]
categorical = ["plan", "payment"]
preprocess = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())]), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])

for name, model in [("logistic", LogisticRegression(max_iter=1000)),
                    ("forest", RandomForestClassifier(n_estimators=300, random_state=0))]:
    pipeline = Pipeline([("prep", preprocess), ("model", model)])
    pipeline.fit(X_train, y_train)                    # imputer/scaler learn from training data only
    print(f"{name:<9} test accuracy = {pipeline.score(X_test, y_test):.3f}")

new_customer = pd.DataFrame([{"tenure_months": np.nan, "monthly_fee": 999, "support_calls": 6,
                              "plan": "basic", "payment": "invoice"}])
print("churn probability:", pipeline.predict_proba(new_customer)[0, 1].round(2))
print(list(pipeline.named_steps["prep"].get_feature_names_out())[:5])

Output

   tenure_months  monthly_fee  support_calls   plan payment  churned
0           48.0        361.0              1    pro     upi        0
1           58.0        659.0              2    pro    card        0
2            2.0        622.0              1  basic    card        1
churn rate: 0.344
logistic  test accuracy = 0.850
forest    test accuracy = 0.815
churn probability: 0.77
['num__tenure_months', 'num__monthly_fee', 'num__support_calls', 'cat__plan_basic', 'cat__plan_enterprise']

Common Mistakes

Scaling or imputing the whole dataset before train_test_split (data leakage).
Label-encoding unordered categories as 0, 1, 2, implying an order that does not exist.
Crashing in production on a new category because the encoder was not created with handle_unknown="ignore".
Applying different preprocessing code in training and in production — save and reuse the pipeline.
Scaling features for tree-based models, which do not need it (harmless, but unnecessary).

Key Points to Remember

Impute missing values, one-hot encode categories and scale numbers.
fit learns from training data; transform applies it anywhere.
ColumnTransformer sends each column group through its own steps.
A Pipeline bundles preprocessing and model into one estimator and prevents leakage.
The same fitted pipeline is used for evaluation and production predictions.

Practice the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

Use your local project environment for these examples. Codelab currently runs Python and HTML/CSS/JavaScript; framework examples may need project dependencies.