By WebNest Studio Engineering Team
Python Tutorial
Data Preprocessing and Pipelines
Models need numbers without gaps, but real data has missing values, text categories and features on wildly different scales. Preprocessing fixes this: imputing missing values, encoding categories, scaling numbers. scikit-learn's Pipeline and ColumnTransformer chain these steps with the model into one object that is trained, evaluated, tuned and saved as a unit.
Pipelines also prevent data leakage — accidentally letting information from the test set influence training — one of the most common reasons models look great in development and fail in production.
Why it matters
Models need numbers on comparable scales with no gaps, and real data has text categories, missing values and columns that differ by orders of magnitude. Preprocessing fixes this, and a pipeline makes sure the same steps are applied in training and in prediction. Getting this wrong causes data leakage, the most common reason a model that looked excellent fails in production.
Transformers
SimpleImputer(strategy="median")— fill missing numbers;strategy="most_frequent"for categories.StandardScaler— rescale to mean 0 and standard deviation 1;MinMaxScaler— rescale to 0–1.OneHotEncoder(handle_unknown="ignore")— one 0/1 column per category; unseen categories at prediction time are safely ignored.OrdinalEncoder— for ordered categories such as small < medium < large.- Transformers learn from data with
fit(e.g. the median) and apply it withtransform.
Pipelines and Data Leakage
If you scale or impute using statistics of the whole dataset before splitting, the test set has leaked into training and your score is optimistic. A Pipeline fits every step only on the training data and reuses what it learned on the test data and new data. ColumnTransformer applies different steps to different columns (numeric vs categorical). The whole pipeline behaves like one estimator: fit, predict, score, cross-validation and grid search all work on it.
Examples
Encoding, scaling and imputing by hand (to see what happens)
import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
df = pd.DataFrame({"age": [25, 32, np.nan, 51], "plan": ["basic", "pro", "basic", "enterprise"]})
imputer = SimpleImputer(strategy="median")
ages = imputer.fit_transform(df[["age"]])
print("median learned:", imputer.statistics_, "->", ages.ravel())
scaler = StandardScaler()
print("scaled:", scaler.fit_transform(ages).ravel().round(2))
encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
print(encoder.fit_transform(df[["plan"]]))
print(encoder.get_feature_names_out())
print(encoder.transform(pd.DataFrame({"plan": ["premium"]}))) # unseen category -> all zeros
median learned: [32.] -> [25. 32. 32. 51.]
scaled: [-1.03 -0.31 -0.31 1.65]
[[1. 0. 0.]
[0. 0. 1.]
[1. 0. 0.]
[0. 1. 0.]]
['plan_basic' 'plan_enterprise' 'plan_pro']
[[0. 0. 0.]]
A full ColumnTransformer + Pipeline for customer churn
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
rng = np.random.default_rng(5)
n = 800
customers = pd.DataFrame({
"tenure_months": rng.integers(1, 72, n).astype(float),
"monthly_fee": rng.uniform(199, 1499, n).round(0),
"support_calls": rng.poisson(2, n),
"plan": rng.choice(["basic", "pro", "enterprise"], n, p=[0.5, 0.35, 0.15]),
"payment": rng.choice(["card", "upi", "invoice"], n),
})
risk = (-0.05 * customers["tenure_months"] + 0.4 * customers["support_calls"]
+ 0.001 * customers["monthly_fee"] + (customers["plan"] == "basic") * 0.8)
customers["churned"] = (risk + rng.normal(0, 0.8, n) > 0.9).astype(int)
customers.loc[rng.choice(n, 40, replace=False), "tenure_months"] = np.nan # missing values
print(customers.head(3))
print("churn rate:", customers["churned"].mean().round(3))
X = customers.drop(columns="churned")
y = customers["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
numeric = ["tenure_months", "monthly_fee", "support_calls"]
categorical = ["plan", "payment"]
preprocess = ColumnTransformer([
("num", Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())]), numeric),
("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])
for name, model in [("logistic", LogisticRegression(max_iter=1000)),
("forest", RandomForestClassifier(n_estimators=300, random_state=0))]:
pipeline = Pipeline([("prep", preprocess), ("model", model)])
pipeline.fit(X_train, y_train) # imputer/scaler learn from training data only
print(f"{name:<9} test accuracy = {pipeline.score(X_test, y_test):.3f}")
new_customer = pd.DataFrame([{"tenure_months": np.nan, "monthly_fee": 999, "support_calls": 6,
"plan": "basic", "payment": "invoice"}])
print("churn probability:", pipeline.predict_proba(new_customer)[0, 1].round(2))
print(list(pipeline.named_steps["prep"].get_feature_names_out())[:5])
tenure_months monthly_fee support_calls plan payment churned
0 48.0 361.0 1 pro upi 0
1 58.0 659.0 2 pro card 0
2 2.0 622.0 1 basic card 1
churn rate: 0.344
logistic test accuracy = 0.850
forest test accuracy = 0.815
churn probability: 0.77
['num__tenure_months', 'num__monthly_fee', 'num__support_calls', 'cat__plan_basic', 'cat__plan_enterprise']
Common Mistakes
- Scaling or imputing the whole dataset before train_test_split (data leakage).
- Label-encoding unordered categories as 0, 1, 2, implying an order that does not exist.
- Crashing in production on a new category because the encoder was not created with handle_unknown="ignore".
- Applying different preprocessing code in training and in production — save and reuse the pipeline.
- Scaling features for tree-based models, which do not need it (harmless, but unnecessary).
Exercise
Standardise the three values with StandardScaler and print them rounded to 2 decimals. One-hot encode the three colours and print the result. Then build a pipeline of a scaler and a 1-nearest-neighbour classifier, fit it, and print its predictions for 2 and 11.
Expected output: [-1.22, 0.0, 1.22], [[0.0, 1.0], [1.0, 0.0], [0.0, 1.0]], [0, 1] (one per line)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
values = [[1.0], [2.0], [3.0]]
# TODO: scale the values and print them as a flat list rounded to 2 decimals
colours = [["red"], ["blue"], ["red"]]
# TODO: one-hot encode the colours (dense output) and print the result as a list
x = [[1], [2], [3], [10], [11], [12]]
y = [0, 0, 0, 1, 1, 1]
# TODO: make a pipeline of StandardScaler and KNeighborsClassifier(n_neighbors=1),
# fit it and print the predictions for [[2], [11]] as a list
Show hints
fit_transformlearns the parameters and applies them.ravel()flattens a two-dimensional array.- Pass
sparse_output=FalsetoOneHotEncoderto get an ordinary array. Categories are ordered alphabetically, so blue comes first.
Show solution
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
values = [[1.0], [2.0], [3.0]]
scaled = StandardScaler().fit_transform(values)
print([round(float(value), 2) for value in scaled.ravel()])
colours = [["red"], ["blue"], ["red"]]
encoded = OneHotEncoder(sparse_output=False).fit_transform(colours)
print(encoded.tolist())
x = [[1], [2], [3], [10], [11], [12]]
y = [0, 0, 0, 1, 1, 1]
pipeline = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=1))
pipeline.fit(x, y)
print(pipeline.predict([[2], [11]]).tolist())
Quick quiz
1. What does StandardScaler do to a feature?
- A. Scales it to between 0 and 1
- B. Removes outliers
- C. Rescales it to a mean of 0 and a standard deviation of 1
- D. Converts it to text
Show answer
C. MinMaxScaler is the one that scales to a fixed range such as 0 to 1.
2. Why is one-hot encoding used for a category such as colour?
- A. To save memory
- B. To sort the data
- C. To remove rare values
- D. To turn categories into numbers without implying an order between them
Show answer
D. Coding red, green and blue as 1, 2 and 3 would suggest that blue is greater than red.
3. On which data should a scaler be fitted?
- A. The training set only
- B. The whole dataset
- C. The test set only
- D. It does not matter
Show answer
A. Fitting on all the data lets information from the test set leak into training.
Interview questions
What is data leakage?
Show answer
Leakage is when information that would not be available at prediction time is used in training, so the model's measured performance is better than it will be in real use. Common causes are fitting a scaler or imputer on the whole dataset before splitting, including a feature that is derived from the target, and letting future data into a time-based model. Splitting first and putting preprocessing inside a pipeline prevents the first kind.
Why use a scikit-learn Pipeline?
Show answer
A pipeline chains the preprocessing steps and the model into one object with a single fit and predict. The transformers are fitted only on the data passed to fit, so cross-validation and grid search apply them correctly within each fold and nothing leaks. The same steps are applied identically at prediction time, and the whole thing can be saved and deployed as one file.
Key Points to Remember
- Impute missing values, one-hot encode categories and scale numbers.
- fit learns from training data; transform applies it anywhere.
- ColumnTransformer sends each column group through its own steps.
- A Pipeline bundles preprocessing and model into one estimator and prevents leakage.
- The same fitted pipeline is used for evaluation and production predictions.
Practice in the playground
Write and run your own code for this topic, or start from the exercise or an example. Runs Python in your browser. Your code for each topic is kept in this browser.
Run the examples
Change an input, predict the result, then compare it with the output. Explain why the result changes.
These examples need a project environment on your computer, so they can be copied but not run here. Codelab runs Python, FastAPI backends with SQLite, plain Java 17 and HTML/CSS/JavaScript.
Written by
WebNest Studio Engineering Team
Practical lessons from developers working with Java, Python, React, databases and production systems.
Led by Vansh Duggal, Founder, WebNest Studio · Full-stack & software engineering