By WebNest Studio Engineering Team
Python Tutorial
Classification: Predicting Categories
Classification predicts a category: will this customer churn, is this email spam, is this tumour benign or malignant, which digit is in this image? This lesson uses scikit-learn's built-in breast cancer dataset (569 tumours, 30 measurements, labelled malignant or benign) to train and compare the most common classifiers — logistic regression, k-nearest neighbours, decision trees and random forests — and to evaluate them properly with the confusion matrix, precision, recall and F1, and probability thresholds.
Why it matters
Is this transaction fraudulent, will this customer cancel, which category does this document belong to? Predicting a category is the most common machine-learning task in business. Knowing how to judge a classifier properly matters as much as training it, because accuracy alone can be badly misleading.
Common Classifiers
- LogisticRegression — despite its name, a linear classifier that outputs probabilities; fast, interpretable, a great baseline (scale the features).
- KNeighborsClassifier — predicts the majority class of the k most similar training examples (needs scaling).
- DecisionTreeClassifier — a flowchart of yes/no questions; easy to explain but overfits unless limited.
- RandomForestClassifier / HistGradientBoostingClassifier — ensembles of many trees; usually the most accurate on tabular data.
- SVC (support vector machine) and GaussianNB (naive Bayes, popular for text) are other classics.
Beyond Accuracy
Accuracy can mislead: if 95% of transactions are genuine, a model that always says "genuine" is 95% accurate and useless. The confusion matrix counts true positives, false positives, true negatives and false negatives. Precision = of the cases predicted positive, how many were right; recall = of the real positives, how many we caught; F1 balances the two. Which matters more depends on the cost of each mistake — missing a cancer (false negative) is far worse than an extra test (false positive). predict_proba gives probabilities, and moving the decision threshold trades precision for recall.
Examples
Comparing four classifiers
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
data = load_breast_cancer(as_frame=True)
X, y = data.data, data.target # target: 0 = malignant, 1 = benign
print(X.shape, dict(zip(data.target_names.tolist(), y.value_counts().sort_index().tolist())))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
models = {
"Logistic regression": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
"k-nearest neighbours": make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=7)),
"Decision tree": DecisionTreeClassifier(max_depth=4, random_state=42),
"Random forest": RandomForestClassifier(n_estimators=300, random_state=42),
}
for name, model in models.items():
model.fit(X_train, y_train)
print(f"{name:<22} train={model.score(X_train, y_train):.3f} test={model.score(X_test, y_test):.3f}")
(569, 30) {'malignant': 212, 'benign': 357}
Logistic regression train=0.988 test=0.986
k-nearest neighbours train=0.974 test=0.979
Decision tree train=0.988 test=0.944
Random forest train=1.000 test=0.958
Confusion matrix, precision, recall and the classification report
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
data.data, data.target, test_size=0.25, random_state=42, stratify=data.target)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)
predicted = model.predict(X_test)
print(confusion_matrix(y_test, predicted)) # rows: actual, columns: predicted
print(classification_report(y_test, predicted, target_names=data.target_names, digits=3))
[[52 1]
[ 1 89]]
precision recall f1-score support
malignant 0.981 0.981 0.981 53
benign 0.989 0.989 0.989 90
accuracy 0.986 143
macro avg 0.985 0.985 0.985 143
weighted avg 0.986 0.986 0.986 143
Probabilities and moving the decision threshold
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
data = load_breast_cancer()
y = 1 - data.target # make "malignant" the positive class (1)
X_train, X_test, y_train, y_test = train_test_split(data.data, y, test_size=0.25, random_state=42, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)
probabilities = model.predict_proba(X_test)[:, 1] # probability of malignant
print("first five probabilities:", probabilities[:5].round(3))
print("threshold precision recall flagged")
for threshold in [0.5, 0.3, 0.1]:
flagged = (probabilities >= threshold).astype(int)
print(f"{threshold:>9} {precision_score(y_test, flagged):>9.3f} {recall_score(y_test, flagged):>6.3f} {flagged.sum():>7}")
# A lower threshold catches more malignant cases (higher recall) at the cost of more false alarms.
first five probabilities: [0.999 0.004 1. 0. 0.035]
threshold precision recall flagged
0.5 0.980 0.925 50
0.3 0.981 0.962 52
0.1 0.897 0.981 58
Common Mistakes
- Judging a classifier by accuracy on imbalanced data.
- Not scaling features for logistic regression, KNN and SVMs.
- Forgetting stratify=y, so rare classes are missing from the test set.
- Always using the default 0.5 threshold when mistakes have very different costs.
- Letting an unlimited decision tree memorise the training set.
Exercise
Train a k-nearest-neighbours classifier with 3 neighbours on the data, in which class 0 has small values and class 1 has large ones. Print the predicted classes for 2.5 and 10.5 as a list, and then the confusion matrix for the training data as a list of lists.
Expected output: [0, 1] then [[3, 0], [0, 3]]
from sklearn.metrics import confusion_matrix
from sklearn.neighbors import KNeighborsClassifier
x = [[1], [2], [3], [10], [11], [12]]
y = [0, 0, 0, 1, 1, 1]
# TODO: create a KNeighborsClassifier with 3 neighbours and fit it
# TODO: print the predictions for [[2.5], [10.5]] as a list
# TODO: print the confusion matrix of y against the predictions for x, as a list
Show hints
- The number of neighbours is set with
KNeighborsClassifier(n_neighbors=3). predictandconfusion_matrixreturn arrays;tolist()converts them.
Show solution
from sklearn.metrics import confusion_matrix
from sklearn.neighbors import KNeighborsClassifier
x = [[1], [2], [3], [10], [11], [12]]
y = [0, 0, 0, 1, 1, 1]
model = KNeighborsClassifier(n_neighbors=3)
model.fit(x, y)
print(model.predict([[2.5], [10.5]]).tolist())
print(confusion_matrix(y, model.predict(x)).tolist())
Quick quiz
1. Of 1,000 transactions, 10 are fraudulent. A model that predicts "not fraud" every time has what accuracy?
- A. 10%
- B. 50%
- C. 0%
- D. 99%
Show answer
D. It is 99% accurate and finds no fraud at all, which is why accuracy is misleading for unbalanced classes.
2. What does recall measure?
- A. Of the actual positives, how many the model found
- B. Of the predicted positives, how many are correct
- C. The share of all predictions that are correct
- D. The training time
Show answer
A. Precision is the share of predicted positives that are correct. Recall matters when missing a positive is costly.
3. What does a confusion matrix show?
- A. The features used
- B. The counts of correct and incorrect predictions for each class
- C. The training loss
- D. The correlation between features
Show answer
B. It separates true positives, false positives, true negatives and false negatives.
Interview questions
What is the difference between precision and recall?
Show answer
Precision is the proportion of positive predictions that are correct: of the emails marked as spam, how many really are spam. Recall is the proportion of actual positives that were found: of all the spam, how much was caught. Raising one usually lowers the other. Precision matters most when a false alarm is costly; recall when a miss is costly, as in disease screening. The F1 score combines the two.
How do you deal with an imbalanced dataset?
Show answer
Stop relying on accuracy and use precision, recall, F1 or the area under the ROC curve. Keep the class proportions the same in the train and test sets with a stratified split. Then consider giving the rare class more weight in the model, oversampling it or undersampling the common class within the training data only, and adjusting the decision threshold to suit the cost of each kind of error.
Key Points to Remember
- Classification predicts categories; start with logistic regression as a baseline.
- Tree ensembles (random forest, gradient boosting) are strong on tabular data.
- Use the confusion matrix, precision, recall and F1 — not just accuracy.
- predict_proba plus a chosen threshold matches the model to the real costs of errors.
- Scale features for distance- and gradient-based models.
Practice in the playground
Write and run your own code for this topic, or start from the exercise or an example. Runs Python in your browser. Your code for each topic is kept in this browser.
Run the examples
Change an input, predict the result, then compare it with the output. Explain why the result changes.
These examples need a project environment on your computer, so they can be copied but not run here. Codelab runs Python, FastAPI backends with SQLite, plain Java 17 and HTML/CSS/JavaScript.
Written by
WebNest Studio Engineering Team
Practical lessons from developers working with Java, Python, React, databases and production systems.
Led by Vansh Duggal, Founder, WebNest Studio · Full-stack & software engineering