Classification: Predicting Categories

Classification predicts a category: will this customer churn, is this email spam, is this tumour benign or malignant, which digit is in this image? This lesson uses scikit-learn's built-in breast cancer dataset (569 tumours, 30 measurements, labelled malignant or benign) to train and compare the most common classifiers — logistic regression, k-nearest neighbours, decision trees and random forests — and to evaluate them properly with the confusion matrix, precision, recall and F1, and probability thresholds.

Common Classifiers

LogisticRegression — despite its name, a linear classifier that outputs probabilities; fast, interpretable, a great baseline (scale the features).
KNeighborsClassifier — predicts the majority class of the k most similar training examples (needs scaling).
DecisionTreeClassifier — a flowchart of yes/no questions; easy to explain but overfits unless limited.
RandomForestClassifier / HistGradientBoostingClassifier — ensembles of many trees; usually the most accurate on tabular data.
SVC (support vector machine) and GaussianNB (naive Bayes, popular for text) are other classics.

Beyond Accuracy

Accuracy can mislead: if 95% of transactions are genuine, a model that always says "genuine" is 95% accurate and useless. The confusion matrix counts true positives, false positives, true negatives and false negatives. Precision = of the cases predicted positive, how many were right; recall = of the real positives, how many we caught; F1 balances the two. Which matters more depends on the cost of each mistake — missing a cancer (false negative) is far worse than an extra test (false positive). predict_proba gives probabilities, and moving the decision threshold trades precision for recall.

Examples

Comparing four classifiers

Python

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

data = load_breast_cancer(as_frame=True)
X, y = data.data, data.target                    # target: 0 = malignant, 1 = benign
print(X.shape, dict(zip(data.target_names.tolist(), y.value_counts().sort_index().tolist())))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

models = {
    "Logistic regression": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
    "k-nearest neighbours": make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=7)),
    "Decision tree": DecisionTreeClassifier(max_depth=4, random_state=42),
    "Random forest": RandomForestClassifier(n_estimators=300, random_state=42),
}
for name, model in models.items():
    model.fit(X_train, y_train)
    print(f"{name:<22} train={model.score(X_train, y_train):.3f}  test={model.score(X_test, y_test):.3f}")

Output

(569, 30) {'malignant': 212, 'benign': 357}
Logistic regression    train=0.988  test=0.986
k-nearest neighbours   train=0.974  test=0.979
Decision tree          train=0.988  test=0.944
Random forest          train=1.000  test=0.958

Confusion matrix, precision, recall and the classification report

Python

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.25, random_state=42, stratify=data.target)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)
predicted = model.predict(X_test)

print(confusion_matrix(y_test, predicted))      # rows: actual, columns: predicted
print(classification_report(y_test, predicted, target_names=data.target_names, digits=3))

Output

[[52  1]
 [ 1 89]]
              precision    recall  f1-score   support

   malignant      0.981     0.981     0.981        53
      benign      0.989     0.989     0.989        90

    accuracy                          0.986       143
   macro avg      0.985     0.985     0.985       143
weighted avg      0.986     0.986     0.986       143

Probabilities and moving the decision threshold

Python

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
y = 1 - data.target                              # make "malignant" the positive class (1)
X_train, X_test, y_train, y_test = train_test_split(data.data, y, test_size=0.25, random_state=42, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)

probabilities = model.predict_proba(X_test)[:, 1]       # probability of malignant
print("first five probabilities:", probabilities[:5].round(3))

print("threshold  precision  recall  flagged")
for threshold in [0.5, 0.3, 0.1]:
    flagged = (probabilities >= threshold).astype(int)
    print(f"{threshold:>9}  {precision_score(y_test, flagged):>9.3f}  {recall_score(y_test, flagged):>6.3f}  {flagged.sum():>7}")
# A lower threshold catches more malignant cases (higher recall) at the cost of more false alarms.

Output

first five probabilities: [0.999 0.004 1.    0.    0.035]
threshold  precision  recall  flagged
      0.5      0.980   0.925       50
      0.3      0.981   0.962       52
      0.1      0.897   0.981       58

Common Mistakes

Judging a classifier by accuracy on imbalanced data.
Not scaling features for logistic regression, KNN and SVMs.
Forgetting stratify=y, so rare classes are missing from the test set.
Always using the default 0.5 threshold when mistakes have very different costs.
Letting an unlimited decision tree memorise the training set.

Key Points to Remember

Classification predicts categories; start with logistic regression as a baseline.
Tree ensembles (random forest, gradient boosting) are strong on tabular data.
Use the confusion matrix, precision, recall and F1 — not just accuracy.
predict_proba plus a chosen threshold matches the model to the real costs of errors.
Scale features for distance- and gradient-based models.

Practice the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

Use your local project environment for these examples. Codelab currently runs Python and HTML/CSS/JavaScript; framework examples may need project dependencies.