Course topics

By WebNest Studio Engineering Team

Python Tutorial

Classification: Predicting Categories

Classification predicts a category: will this customer churn, is this email spam, is this tumour benign or malignant, which digit is in this image? This lesson uses scikit-learn's built-in breast cancer dataset (569 tumours, 30 measurements, labelled malignant or benign) to train and compare the most common classifiers — logistic regression, k-nearest neighbours, decision trees and random forests — and to evaluate them properly with the confusion matrix, precision, recall and F1, and probability thresholds.

Why it matters

Is this transaction fraudulent, will this customer cancel, which category does this document belong to? Predicting a category is the most common machine-learning task in business. Knowing how to judge a classifier properly matters as much as training it, because accuracy alone can be badly misleading.

Common Classifiers

  • LogisticRegression — despite its name, a linear classifier that outputs probabilities; fast, interpretable, a great baseline (scale the features).
  • KNeighborsClassifier — predicts the majority class of the k most similar training examples (needs scaling).
  • DecisionTreeClassifier — a flowchart of yes/no questions; easy to explain but overfits unless limited.
  • RandomForestClassifier / HistGradientBoostingClassifier — ensembles of many trees; usually the most accurate on tabular data.
  • SVC (support vector machine) and GaussianNB (naive Bayes, popular for text) are other classics.

Beyond Accuracy

Accuracy can mislead: if 95% of transactions are genuine, a model that always says "genuine" is 95% accurate and useless. The confusion matrix counts true positives, false positives, true negatives and false negatives. Precision = of the cases predicted positive, how many were right; recall = of the real positives, how many we caught; F1 balances the two. Which matters more depends on the cost of each mistake — missing a cancer (false negative) is far worse than an extra test (false positive). predict_proba gives probabilities, and moving the decision threshold trades precision for recall.

Examples

Comparing four classifiers

Python
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

data = load_breast_cancer(as_frame=True)
X, y = data.data, data.target                    # target: 0 = malignant, 1 = benign
print(X.shape, dict(zip(data.target_names.tolist(), y.value_counts().sort_index().tolist())))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

models = {
    "Logistic regression": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
    "k-nearest neighbours": make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=7)),
    "Decision tree": DecisionTreeClassifier(max_depth=4, random_state=42),
    "Random forest": RandomForestClassifier(n_estimators=300, random_state=42),
}
for name, model in models.items():
    model.fit(X_train, y_train)
    print(f"{name:<22} train={model.score(X_train, y_train):.3f}  test={model.score(X_test, y_test):.3f}")
Output
(569, 30) {'malignant': 212, 'benign': 357}
Logistic regression    train=0.988  test=0.986
k-nearest neighbours   train=0.974  test=0.979
Decision tree          train=0.988  test=0.944
Random forest          train=1.000  test=0.958

Confusion matrix, precision, recall and the classification report

Python
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.25, random_state=42, stratify=data.target)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)
predicted = model.predict(X_test)

print(confusion_matrix(y_test, predicted))      # rows: actual, columns: predicted
print(classification_report(y_test, predicted, target_names=data.target_names, digits=3))
Output
[[52  1]
 [ 1 89]]
              precision    recall  f1-score   support

   malignant      0.981     0.981     0.981        53
      benign      0.989     0.989     0.989        90

    accuracy                          0.986       143
   macro avg      0.985     0.985     0.985       143
weighted avg      0.986     0.986     0.986       143

Probabilities and moving the decision threshold

Python
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
y = 1 - data.target                              # make "malignant" the positive class (1)
X_train, X_test, y_train, y_test = train_test_split(data.data, y, test_size=0.25, random_state=42, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)

probabilities = model.predict_proba(X_test)[:, 1]       # probability of malignant
print("first five probabilities:", probabilities[:5].round(3))

print("threshold  precision  recall  flagged")
for threshold in [0.5, 0.3, 0.1]:
    flagged = (probabilities >= threshold).astype(int)
    print(f"{threshold:>9}  {precision_score(y_test, flagged):>9.3f}  {recall_score(y_test, flagged):>6.3f}  {flagged.sum():>7}")
# A lower threshold catches more malignant cases (higher recall) at the cost of more false alarms.
Output
first five probabilities: [0.999 0.004 1.    0.    0.035]
threshold  precision  recall  flagged
      0.5      0.980   0.925       50
      0.3      0.981   0.962       52
      0.1      0.897   0.981       58

Common Mistakes

  • Judging a classifier by accuracy on imbalanced data.
  • Not scaling features for logistic regression, KNN and SVMs.
  • Forgetting stratify=y, so rare classes are missing from the test set.
  • Always using the default 0.5 threshold when mistakes have very different costs.
  • Letting an unlimited decision tree memorise the training set.

Exercise

Train a k-nearest-neighbours classifier with 3 neighbours on the data, in which class 0 has small values and class 1 has large ones. Print the predicted classes for 2.5 and 10.5 as a list, and then the confusion matrix for the training data as a list of lists.

Expected output: [0, 1] then [[3, 0], [0, 3]]

Python · starter code
from sklearn.metrics import confusion_matrix
from sklearn.neighbors import KNeighborsClassifier

x = [[1], [2], [3], [10], [11], [12]]
y = [0, 0, 0, 1, 1, 1]

# TODO: create a KNeighborsClassifier with 3 neighbours and fit it
# TODO: print the predictions for [[2.5], [10.5]] as a list
# TODO: print the confusion matrix of y against the predictions for x, as a list
Show hints
  • The number of neighbours is set with KNeighborsClassifier(n_neighbors=3).
  • predict and confusion_matrix return arrays; tolist() converts them.
Show solution
Python · solution
from sklearn.metrics import confusion_matrix
from sklearn.neighbors import KNeighborsClassifier

x = [[1], [2], [3], [10], [11], [12]]
y = [0, 0, 0, 1, 1, 1]

model = KNeighborsClassifier(n_neighbors=3)
model.fit(x, y)

print(model.predict([[2.5], [10.5]]).tolist())
print(confusion_matrix(y, model.predict(x)).tolist())

Quick quiz

1. Of 1,000 transactions, 10 are fraudulent. A model that predicts "not fraud" every time has what accuracy?

  • A. 10%
  • B. 50%
  • C. 0%
  • D. 99%
Show answer

D. It is 99% accurate and finds no fraud at all, which is why accuracy is misleading for unbalanced classes.

2. What does recall measure?

  • A. Of the actual positives, how many the model found
  • B. Of the predicted positives, how many are correct
  • C. The share of all predictions that are correct
  • D. The training time
Show answer

A. Precision is the share of predicted positives that are correct. Recall matters when missing a positive is costly.

3. What does a confusion matrix show?

  • A. The features used
  • B. The counts of correct and incorrect predictions for each class
  • C. The training loss
  • D. The correlation between features
Show answer

B. It separates true positives, false positives, true negatives and false negatives.

Interview questions

What is the difference between precision and recall?

Show answer

Precision is the proportion of positive predictions that are correct: of the emails marked as spam, how many really are spam. Recall is the proportion of actual positives that were found: of all the spam, how much was caught. Raising one usually lowers the other. Precision matters most when a false alarm is costly; recall when a miss is costly, as in disease screening. The F1 score combines the two.

How do you deal with an imbalanced dataset?

Show answer

Stop relying on accuracy and use precision, recall, F1 or the area under the ROC curve. Keep the class proportions the same in the train and test sets with a stratified split. Then consider giving the rare class more weight in the model, oversampling it or undersampling the common class within the training data only, and adjusting the decision threshold to suit the cost of each kind of error.

Key Points to Remember

  • Classification predicts categories; start with logistic regression as a baseline.
  • Tree ensembles (random forest, gradient boosting) are strong on tabular data.
  • Use the confusion matrix, precision, recall and F1 — not just accuracy.
  • predict_proba plus a chosen threshold matches the model to the real costs of errors.
  • Scale features for distance- and gradient-based models.

Practice in the playground

Write and run your own code for this topic, or start from the exercise or an example. Runs Python in your browser. Your code for each topic is kept in this browser.

Run the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

These examples need a project environment on your computer, so they can be copied but not run here. Codelab runs Python, FastAPI backends with SQLite, plain Java 17 and HTML/CSS/JavaScript.

WebNest Studio

Written by

WebNest Studio Engineering Team

Practical lessons from developers working with Java, Python, React, databases and production systems.

Led by Vansh Duggal, Founder, WebNest Studio · Full-stack & software engineering