Course topics

By WebNest Studio Engineering Team

Python Tutorial

Model Evaluation, Cross-Validation and Hyperparameter Tuning

A single train/test split gives one noisy estimate: a lucky or unlucky split can change the score by several points. Cross-validation gives a more reliable estimate by training and testing several times on different splits. Models also have hyperparameters — settings you choose before training, such as a tree's maximum depth or the number of neighbours — and choosing them well can make a big difference.

This lesson covers k-fold cross-validation, choosing metrics, grid and randomised search, the three-way split between training, validation and test data, and ROC AUC.

Why it matters

A single train-test split can make a model look better or worse than it is, purely by chance. Cross-validation gives a more reliable estimate, and a systematic search finds good settings for a model without guesswork. These are the techniques that separate a result you can trust from one that was simply lucky.

Cross-Validation

In k-fold cross-validation the data is split into k parts (folds); the model is trained on k−1 folds and tested on the remaining one, k times, and the scores are averaged. cross_val_score(model, X, y, cv=5, scoring="f1") does it in one line; StratifiedKFold keeps class proportions in every fold and is the default for classifiers. Report the mean and the standard deviation. For time-ordered data use TimeSeriesSplit so the model never trains on the future.

Hyperparameter Search

GridSearchCV tries every combination in a grid of settings with cross-validation and keeps the best; RandomizedSearchCV samples combinations and is better when the grid is large. With pipelines, name parameters as step__parameter (e.g. model__max_depth). Tune on the training data only and keep the test set untouched until the very end — otherwise you are tuning to the test set and your final score is optimistic.

ROC and AUC

For probabilistic classifiers the ROC curve plots the true positive rate against the false positive rate at every threshold, and the AUC (area under the curve) summarises it: 0.5 is random guessing, 1.0 is perfect. AUC measures how well the model ranks positives above negatives regardless of the threshold, which makes it useful for comparing models.

Examples

Cross-validation versus a single split

Python
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))

single = []
for seed in range(5):                            # the single-split score depends on luck
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=seed, stratify=y)
    single.append(round(model.fit(X_tr, y_tr).score(X_te, y_te), 3))
print("single splits:", single)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(model, X, y, cv=cv)
print("5-fold scores:", scores.round(3))
print(f"accuracy = {scores.mean():.3f} +/- {scores.std():.3f}")
print("F1:", cross_val_score(model, X, y, cv=cv, scoring="f1").mean().round(3),
      "| ROC AUC:", cross_val_score(model, X, y, cv=cv, scoring="roc_auc").mean().round(3))
Output
single splits: [0.982, 0.991, 0.982, 0.974, 0.974]
5-fold scores: [0.956 0.974 0.982 1.    0.982]
accuracy = 0.979 +/- 0.014
F1: 0.983 | ROC AUC: 0.995

Tuning a pipeline with GridSearchCV and a final test

Python
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import GridSearchCV, train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

grid = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid={"n_estimators": [100, 300], "max_depth": [3, 6, None], "min_samples_leaf": [1, 5]},
    cv=5,
    scoring="roc_auc",
    n_jobs=-1,
)
grid.fit(X_train, y_train)                        # 12 combinations x 5 folds = 60 fits
print("best parameters:", grid.best_params_)
print("best CV ROC AUC:", round(grid.best_score_, 4))

best = grid.best_estimator_                       # already refitted on all training data
print("test ROC AUC   :", round(roc_auc_score(y_test, best.predict_proba(X_test)[:, 1]), 4))
print("test accuracy  :", round(best.score(X_test, y_test), 3))
Output
best parameters: {'max_depth': 6, 'min_samples_leaf': 5, 'n_estimators': 300}
best CV ROC AUC: 0.9889
test ROC AUC   : 0.9917
test accuracy  : 0.956

Validation curve: finding the sweet spot between under- and overfitting

Python
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import validation_curve
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
k_values = [1, 3, 5, 9, 15, 25, 51]
train_scores, valid_scores = validation_curve(
    make_pipeline(StandardScaler(), KNeighborsClassifier()), X, y,
    param_name="kneighborsclassifier__n_neighbors", param_range=k_values, cv=5)

print("  k  train  valid")
for k, tr, va in zip(k_values, train_scores.mean(axis=1), valid_scores.mean(axis=1)):
    print(f"{k:>3}  {tr:.3f}  {va:.3f}")
# k=1 memorises (train 1.000); very large k underfits; the best validation score lies in between.
Output
  k  train  valid
  1  1.000  0.954
  3  0.982  0.960
  5  0.974  0.965
  9  0.971  0.967
 15  0.967  0.961
 25  0.956  0.953
 51  0.951  0.951

Common Mistakes

  • Tuning hyperparameters on the test set and then reporting the test score as unbiased.
  • Reporting a single split score without cross-validation or a standard deviation.
  • Using ordinary k-fold on time series, letting the model peek at the future.
  • Huge grids that take hours when RandomizedSearchCV would find a good setting faster.
  • Optimising a metric that does not reflect the business goal.

Exercise

Using the cleanly separated data below, print the mean accuracy of a 1-nearest-neighbour classifier under 3-fold cross-validation. Then use GridSearchCV to choose between 1 and 3 neighbours, and print the best parameters it finds.

Expected output: 1.0 then {'n_neighbors': 1}

Python · starter code
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.neighbors import KNeighborsClassifier

x = [[1], [2], [3], [4], [5], [6], [10], [11], [12], [13], [14], [15]]
y = [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]

# TODO: run 3-fold cross-validation for KNeighborsClassifier(n_neighbors=1)
#       and print the mean of the scores

# TODO: grid search over n_neighbors 1 and 3 with cv=3, fit it,
#       and print the best parameters
Show hints
  • cross_val_score(model, x, y, cv=3) returns one score per fold.
  • The grid is a dictionary: {"n_neighbors": [1, 3]}. After fit, read search.best_params_. When settings tie, the first one tried is kept.
Show solution
Python · solution
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.neighbors import KNeighborsClassifier

x = [[1], [2], [3], [4], [5], [6], [10], [11], [12], [13], [14], [15]]
y = [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]

scores = cross_val_score(KNeighborsClassifier(n_neighbors=1), x, y, cv=3)
print(float(scores.mean()))

search = GridSearchCV(KNeighborsClassifier(), {"n_neighbors": [1, 3]}, cv=3)
search.fit(x, y)
print(search.best_params_)

Quick quiz

1. In 5-fold cross-validation, how many times is the model trained?

  • A. Once
  • B. Five times
  • C. Ten times
  • D. Twenty-five times
Show answer

B. Each fold is used once for validation while the other four are used for training.

2. What is a hyperparameter?

  • A. A value the model learns from the data
  • B. The target column
  • C. A setting chosen before training, such as the number of neighbours or the depth of a tree
  • D. A kind of metric
Show answer

C. Coefficients and weights are parameters learned in training; hyperparameters control how that learning happens.

3. After choosing hyperparameters with cross-validation, on what should the final performance be reported?

  • A. The training set
  • B. The validation folds
  • C. A test set that was never used during tuning
  • D. Any of the above
Show answer

C. The tuning has adapted to the validation data, so only untouched data gives an honest estimate.

Interview questions

What is cross-validation and why is it used?

Show answer

The data is divided into k parts. The model is trained k times, each time on k − 1 parts and validated on the remaining one, and the scores are averaged. Every example is used for validation exactly once, so the estimate depends far less on how one particular split happened to fall, and the spread of the scores shows how stable the model is. It costs k times the training time.

What is the difference between grid search and randomised search?

Show answer

Grid search tries every combination of the listed hyperparameter values, which is thorough but grows very quickly as more parameters are added. Randomised search tries a fixed number of combinations drawn at random from given ranges; it covers a large space at a set cost and often finds settings that are as good, because usually only a few parameters matter. Both evaluate each candidate with cross-validation.

Key Points to Remember

  • cross_val_score with (Stratified)KFold gives a reliable mean ± std estimate.
  • Hyperparameters are chosen, not learned; search them with GridSearchCV/RandomizedSearchCV.
  • In pipelines, refer to parameters as step__parameter.
  • Keep a final test set untouched until the very end.
  • ROC AUC compares classifiers independently of the threshold.

Practice in the playground

Write and run your own code for this topic, or start from the exercise or an example. Runs Python in your browser. Your code for each topic is kept in this browser.

Run the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

These examples need a project environment on your computer, so they can be copied but not run here. Codelab runs Python, FastAPI backends with SQLite, plain Java 17 and HTML/CSS/JavaScript.

WebNest Studio

Written by

WebNest Studio Engineering Team

Practical lessons from developers working with Java, Python, React, databases and production systems.

Led by Vansh Duggal, Founder, WebNest Studio · Full-stack & software engineering