Model Evaluation, Cross-Validation and Hyperparameter Tuning

A single train/test split gives one noisy estimate: a lucky or unlucky split can change the score by several points. Cross-validation gives a more reliable estimate by training and testing several times on different splits. Models also have hyperparameters — settings you choose before training, such as a tree's maximum depth or the number of neighbours — and choosing them well can make a big difference.

This lesson covers k-fold cross-validation, choosing metrics, grid and randomised search, the three-way split between training, validation and test data, and ROC AUC.

Cross-Validation

In k-fold cross-validation the data is split into k parts (folds); the model is trained on k−1 folds and tested on the remaining one, k times, and the scores are averaged. cross_val_score(model, X, y, cv=5, scoring="f1") does it in one line; StratifiedKFold keeps class proportions in every fold and is the default for classifiers. Report the mean and the standard deviation. For time-ordered data use TimeSeriesSplit so the model never trains on the future.

Hyperparameter Search

GridSearchCV tries every combination in a grid of settings with cross-validation and keeps the best; RandomizedSearchCV samples combinations and is better when the grid is large. With pipelines, name parameters as step__parameter (e.g. model__max_depth). Tune on the training data only and keep the test set untouched until the very end — otherwise you are tuning to the test set and your final score is optimistic.

ROC and AUC

For probabilistic classifiers the ROC curve plots the true positive rate against the false positive rate at every threshold, and the AUC (area under the curve) summarises it: 0.5 is random guessing, 1.0 is perfect. AUC measures how well the model ranks positives above negatives regardless of the threshold, which makes it useful for comparing models.

Examples

Cross-validation versus a single split

Python

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))

single = []
for seed in range(5):                            # the single-split score depends on luck
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=seed, stratify=y)
    single.append(round(model.fit(X_tr, y_tr).score(X_te, y_te), 3))
print("single splits:", single)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(model, X, y, cv=cv)
print("5-fold scores:", scores.round(3))
print(f"accuracy = {scores.mean():.3f} +/- {scores.std():.3f}")
print("F1:", cross_val_score(model, X, y, cv=cv, scoring="f1").mean().round(3),
      "| ROC AUC:", cross_val_score(model, X, y, cv=cv, scoring="roc_auc").mean().round(3))

Output

single splits: [0.982, 0.991, 0.982, 0.974, 0.974]
5-fold scores: [0.956 0.974 0.982 1.    0.982]
accuracy = 0.979 +/- 0.014
F1: 0.983 | ROC AUC: 0.995

Tuning a pipeline with GridSearchCV and a final test

Python

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import GridSearchCV, train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

grid = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid={"n_estimators": [100, 300], "max_depth": [3, 6, None], "min_samples_leaf": [1, 5]},
    cv=5,
    scoring="roc_auc",
    n_jobs=-1,
)
grid.fit(X_train, y_train)                        # 12 combinations x 5 folds = 60 fits
print("best parameters:", grid.best_params_)
print("best CV ROC AUC:", round(grid.best_score_, 4))

best = grid.best_estimator_                       # already refitted on all training data
print("test ROC AUC   :", round(roc_auc_score(y_test, best.predict_proba(X_test)[:, 1]), 4))
print("test accuracy  :", round(best.score(X_test, y_test), 3))

Output

best parameters: {'max_depth': 6, 'min_samples_leaf': 5, 'n_estimators': 300}
best CV ROC AUC: 0.9889
test ROC AUC   : 0.9917
test accuracy  : 0.956

Validation curve: finding the sweet spot between under- and overfitting

Python

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import validation_curve
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
k_values = [1, 3, 5, 9, 15, 25, 51]
train_scores, valid_scores = validation_curve(
    make_pipeline(StandardScaler(), KNeighborsClassifier()), X, y,
    param_name="kneighborsclassifier__n_neighbors", param_range=k_values, cv=5)

print("  k  train  valid")
for k, tr, va in zip(k_values, train_scores.mean(axis=1), valid_scores.mean(axis=1)):
    print(f"{k:>3}  {tr:.3f}  {va:.3f}")
# k=1 memorises (train 1.000); very large k underfits; the best validation score lies in between.

Output

  k  train  valid
  1  1.000  0.954
  3  0.982  0.960
  5  0.974  0.965
  9  0.971  0.967
 15  0.967  0.961
 25  0.956  0.953
 51  0.951  0.951

Common Mistakes

Tuning hyperparameters on the test set and then reporting the test score as unbiased.
Reporting a single split score without cross-validation or a standard deviation.
Using ordinary k-fold on time series, letting the model peek at the future.
Huge grids that take hours when RandomizedSearchCV would find a good setting faster.
Optimising a metric that does not reflect the business goal.

Key Points to Remember

cross_val_score with (Stratified)KFold gives a reliable mean ± std estimate.
Hyperparameters are chosen, not learned; search them with GridSearchCV/RandomizedSearchCV.
In pipelines, refer to parameters as step__parameter.
Keep a final test set untouched until the very end.
ROC AUC compares classifiers independently of the threshold.

Practice the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

Use your local project environment for these examples. Codelab currently runs Python and HTML/CSS/JavaScript; framework examples may need project dependencies.