By WebNest Studio Engineering Team
Python Tutorial
Model Evaluation, Cross-Validation and Hyperparameter Tuning
A single train/test split gives one noisy estimate: a lucky or unlucky split can change the score by several points. Cross-validation gives a more reliable estimate by training and testing several times on different splits. Models also have hyperparameters — settings you choose before training, such as a tree's maximum depth or the number of neighbours — and choosing them well can make a big difference.
This lesson covers k-fold cross-validation, choosing metrics, grid and randomised search, the three-way split between training, validation and test data, and ROC AUC.
Why it matters
A single train-test split can make a model look better or worse than it is, purely by chance. Cross-validation gives a more reliable estimate, and a systematic search finds good settings for a model without guesswork. These are the techniques that separate a result you can trust from one that was simply lucky.
Cross-Validation
In k-fold cross-validation the data is split into k parts (folds); the model is trained on k−1 folds and tested on the remaining one, k times, and the scores are averaged. cross_val_score(model, X, y, cv=5, scoring="f1") does it in one line; StratifiedKFold keeps class proportions in every fold and is the default for classifiers. Report the mean and the standard deviation. For time-ordered data use TimeSeriesSplit so the model never trains on the future.
Hyperparameter Search
GridSearchCV tries every combination in a grid of settings with cross-validation and keeps the best; RandomizedSearchCV samples combinations and is better when the grid is large. With pipelines, name parameters as step__parameter (e.g. model__max_depth). Tune on the training data only and keep the test set untouched until the very end — otherwise you are tuning to the test set and your final score is optimistic.
ROC and AUC
For probabilistic classifiers the ROC curve plots the true positive rate against the false positive rate at every threshold, and the AUC (area under the curve) summarises it: 0.5 is random guessing, 1.0 is perfect. AUC measures how well the model ranks positives above negatives regardless of the threshold, which makes it useful for comparing models.
Examples
Cross-validation versus a single split
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
single = []
for seed in range(5): # the single-split score depends on luck
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=seed, stratify=y)
single.append(round(model.fit(X_tr, y_tr).score(X_te, y_te), 3))
print("single splits:", single)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(model, X, y, cv=cv)
print("5-fold scores:", scores.round(3))
print(f"accuracy = {scores.mean():.3f} +/- {scores.std():.3f}")
print("F1:", cross_val_score(model, X, y, cv=cv, scoring="f1").mean().round(3),
"| ROC AUC:", cross_val_score(model, X, y, cv=cv, scoring="roc_auc").mean().round(3))
single splits: [0.982, 0.991, 0.982, 0.974, 0.974]
5-fold scores: [0.956 0.974 0.982 1. 0.982]
accuracy = 0.979 +/- 0.014
F1: 0.983 | ROC AUC: 0.995
Tuning a pipeline with GridSearchCV and a final test
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import GridSearchCV, train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
grid = GridSearchCV(
RandomForestClassifier(random_state=42),
param_grid={"n_estimators": [100, 300], "max_depth": [3, 6, None], "min_samples_leaf": [1, 5]},
cv=5,
scoring="roc_auc",
n_jobs=-1,
)
grid.fit(X_train, y_train) # 12 combinations x 5 folds = 60 fits
print("best parameters:", grid.best_params_)
print("best CV ROC AUC:", round(grid.best_score_, 4))
best = grid.best_estimator_ # already refitted on all training data
print("test ROC AUC :", round(roc_auc_score(y_test, best.predict_proba(X_test)[:, 1]), 4))
print("test accuracy :", round(best.score(X_test, y_test), 3))
best parameters: {'max_depth': 6, 'min_samples_leaf': 5, 'n_estimators': 300}
best CV ROC AUC: 0.9889
test ROC AUC : 0.9917
test accuracy : 0.956
Validation curve: finding the sweet spot between under- and overfitting
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import validation_curve
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
k_values = [1, 3, 5, 9, 15, 25, 51]
train_scores, valid_scores = validation_curve(
make_pipeline(StandardScaler(), KNeighborsClassifier()), X, y,
param_name="kneighborsclassifier__n_neighbors", param_range=k_values, cv=5)
print(" k train valid")
for k, tr, va in zip(k_values, train_scores.mean(axis=1), valid_scores.mean(axis=1)):
print(f"{k:>3} {tr:.3f} {va:.3f}")
# k=1 memorises (train 1.000); very large k underfits; the best validation score lies in between.
k train valid
1 1.000 0.954
3 0.982 0.960
5 0.974 0.965
9 0.971 0.967
15 0.967 0.961
25 0.956 0.953
51 0.951 0.951
Common Mistakes
- Tuning hyperparameters on the test set and then reporting the test score as unbiased.
- Reporting a single split score without cross-validation or a standard deviation.
- Using ordinary k-fold on time series, letting the model peek at the future.
- Huge grids that take hours when RandomizedSearchCV would find a good setting faster.
- Optimising a metric that does not reflect the business goal.
Exercise
Using the cleanly separated data below, print the mean accuracy of a 1-nearest-neighbour classifier under 3-fold cross-validation. Then use GridSearchCV to choose between 1 and 3 neighbours, and print the best parameters it finds.
Expected output: 1.0 then {'n_neighbors': 1}
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
x = [[1], [2], [3], [4], [5], [6], [10], [11], [12], [13], [14], [15]]
y = [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
# TODO: run 3-fold cross-validation for KNeighborsClassifier(n_neighbors=1)
# and print the mean of the scores
# TODO: grid search over n_neighbors 1 and 3 with cv=3, fit it,
# and print the best parameters
Show hints
cross_val_score(model, x, y, cv=3)returns one score per fold.- The grid is a dictionary:
{"n_neighbors": [1, 3]}. Afterfit, readsearch.best_params_. When settings tie, the first one tried is kept.
Show solution
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
x = [[1], [2], [3], [4], [5], [6], [10], [11], [12], [13], [14], [15]]
y = [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
scores = cross_val_score(KNeighborsClassifier(n_neighbors=1), x, y, cv=3)
print(float(scores.mean()))
search = GridSearchCV(KNeighborsClassifier(), {"n_neighbors": [1, 3]}, cv=3)
search.fit(x, y)
print(search.best_params_)
Quick quiz
1. In 5-fold cross-validation, how many times is the model trained?
- A. Once
- B. Five times
- C. Ten times
- D. Twenty-five times
Show answer
B. Each fold is used once for validation while the other four are used for training.
2. What is a hyperparameter?
- A. A value the model learns from the data
- B. The target column
- C. A setting chosen before training, such as the number of neighbours or the depth of a tree
- D. A kind of metric
Show answer
C. Coefficients and weights are parameters learned in training; hyperparameters control how that learning happens.
3. After choosing hyperparameters with cross-validation, on what should the final performance be reported?
- A. The training set
- B. The validation folds
- C. A test set that was never used during tuning
- D. Any of the above
Show answer
C. The tuning has adapted to the validation data, so only untouched data gives an honest estimate.
Interview questions
What is cross-validation and why is it used?
Show answer
The data is divided into k parts. The model is trained k times, each time on k − 1 parts and validated on the remaining one, and the scores are averaged. Every example is used for validation exactly once, so the estimate depends far less on how one particular split happened to fall, and the spread of the scores shows how stable the model is. It costs k times the training time.
What is the difference between grid search and randomised search?
Show answer
Grid search tries every combination of the listed hyperparameter values, which is thorough but grows very quickly as more parameters are added. Randomised search tries a fixed number of combinations drawn at random from given ranges; it covers a large space at a set cost and often finds settings that are as good, because usually only a few parameters matter. Both evaluate each candidate with cross-validation.
Key Points to Remember
- cross_val_score with (Stratified)KFold gives a reliable mean ± std estimate.
- Hyperparameters are chosen, not learned; search them with GridSearchCV/RandomizedSearchCV.
- In pipelines, refer to parameters as step__parameter.
- Keep a final test set untouched until the very end.
- ROC AUC compares classifiers independently of the threshold.
Practice in the playground
Write and run your own code for this topic, or start from the exercise or an example. Runs Python in your browser. Your code for each topic is kept in this browser.
Run the examples
Change an input, predict the result, then compare it with the output. Explain why the result changes.
These examples need a project environment on your computer, so they can be copied but not run here. Codelab runs Python, FastAPI backends with SQLite, plain Java 17 and HTML/CSS/JavaScript.
Written by
WebNest Studio Engineering Team
Practical lessons from developers working with Java, Python, React, databases and production systems.
Led by Vansh Duggal, Founder, WebNest Studio · Full-stack & software engineering