Regression: Predicting Numbers with Linear Regression

Regression predicts a continuous number: a house price, tomorrow's demand, a delivery time. Linear regression is the classic starting point: it fits the line (or, with several features, the flat surface) y = w1·x1 + w2·x2 + … + b that minimises the squared prediction errors. It is fast, easy to interpret — each coefficient says how much the prediction changes per unit of a feature — and a strong baseline to beat.

This lesson predicts house prices from several features, evaluates the model with MAE, RMSE and R², interprets the coefficients, adds polynomial features for curved relationships, and compares with a tree-based model.

Regression Metrics

MAE (mean absolute error) — average size of the errors, in the target's units. Easy to explain: "off by Rs 2.1 lakh on average".
RMSE (root mean squared error) — like MAE but punishes large errors more.
R² (coefficient of determination) — share of the variation explained: 1.0 is perfect, 0 is no better than always predicting the mean, and it can be negative.
Always compare against a baseline, such as predicting the average price for every house.

Beyond Straight Lines

If the relationship curves, PolynomialFeatures adds squared and interaction terms so a linear model can fit curves. Regularised versions — Ridge and Lasso — shrink coefficients to reduce overfitting when there are many features. Tree ensembles such as RandomForestRegressor and HistGradientBoostingRegressor capture non-linear patterns and interactions automatically and are often the most accurate choice on tabular data, at the cost of interpretability.

Examples

Predicting house prices with several features

Python

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(42)
n = 500
houses = pd.DataFrame({
    "area_sqft": rng.integers(500, 3000, n),
    "bedrooms": rng.integers(1, 5, n),
    "age_years": rng.integers(0, 40, n),
    "distance_km": rng.uniform(1, 25, n).round(1),
})
# The "true" price formula (in lakh Rs) plus noise — in real life this is unknown.
houses["price_lakh"] = (0.045 * houses["area_sqft"] + 4 * houses["bedrooms"]
                        - 0.6 * houses["age_years"] - 1.5 * houses["distance_km"]
                        + 20 + rng.normal(0, 6, n)).round(1)

X = houses.drop(columns="price_lakh")
y = houses["price_lakh"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)

model = LinearRegression().fit(X_train, y_train)
predictions = model.predict(X_test)

print("MAE :", round(mean_absolute_error(y_test, predictions), 2), "lakh")
print("RMSE:", round(mean_squared_error(y_test, predictions) ** 0.5, 2), "lakh")
print("R^2 :", round(r2_score(y_test, predictions), 3))
print("baseline MAE (always predict the mean):", round((y_test - y_train.mean()).abs().mean(), 2))

for feature, weight in zip(X.columns, model.coef_):
    print(f"{feature:<12} {weight:+.3f} lakh per unit")
new_house = pd.DataFrame([{"area_sqft": 1200, "bedrooms": 2, "age_years": 5, "distance_km": 8}])
print("estimate for a new house:", model.predict(new_house).round(1)[0], "lakh")

Output

MAE : 4.47 lakh
RMSE: 5.65 lakh
R^2 : 0.972
baseline MAE (always predict the mean): 28.11
area_sqft    +0.045 lakh per unit
bedrooms     +3.797 lakh per unit
age_years    -0.586 lakh per unit
distance_km  -1.464 lakh per unit
estimate for a new house: 66.5 lakh

Curved relationships: polynomial features vs a straight line

Python

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

rng = np.random.default_rng(0)
speed = np.linspace(10, 120, 60).reshape(-1, 1)                 # km/h
fuel = 0.002 * (speed.ravel() - 60) ** 2 + 5 + rng.normal(0, 0.3, 60)   # U-shaped curve

straight = LinearRegression().fit(speed, fuel)
curve = make_pipeline(PolynomialFeatures(degree=2), LinearRegression()).fit(speed, fuel)

print("straight line R^2:", round(r2_score(fuel, straight.predict(speed)), 3))
print("degree-2 curve R^2:", round(r2_score(fuel, curve.predict(speed)), 3))
print("most efficient speed ~", int(speed[curve.predict(speed).argmin()][0]), "km/h")

Output

straight line R^2: 0.121
degree-2 curve R^2: 0.982
most efficient speed ~ 60 km/h

Comparing linear models with a random forest on real data

Python

from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True, as_frame=True)   # disease progression after one year
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=7)

models = {
    "LinearRegression": LinearRegression(),
    "Ridge(alpha=1)": Ridge(alpha=1.0),
    "RandomForest": RandomForestRegressor(n_estimators=300, random_state=7),
}
for name, model in models.items():
    model.fit(X_train, y_train)
    print(f"{name:<17} MAE={mean_absolute_error(y_test, model.predict(X_test)):.1f}  R^2={model.score(X_test, y_test):.3f}")

forest = models["RandomForest"]
top = sorted(zip(forest.feature_importances_, X.columns), reverse=True)[:3]
print("most important features:", [(name, round(float(imp), 2)) for imp, name in top])
# Here the simple linear model wins: always compare against simple baselines.

Output

LinearRegression  MAE=41.9  R^2=0.503
Ridge(alpha=1)    MAE=47.1  R^2=0.457
RandomForest      MAE=44.0  R^2=0.484
most important features: [('s5', 0.28), ('bmi', 0.28), ('bp', 0.11)]

Common Mistakes

Reporting R² alone without an error in real units (MAE/RMSE) and a baseline.
Interpreting coefficients of features on very different scales as importance without scaling.
Extrapolating far outside the range of the training data.
Using a high polynomial degree and overfitting wildly.
Assuming a linear model is wrong just because a complex one scores slightly higher — interpretability matters.

Key Points to Remember

Regression predicts numbers; LinearRegression fits y = Σ wᵢxᵢ + b.
Evaluate with MAE, RMSE and R² on a test set, compared to a baseline.
Coefficients explain how each feature moves the prediction.
PolynomialFeatures handles curves; Ridge/Lasso regularise.
Tree ensembles are strong non-linear alternatives on tabular data.

Practice the examples

Change an input, predict the result, then compare it with the output. Explain why the result changes.

Use your local project environment for these examples. Codelab currently runs Python and HTML/CSS/JavaScript; framework examples may need project dependencies.