How Linear Regression Behaves When Data Distribution Changes
A hands-on experiment: what happens to a Linear Regression model's predictions when the test distribution shifts away from training? The results are more nuanced than you'd expect.
Linear Regression is often presented as "just fitting a line." But what happens when you train on one distribution and evaluate on another? The behavior is more nuanced than most introductory materials suggest.
The Experiment
I trained a simple Linear Regression model on synthetic data with:
- Feature X ~ Normal(0, 1)
- Target Y = 2X + 1 + noise
Then I evaluated it on test sets with shifted distributions:
- Normal(2, 1) - mean shift
- Normal(0, 3) - variance increase
- Uniform(-2, 2) - completely different distribution shape
Observations
Mean shift: The model extrapolates linearly beyond its training range. Predictions are still structurally correct (the linear relationship holds), but the absolute error increases with the magnitude of the shift. R² degrades gradually.
Variance increase: More interesting. The model was trained on a narrow range; when evaluated on a wider range, it actually improves on the wider range if the true relationship is linear. The extrapolation works because the underlying function is linear.
Distribution shape change: When the test set follows a Uniform distribution while training was Normal, the model's predictions don't change (Linear Regression doesn't model the feature distribution - only the conditional E[Y|X]). This is a key insight.
The Key Insight
Linear Regression models E[Y|X] - the conditional expectation of Y given X. It makes no assumption about the marginal distribution of X. This means:
- Distribution shift in X alone doesn't break the model (if the linear relationship holds)
- Distribution shift that changes the *relationship between X and Y* (covariate shift that violates the linear assumption) does break it
- Extrapolation quality depends on whether the true function is actually linear outside the training range
When Does It Actually Break?
The model breaks under:
- Concept drift - the Y|X relationship changes over time (e.g., economic data post-shock)
- Non-linearity outside training range - the true function curves outside what the model saw
- Outliers in test set - Linear Regression is highly sensitive to outliers; they pull predictions toward them
This is why monitoring distribution shift in production ML requires looking at both the features *and* the label distribution - not just one.