Two of the most common reasons a machine-learning model performs badly are overfitting and underfitting. They are two ends of the same spectrum, and finding the middle is much of the craft of machine learning. This article explains both with a simple picture in words, shows how to detect them and lists practical fixes.
The student analogy
Imagine three students preparing for an exam using last year's questions.
- The underfitter skimmed the book and understood only the basic ideas. They fail both old and new questions.
- The overfitter memorised the exact answers to last year's questions. They score perfectly on those but fail when the questions change.
- The well-fitted student understood the concepts. They do well on old and new questions.
Models behave the same way.
A picture in words
Picture points scattered around a gently curving trend line, with some random noise.
- A straight line through the points misses the curve. This is underfitting (high bias): the model is too simple.
- A wiggly curve that passes through every point, following even the noise, is overfitting (high variance): the model is too flexible and learns the noise.
- A smooth curve that follows the trend but ignores random wiggles is a good fit.
Bias and variance
- Bias is error from overly simple assumptions. High-bias models are consistently wrong in the same direction.
- Variance is error from sensitivity to small changes in the training data. High-variance models change a lot when trained on different samples.
Increasing model complexity usually lowers bias and raises variance. The best model balances them, which is called the bias-variance trade-off.
How to detect them
Compare performance on training data and on validation data.
| Symptom | Training error | Validation error | Diagnosis |
|---|---|---|---|
| Both poor | High | High | Underfitting |
| Great on train, poor on validation | Low | High | Overfitting |
| Both good and close | Low | Low | Good fit |
Learning curves plot error as the training set grows. If training and validation errors converge at a high level, you have high bias: more data will not help, a richer model will. If there is a big gap, you have high variance: more data or regularisation will help.
Demonstration in Python
This example fits polynomials of different degrees to noisy data and compares training and test errors.
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error
rng = np.random.default_rng(0)
X = np.sort(rng.uniform(0, 1, 60)).reshape(-1, 1)
y = np.sin(2 * np.pi * X).ravel() + rng.normal(0, 0.25, 60)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.4, random_state=1)
for degree in (1, 3, 15):
model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(X_tr, y_tr)
tr = mean_squared_error(y_tr, model.predict(X_tr))
te = mean_squared_error(y_te, model.predict(X_te))
print(f"degree {degree:2d}: train MSE {tr:.3f} | test MSE {te:.3f}")You should see degree 1 with high error on both (underfitting), degree 15 with very low training error but higher test error (overfitting), and degree 3 doing best on test data.
How to fix underfitting
- Use a more expressive model (more layers, more trees, higher-degree features).
- Add informative features.
- Reduce regularisation.
- Train longer, for iterative models.
- Check that the data actually contains a learnable signal.
How to fix overfitting
- Get more data, or generate more with augmentation (for images, flips and crops).
- Simplify the model: fewer parameters, shallower trees, smaller networks.
- Regularisation: L1 or L2 penalties discourage large weights; dropout randomly disables neurons during training.
- Early stopping: stop training when validation error stops improving.
- Cross-validation to choose settings more reliably.
- Feature selection: remove noisy or redundant inputs.
- Ensembles: averaging many models, as in random forests, reduces variance.
- Prevent leakage: make sure information from the test set or the future does not sneak into training.
A regularisation example:
from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.pipeline import make_pipeline
model = make_pipeline(PolynomialFeatures(15), StandardScaler(), Ridge(alpha=1.0))
model.fit(X_tr, y_tr) # the penalty tames the wiggly curveOverfitting in the real world
Beyond the training set, you can "overfit" to the validation set if you tune many settings against it, or even to the test set if you repeatedly peek. Keep a final test set untouched until the end. Also beware that a model can fit patterns that exist in your data but not in the real world, such as a hospital-specific scanner artefact.
Deep learning and large models
Big neural networks can memorise their training data yet still generalise surprisingly well when trained with enough data and regularisation. But overfitting remains a problem on small datasets. Monitor validation loss, use augmentation and consider transfer learning from a pretrained model.
Frequently Asked Questions
Is overfitting always bad?
A model that memorises training data and fails on new data is not useful. A little gap between training and validation scores is normal.
How much data do I need to avoid overfitting?
It depends on model complexity and the noise level. Simpler models need less data.
Can a model overfit and underfit at the same time?
It can underfit in some regions of the input space and overfit in others, for example a model that is too simple for most cases but memorises a few odd ones.
Is a high training accuracy a good sign?
Only if validation accuracy is also high. Training accuracy alone says little about future performance.
Conclusion
Underfitting means the model is too simple to capture the pattern, and overfitting means it learned the noise. Compare training and validation performance, use learning curves, and apply more data, regularisation or a better-sized model to find the balance.