When you build a machine-learning model, a basic question is how to know whether it really works on new data. The standard answer is to split your dataset into parts with different jobs. Getting this wrong is one of the most common ways to fool yourself with impressive but meaningless results.
The three sets and their jobs
Training set
The data the model learns from. The model adjusts its parameters to fit these examples. Typically 60 to 80 percent of the data.
Validation set
Data used during development to compare models, tune settings (hyperparameters), choose features and decide when to stop training. You look at it many times. Typically 10 to 20 percent.
Test set
Data used once at the end to estimate how the final model will perform on unseen data. If you look at it repeatedly and adjust the model in response, it stops being a fair test. Typically 10 to 20 percent.
A helpful analogy: training is studying, validation is a practice exam you can retake and learn from, and the test is the real exam that you only take once.
Why not just two sets?
If you tune your model by repeatedly checking against the test set, information from the test set leaks into your choices. You will select the settings that happen to do best on that particular sample, so the reported score becomes optimistic. A separate validation set keeps the test set clean.
How to split correctly
Random split (the default)
For independent examples, shuffle and split. Use a fixed random seed for reproducibility.
from sklearn.model_selection import train_test_split
X_train, X_temp, y_train, y_temp = train_test_split(
X, y, test_size=0.3, random_state=42, stratify=y)
X_val, X_test, y_val, y_test = train_test_split(
X_temp, y_temp, test_size=0.5, random_state=42, stratify=y_temp)
print(len(X_train), len(X_val), len(X_test)) # about 70% / 15% / 15%stratify=y keeps class proportions similar in every split, which is vital for imbalanced data.
Time-based split
If the data has a time order, such as sales or sensor readings, do not shuffle. Train on the past and test on the future, just as the model will be used. Random splits would let the model peek at future patterns.
Group-based split
When several rows belong to the same person, patient, store or document, keep each group entirely in one set. Otherwise the model can recognise the person rather than learn the general pattern. Scikit-learn's GroupShuffleSplit and GroupKFold help.
Cross-validation
When data is limited, a single validation split can be noisy. In k-fold cross-validation, the training data is divided into k parts (folds). The model is trained k times, each time using a different fold for validation and the rest for training, and the scores are averaged. This gives a more reliable estimate and uses all data for both training and validation.
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(RandomForestClassifier(random_state=0), X_train, y_train,
cv=cv, scoring="f1")
print(scores.round(3), "mean:", scores.mean().round(3))Still keep a final test set aside. Cross-validation replaces the validation set, not the test set.
Data leakage: the silent killer
Leakage happens when information that would not be available at prediction time influences training. Typical examples:
- Preprocessing before splitting: fitting a scaler, imputer or encoder on the whole dataset lets test-set statistics leak into training. Always fit on the training data only, then apply to validation and test.
- Future information: using a feature recorded after the event you are predicting, such as "refund issued" when predicting if an order will be cancelled.
- Duplicates: the same or near-identical records in both training and test sets.
- Target-derived features: a column that is a transformed version of the label.
- Group leakage: the same customer appearing in train and test.
Use pipelines to prevent preprocessing leaks:
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=500))
pipe.fit(X_train, y_train) # scaler learns only from training data
print(pipe.score(X_val, y_val))Signs of leakage
- Unrealistically high scores (99% on a hard problem)
- A single feature with extreme importance
- Performance that collapses in production
How big should the splits be?
There is no fixed rule. With millions of examples, 1 percent for validation and test can be plenty. With a few hundred examples, use cross-validation and keep a small but meaningful test set. The test set must be large enough to give a stable estimate, especially for rare classes.
Deployment and drift
After launch, data changes over time (concept drift). Keep monitoring live performance and periodically build new test sets from recent data. A test set from last year may not represent today.
Checklist
- Define how the model will be used, and mimic that in your split.
- Split before any preprocessing or feature selection.
- Stratify for classification, split by time or group when appropriate.
- Use the validation set or cross-validation for all tuning.
- Touch the test set once, at the end.
- Report the test result honestly, even if disappointing.
Frequently Asked Questions
Can I retrain on all the data after testing?
Yes. After choosing the final settings, many teams retrain on training plus validation data (and sometimes test data) for the deployed model, using the earlier test result as the estimate.
What if my test score is much lower than validation?
You may have overfit to the validation set, or the test data may differ from it. Investigate for leakage and distribution differences.
Is a 80/20 split always right?
No. It is a common starting point. Adjust for dataset size, class balance and the need for stable estimates.
Do deep learning projects need all three?
Yes. Validation loss guides training decisions such as early stopping, and a held-out test set measures the final result.
Conclusion
Training teaches, validation guides decisions and the test set gives the honest final grade. Split carefully, prevent leakage, use cross-validation for small data and keep the test set untouched until the end.