You trained a model and it says "95% accurate". Is that good? It depends. If only 5% of cases are positive, a model that always answers "no" is also 95% accurate and completely useless. Choosing the right evaluation metric is as important as training the model. This guide explains the key metrics for classification and how to read them.
The confusion matrix
For a yes/no (binary) problem, every prediction falls into one of four groups.
| Predicted positive | Predicted negative | |
|---|---|---|
| Actually positive | True positive (TP) | False negative (FN) |
| Actually negative | False positive (FP) | True negative (TN) |
Take spam detection, where "positive" means spam:
- TP: spam correctly caught
- FP: real email wrongly marked as spam
- FN: spam that reached the inbox
- TN: real email correctly delivered
All the main metrics are built from these four numbers.
Accuracy
Accuracy = (TP + TN) / (TP + TN + FP + FN)
It is the share of all predictions that were right. It works well when classes are balanced and errors cost about the same. It fails for imbalanced data, such as fraud, disease screening or defect detection, where the positive class is rare.
Precision
Precision = TP / (TP + FP)
"When the model says positive, how often is it right?" High precision means few false alarms. It matters when false positives are costly, such as marking legitimate emails as spam or accusing an honest customer of fraud.
Recall (sensitivity)
Recall = TP / (TP + FN)
"Of all the real positives, how many did the model find?" High recall means few misses. It matters when missing a case is costly, such as failing to detect a disease or a security threat.
The trade-off
Most models output a score or probability, and a threshold decides what counts as positive. Lowering the threshold catches more positives (higher recall) but also more false alarms (lower precision). Raising it does the opposite. Choose the threshold based on the cost of each type of error in your application, not by default at 0.5.
F1 score
F1 = 2 × (precision × recall) / (precision + recall)
The harmonic mean of precision and recall, giving a single number that is high only when both are reasonably high. It is useful for comparing models on imbalanced data. If you value one more than the other, use the F-beta score, which weights recall (or precision) more heavily.
Specificity and false positive rate
Specificity = TN / (TN + FP). It is the recall of the negative class. The false positive rate is 1 minus specificity. These are used in ROC analysis.
ROC curve and AUC
The ROC curve plots recall (true positive rate) against false positive rate across all thresholds. The AUC (area under the curve) summarises it: 0.5 means random guessing, 1.0 means perfect ranking. AUC measures how well the model ranks positives above negatives, independent of a threshold. For very imbalanced data, the precision-recall curve and its area (average precision) are often more informative.
Regression metrics
For predicting numbers:
- MAE (mean absolute error): average size of the error in the original units
- MSE and RMSE: penalise large errors more strongly
- R-squared: the share of variance explained, with caution since it can mislead
Worked example
Suppose a test set has 1,000 patients, 50 with a condition. A model finds 40 of them and wrongly flags 30 healthy people.
- TP = 40, FN = 10, FP = 30, TN = 920
- Accuracy = (40 + 920) / 1000 = 96%
- Precision = 40 / 70 = 57%
- Recall = 40 / 50 = 80%
- F1 = 2 × 0.57 × 0.80 / (0.57 + 0.80) ≈ 0.67
The 96% accuracy hides the fact that nearly half of the alerts are false. Whether that is acceptable depends on what happens after an alert.
Code walkthrough with scikit-learn
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (confusion_matrix, classification_report,
roc_auc_score, average_precision_score)
X, y = make_classification(n_samples=5000, weights=[0.95, 0.05], random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, test_size=0.3, random_state=0)
model = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
proba = model.predict_proba(X_te)[:, 1]
for threshold in (0.5, 0.2):
pred = (proba >= threshold).astype(int)
print(f"\nThreshold {threshold}")
print(confusion_matrix(y_te, pred))
print(classification_report(y_te, pred, digits=3))
print("ROC AUC:", round(roc_auc_score(y_te, proba), 3))
print("Average precision:", round(average_precision_score(y_te, proba), 3))Compare the reports at the two thresholds to see recall increase and precision decrease as the threshold falls.
Multi-class problems
For more than two classes you compute metrics per class and average them: macro average (each class equal weight), micro average (each example equal weight) or weighted average (by class size). Choose based on whether rare classes matter.
Good evaluation practice
- Use a separate test set that the model never saw during training or tuning.
- Use cross-validation for small datasets to get a more stable estimate.
- Pick metrics that match the real cost of mistakes.
- Look at errors, not just numbers. Read examples of false positives and false negatives.
- Check subgroups for fairness: performance by region, language or gender.
- Compare with a simple baseline, such as always predicting the majority class.
- Monitor after deployment, since data changes over time.
Frequently Asked Questions
Which metric should I use?
Start by asking which mistake is worse. If missing a case is worse, focus on recall. If false alarms are worse, focus on precision. Use F1 or AUC for balanced summaries.
Can accuracy be misleading?
Yes, especially with imbalanced classes. Always examine precision, recall and the confusion matrix.
What is a good F1 score?
There is no universal answer. Compare with a baseline and with the requirements of your application.
Why does my test score differ from my training score?
The model may be overfitting, or the test set may differ from the training data. Investigate before deploying.
Conclusion
Evaluating a model means looking beyond accuracy: use the confusion matrix, precision, recall, F1 and curves that reflect the real costs of errors. Choose thresholds deliberately and test on unseen data.