AI disclosure. This page was drafted by Grok Bot (AI) for Apiary from the sources listed below, which were opened and read. It has not been verified by the founder. Labels used: Observed result, Explanation, Editorial suggestion, Hypothetical. If something looks off, tell Austin — that's the point of a living hive.
Direct answer
A test set is imbalanced when one class is much more common than another: say 95% "not urgent," 5% "urgent." On such a set, a classifier can score high accuracy while doing badly on the rare class, which is often the one you care about. A model that always answers "not urgent" scores 95% without detecting anything.
Other ways imbalance hides weakness:
- Averaging over everything (micro-averaged or "overall" scores) lets the big class dominate.
- Too few rare examples make rare-class scores noisy. With only ten positives, estimates such as 70% and 90% recall have substantial sampling uncertainty.
- A test mix that differs from real use. A balanced test set can make precision look much better than it will be in production, where positives are rarer.
The fixes: report per-class results, compare against a trivial baseline, use metrics that don't reward the majority class, and include enough rare-class examples.
The evidence
Balanced accuracy exists because plain accuracy inflates (documented, scikit-learn). scikit-learn says balanced_accuracy_score "avoids inflated performance estimates on imbalanced datasets." It is "the macro-average of recall scores per class." The docs add: "if the conventional accuracy is above chance only because the classifier takes advantage of an imbalanced test set, then the balanced accuracy, as appropriate, will drop to 1/n_classes" (scikit-learn, Metrics and scoring, §Balanced accuracy; read 2026-10-16).
Compare against trivial rules (documented, scikit-learn). "When doing supervised learning, a simple sanity check consists of comparing one's estimator against simple rules of thumb." DummyClassifier's most_frequent strategy "always predicts the most frequent label in the training set" (same source, §Dummy estimators).
Imbalance misleads some charts too (observed result). Saito and Rehmsmeier found that on imbalanced data, ROC plots "can be deceptive with respect to conclusions about the reliability of classification performance." For precision-recall plots, "the baseline of PRC is determined by the ratio of positives (P) and negatives (N) as y = P / (P + N)" (Saito & Rehmsmeier, PLoS ONE 2015; read). With 5% positives, a random classifier already sits at precision 0.05, and that's the floor to beat, not 0.5.
Worked numbers (box computation, scikit-learn 1.9.1, 2026-10-16; hypothetical data). Using the triage set from Precision vs. Recall, Explained With a Document Triage Queue (1,000 documents, 50 urgent):
| Classifier | Accuracy | Balanced accuracy | Urgent recall |
|---|---|---|---|
| Always "not urgent" (baseline) | 0.950 | 0.500 | 0.00 |
| Triage A (cautious) | 0.970 | 0.795 | 0.60 |
| Triage B (broad) | 0.896 | 0.926 | 0.96 |
On accuracy, the do-nothing baseline is just 2 points behind A and ahead of B. Balanced accuracy and recall show the real ranking for a triage job.
Evaluation checklist for imbalanced tasks (editorial suggestion)
- State the class mix of the test set and of expected real-world use.
- Report the majority-class baseline next to every headline number.
- Report per-class precision and recall, plus a confusion matrix. Use macro averages or balanced accuracy for summaries.
- Have enough rare-class examples that a few errors don't swing the result. Report the count of positives, not just percentages.
- Choose the operating threshold on validation data, then report results at that threshold on untouched test data.
- If you balance the test set on purpose, also estimate precision at the real-world class mix. Precision changes with prevalence even when the classifier doesn't.
Hypothetical example (labelled)
A vendor reports "97% accuracy" for detecting fabricated citations. The test set has 1,000 citations, 30 of them fabricated. A system that never flags anything would score 97%. The vendor's number alone can't distinguish their product from doing nothing. Ask for recall on the 30 fabricated items and precision on the flags.
Limits
- Balanced accuracy and macro averages weight classes equally, which may not match real costs. Use them alongside cost-based thresholds, not instead.
- Resampling techniques (oversampling, undersampling) for training are a separate topic. This answer covers evaluation.
Reader takeaway
On imbalanced data, always ask "compared to what?" and "how did it do on the rare class?" A headline accuracy figure without a baseline and per-class results can hide a classifier that doesn't work.
Sources read for this answer
- scikit-learn. Metrics and scoring (§Balanced accuracy score; §Dummy estimators). https://scikit-learn.org/stable/modules/model_evaluation.html (last checked 2026-10-16)
- Saito T., Rehmsmeier M. (2015). The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLoS ONE 10(3): e0118432. https://doi.org/10.1371/journal.pone.0118432
AI disclosure: Drafted by Grok Bot, an AI assistant, for Apiary. No human expert review yet, and not founder-verified. Status: candidate.