Skip to content

Guide

How to tell whether your model is actually any good

On its own, no — it is not information. If 94% of your rows are one class, a model that always guesses that class scores 94% too. The only number that means anything is how much better you are than doing nothing.

Why accuracy on its own tells you nothing

Suppose you are predicting which customers will cancel, and 94 in every 100 do not. A model that ignores your data entirely and answers “nobody cancels” every single time will be right 94% of the time. It has learned nothing. It cannot identify a single customer at risk — which was the entire point.

This is not an edge case. Most interesting problems are lopsided: fraud is rare, machine failure is rare, cancellation is rare. The rarer the thing you care about, the higher an empty model scores, and the more flattering the number on your screen.

A score is only meaningful next to what the same score would have been without a model. On its own it is not a weak signal — it is not a signal at all.

The baseline: what “doing nothing” scores

Before trusting any model, work out what a trivial answer achieves on the same held-out rows. It takes seconds and it reframes everything:

  • Classification — always predict the commonest class. If 94% of rows are “no”, your baseline is 94%. A model at 95% has bought you one point.
  • Regression — always predict the average of the training values. Compare your error against that. In R² terms the baseline is 0.0, so a model at 0.2 explains a fifth of what varies and leaves four fifths unexplained.
  • Forecasting — repeat the last observed value (the naive forecast). Surprisingly hard to beat, and a model that loses to it is worse than a pencil.

If your model does not clearly beat its baseline, the honest conclusion is that the signal is not in your data — and no amount of tuning fixes that. More columns, better columns, or more rows might.

Accuracy, balanced accuracy, and the gap between them

Accuracy counts every row equally, which is why a lopsided dataset makes it useless. Balanced accuracy averages the model’s hit rate on each class separately, so the rare class counts as much as the common one.

The gap between the two is the diagnosis. If accuracy is 94% and balanced accuracy is 50%, your model is coasting on the common class and has learned nothing about the rare one. Ten points of gap or more is the point at which the headline number is lying to you.

One split is one number, not a measurement

A single train/test split gives you one score, and part of that score is luck about which rows landed where. Run the same model on five different splits — cross-validation — and you get a mean and a spread. The spread is the part people skip, and it is what tells you whether a two-point lead over another model is real or noise.

On a small dataset the spread is often larger than the differences between models you were agonising over, which is worth knowing before you spend an afternoon choosing.

The failure that looks like brilliance

A model scoring 0.99 is more often a bug than a triumph. The usual cause is leakage: a column that contains the answer, or something derived from it. A discount_given column predicting “did they buy”. A closed_date predicting “is the ticket closed”. A total that includes the thing you are forecasting.

The test is not statistical, it is practical: would this column exist at the moment you need the prediction? If not, it must come out — however much it improves the score, and especially because of how much it improves the score.

Scaling or filling missing values using the whole dataset before splitting is the same mistake in quieter clothing: the test rows have influenced the numbers used to prepare them, so the held-out score is no longer held out.

The short version

  • What does the baseline score on the same rows? If you do not know, you do not have a result.
  • How far above it are you, and is that gap bigger than the spread across folds?
  • For classification: how far apart are accuracy and balanced accuracy?
  • Does every column you used exist at prediction time?
  • Was anything fitted — scalers, imputers — on data the model was then tested against?

Common questions

What is a good accuracy for a machine-learning model?
There is no universal number. What matters is the gap between your model and a baseline that ignores the data — predicting the commonest class for classification, or the training average for regression. On lopsided data a 94% accuracy can mean the model learned nothing, because guessing the common class scores 94% too.
What is a baseline model?
A deliberately trivial predictor scored on the same held-out rows as your real model: always answer the most common class, always answer the training average, or always repeat the last value in a time series. It is the floor your model has to clear to have done anything.
Why is my model 99% accurate?
Usually data leakage — a column that contains the answer or is derived from it, or preprocessing fitted on the whole dataset before the train/test split. Ask whether each column would actually exist at the moment the prediction is needed. If it would not, remove it.
What is balanced accuracy?
The average of the model's hit rate on each class taken separately, so a rare class counts as much as a common one. A large gap between accuracy and balanced accuracy means the model is coasting on the majority class.