Skip to content

Guide

Missing values: drop, mean, median or something else

Drop rows when few are affected, use the median rather than the mean when a column has outliers, and add a flag when the fact that something was missing is itself meaningful. Whatever you choose, decide it on the training data only.

First, why is it missing?

The right treatment depends on the reason, and the reason is usually knowable by looking.

  • Missing at random — a sensor dropped a reading, someone skipped a field. Filling it in is reasonable.
  • Missing for a reason — income is blank because the person is unemployed; a discount column is blank because there was no discount. Here the blank is the information, and filling it with an average destroys it.
  • Missing because the column barely exists — 90% empty. Nothing to recover. Drop the column.

The methods, and when each is right

  • Drop the rows — safest when only a few are affected. You lose data, which matters most when you have little.
  • Drop the column — when it is mostly empty, or when what remains is not worth the complication.
  • Fill with the mean — simple, and easily distorted. One extreme value drags the average away from where most of the column actually sits.
  • Fill with the median — the middle value, unmoved by extremes. The better default for almost any numeric column, and strictly better on income, prices or counts.
  • Fill with the most common value — the usual answer for text columns.
  • Carry the previous value forward — for data in time order, where yesterday’s reading is a fair guess for today’s.
  • Interpolate — estimate from the values either side, as if drawing a line between them. Right for readings that change smoothly; wrong for anything that jumps.
  • Fill from similar rows (KNN) — slower, usually more accurate than an average, and it uses the rest of the row rather than ignoring it.
  • Add a “was missing” flag — keeps the fact that something was absent as a column of its own. Combine with any fill above when the absence is meaningful.

The mistake that quietly inflates every score

Whatever you choose, the value you fill with must be computed from the training rows only. Fill from the whole dataset and the test rows have contributed to their own preparation — the held-out score is no longer held out, and it is optimistic by an amount nobody can measure afterwards.

Split first, then decide the fill value on the training half, then apply that same value to the test half. The same rule covers scaling, encoding and feature selection.

It matters at prediction time too. A model in use must fill gaps with the constants learned during training — not with a median recomputed from whatever batch of rows arrives today, which would make the model quietly different on every call.

The short version

  • A few missing rows, plenty of data → drop the rows.
  • Mostly empty column → drop the column.
  • Numeric, some outliers → median.
  • Text → most common value.
  • Time-ordered readings → carry forward, or interpolate if it changes smoothly.
  • The blank means something → fill, and add a flag.

Common questions

Should I use the mean or the median to fill missing values?
The median, in most cases. The mean is pulled away from the centre by a few extreme values, so on income, prices or counts it fills gaps with a number that is not typical of the column. The mean is fine only when the column is roughly symmetric and free of outliers.
Is it better to drop rows with missing values or fill them?
Drop when only a few rows are affected and you have data to spare; filling adds invented values. Fill when dropping would cost a large share of the dataset. If a column is mostly empty, drop the column rather than either.
Does it matter if I clean my data before splitting it?
Yes, and it is the commonest silent mistake. A fill value computed from the whole dataset uses the test rows, so they have influenced their own preparation and the final score is optimistic by an amount you cannot measure afterwards. Split first.
What does adding a missing-value indicator do?
It adds a yes/no column recording where values were absent, so the model can learn from the absence itself. Useful when a blank means something — no discount given, no previous purchase — rather than a lost reading.