Guide
Choosing which columns to use in a model
Scoring each column against the target on its own is fast and misses anything that only matters in combination. When your columns are correlated — as sensor readings usually are — you need a method that judges them together.
Why drop columns at all
More columns is not more information. Columns that carry no signal give a model more ways to find patterns that are not there, which costs you on data it has never seen. Fewer, better columns also train faster and are far easier to explain to somebody who has to act on the result.
Two families, and the difference matters
Univariate methods score each column against the target on its own, then keep the best few. They are fast, simple, and blind to one thing: a column that only matters in combination with another.
Multivariate methods judge columns together — by fitting a model and asking which it actually relied on. Slower, and they see the combinations.
Weight-on-bit tells you little about drilling speed. Weight-on-bit given rotation speed tells you a lot. A univariate score rates the first and never sees the second.
The case where univariate goes badly wrong
Sensor data, financial data and survey data all share a property: the columns are correlated. Five readings that move together will all score well individually, so a univariate method keeps all five — five copies of one piece of information — and drops something that carried a different piece but scored lower alone.
Check the correlation between your columns before choosing. If any pair is above about 0.8, they are largely saying the same thing, and a method that judges columns independently cannot tell.
Which method to use
- Correlated columns, enough rows → tree importance. It judges columns together, handles curved relationships, needs no scaling, and costs one model fit.
- You want the smallest set that still works → recursive elimination. Drops the weakest column and refits, repeatedly. Thorough and slow.
- You want a simple, explainable model → Lasso. Pushes useless columns to exactly zero and keeps what survives.
- Few rows relative to columns → a univariate method, deliberately. With sixty rows and twenty candidates, a model-based selector is fitting noise, and the sophisticated answer is the wrong one.
- No target yet → drop near-constant columns and one of each highly correlated pair. Neither needs to know what you are predicting.
Two things not to do
Do not select features using the whole dataset. Choosing columns by how well they predict the target across all rows, then testing on some of those rows, is leakage — the selection has already seen the answers. Split first.
Do not keep a column that will not exist when you need the prediction. It will improve every score you look at and be unavailable on the day it matters. A column that predicts too well is a warning, not a win — see how to tell if a model is any good.
Common questions
- How do I choose which features to use in a model?
- If your columns are correlated — as sensor, financial and survey data usually are — use a method that judges them together, such as tree importance or recursive elimination. Scoring each column on its own is faster but keeps several columns carrying the same information and misses any that only matter in combination.
- What is the difference between univariate and multivariate feature selection?
- Univariate scores each column against the target independently; multivariate fits a model and asks which columns it actually relied on. Univariate is fast and blind to interactions between columns. Multivariate is slower and sees them.
- Does removing features improve a model?
- Often, yes. Columns with no signal give a model more ways to fit noise, which costs accuracy on unseen data. Removing them also speeds up training and makes the result easier to explain.
- Should I do feature selection before or after splitting the data?
- After. Selecting columns using the whole dataset means the choice has already seen the test rows' answers, so the final score is optimistic. Split first, select on the training half, then apply the same column list to the test half.