Why this matters
A model that looks excellent on the data it learned from can fail badly on new cases. When that model decides who gets a loan offer or which villages get extra seed, the failure lands on real people. Testing models honestly is a core part of the job, named directly in skills frameworks for data scientists.
Overfitting
A model overfits when it memorises quirks of its training data instead of general patterns. A very deep decision tree can score perfectly on training data and poorly on anything else. The cure is to always judge a model on data it has not seen, and to prefer simpler models when they do about as well.
Train, validate, test
- Training set: the model learns from it.
- Validation: used to compare models and settings.
- Test set: locked away and used once at the end, to report final performance.
If you keep checking the test set while tuning, it stops being a fair test.
Cross validation
With modest data, one split can be lucky or unlucky. Cross validation splits the data into, say, five folds, trains on four and tests on the fifth, rotating through all five. Here model is the scaling and logistic regression pipeline from the previous lesson: