Module 7. Honest comparison¶

After this module you will be able to

Explain why three samples are needed, not two, and what happens when there are two.
Measure on a simulation how much the test is spoiled by peeking.
Find a leak by the symptom "the metric is suspiciously good".
Split a time series so that the result is not a fabrication.
Keep a run log from which, half a year later, it is clear where a number came from.

Time: about a week. Prerequisites: Module 6. Notebook: open in Colab · notebooks/07-honest-comparison.ipynb

Why this¶

The shortest module of Part II and the most expensive. Everything before produced numbers. This module decides whether they mean anything.

Module 1 listed six ways to get an improvement that is not there. Three of them — peeking at the test, a leak and stopping on the result — live exactly here, in the comparison procedure. Here they are also cured.

Three samples, not two¶

Training. On it the weights are fitted.

Validation. On it decisions are made: which model, which depth, which \(\lambda\), when to stop.

Test. On it the final result is measured. Opened once.

Two samples are not enough, and the reason is subtle. Choosing the best of twenty models by validation, you choose both the true quality and the luck on this specific sample. The winner won partly by chance, and its validation result is inflated.

This is the same multiple comparison as in Module 1, the sixth point. Of twenty models one will turn out lucky simply by construction.

The test exists to measure the winner on data that took no part in choosing it. If the test was used for choosing, it is no longer a test but a second validation.

The test is opened once

Looked at the test, changed the model, looked again — the test is spoiled. Not by code, but by you: you became the channel through which information from the test seeped into the model.

After twenty such iterations the test metric is inflated just like the validation one. This is not theory — it is measured, and in the notebook it is the third thread.

Cross-validation¶

One split gives one number, and from Module 1 it is known what one number is worth.

K-fold: split the data into \(K\) parts, train on \(K-1\) and measure on the remaining one, repeat \(K\) times. You get \(K\) measurements: there is a mean, there is a spread, there is a confidence interval by the bootstrap from Module 1.

What is important here and often ignored: the spread between folds is usually larger than the difference between the models being compared. Model A with mean 0.85 against model B with mean 0.84, with a fold spread of 0.03, is not a victory of A but a lack of data for a conclusion. The interval-overlap rule from Module 1 works here too.

The choice of \(K\): five or ten. More — more expensive and almost no benefit. Less — the training sample is noticeably cut down.

If the hyperparameters are chosen by cross-validation, and the quality is measured by it too, you are back to two samples. The honest option is nested cross-validation: the outer loop measures, the inner one chooses. Expensive, so more often a separate test is kept.

The leak¶

A leak is the entry into training of information that will not be there at the moment of prediction.

The symptom is always the same: the metric suddenly got much better. A pleasant surprise in machine learning almost always means an error, and the first thing to do is look for it, not rejoice.

A catalogue of cases, each of which occurs regularly:

Preprocessing before the split. Normalisation, feature selection, filling in missing values, encoding categories — computed over all the data at once. The mean and variance of the test entered the training through the scale.

Duplicates. One object in the training and in the test. On user data this is a common thing: one person left several records.

A feature from the future. The "deal close date" in the task of predicting a deal's closing. The check is one: would this feature have been known at the moment the prediction is actually needed?

A consequence feature. Not the future formally, but generated by the target. The number of support requests as a feature of churn: they contact support because they are already leaving.

A group leak. Records of one patient, one shop, one session drifted apart into training and test. The model recognises the group, not the regularity. Cured by grouping at the split.

Time series¶

A separate case where the standard random split is simply wrong.

Shuffling the data across time at random, you train the model on the future and measure on the past. The metric will come out excellent and will mean nothing.

The right way is a split by time: training on the past, checking on the future, the window moving forward. Trained on January–March, measured on April. Trained on January–April, measured on May.

The cost of the error is measurable. In the notebook, on a series where the link of past to future slowly drifts — and that is how demand, prices and user behaviour behave — the random split shows an error two and a half times smaller than the real one.

The same principle applies to any task where the data arrives in a stream — and that is the majority of tasks that are actually deployed.

The run log¶

The last thread, and it is about half a year from now.

A number in a report lives longer than the memory of how it was obtained. Half a year later the question "why is it 0.87 here" has no answer if it is not written down.

The minimum worth recording for each run:

which model and all its parameters,
the seed and all seeds, if there are several,
which data — the version of the dataset, not just the file name,
the git commit of the code,
the versions of the key libraries,
the metrics on validation and on the test, if the test was opened,
the date and time.

Why library versions are on the list

This course's Module 5 notebook computed the area under a curve via np.trapz. The author's machine had NumPy 1.26, and everything worked. In NumPy 2 this function was removed, and the very first run in CI fell with an AttributeError.

The code had not changed, the data had not changed, the seed was the same. A line in someone else's changelog broke the run. Without a record of versions such an investigation starts from zero.

The fix, by the way, came out better than the original: the area is now computed in two lines directly, and the question of the function's name no longer arises in any version.

The format is secondary. CSV works; for large projects there are tools — in particular mlango, where a run is opened with one command and recorded together with the commit and the artefact. But a CSV you actually keep is better than a tool you installed and forgot.

The check of a log's quality is one: can the number be reproduced from the record. You cannot — the log does not work.

Practice¶

Part 1. The notebook¶

Open notebooks/07-honest-comparison.ipynb.

What is inside:

Choosing the best of twenty models by validation: how inflated the winner's result is.
K-fold by hand. The spread between folds against the difference between models.
Peeking at the test: sixty iterations of "improvements", we measure the degradation.
A leak through normalisation before the split: we measure how much it gives away.
A group leak on data with repeating users.
A time series: a random split against a split by time on the same data.
A run log in twenty lines.

Part 2. An audit of your own task¶

Take any of your own tasks with a trained model.

Go through the catalogue of leaks and for each point write "no, because…". In writing precisely: out loud everything is always fine.
Count how many times you looked at the test metric. Honestly.
Recompute the quality by cross-validation and build a confidence interval.
Compare it with the number you previously considered the result.

The fourth point can be unpleasant. That is the benefit of the module.

Assignment¶

Implement K-fold with grouping: objects of one group do not drift apart across folds.
Simulate peeking: fifty random models, choosing the best by the test. Build the gap between the test estimate and the honest one.
Take a dataset with a date. Measure the quality by a random split and by a split by time. The difference is the cost of the error.
Introduce a leak deliberately: add a feature slightly correlated with the target through the future. Find it with the permutation importance from Module 6.
Start a run log and run all the experiments of this module through it. A week later try to reproduce any number from the record alone.

Self-check¶

Why three samples? What exactly is inflated when there are two?
Why is the test opened once?
The spread between folds is larger than the difference between models. What conclusion?
What is the only reliable symptom of a leak?
What question to ask about every feature to catch a feature from the future?
Why does a random split of a time series give a meaningless metric?
What is a group leak and how is it cured?
What should be in a run record so that the number can be reproduced?

Next¶

Part II is finished. There are models, metrics and a procedure that makes their comparison honest. Next — Part III: neural networks, starting with backpropagation written from scratch in NumPy.

Everything that appears there will be compared against the boosting from Module 6 by the rules of Module 7. Otherwise it will be not a result but an impression.

A pleasant surprise in machine learning almost always means a leak. Look for it, do not rejoice.