Skip to content

Module 25. How the field is organised

After this module you will be able to

  • Describe a paper's path: idea → arXiv preprint → conference → peer review → proceedings.
  • Say what peer review catches and what it does not, and why it is a filter, not a proof.
  • Explain what a benchmark and a leaderboard are and why they are a poor source of truth.
  • Show how the race for first place overfits the whole community to the public test.
  • Use a leaderboard as a map, not a verdict.

Time: about one week. Prerequisites: Module 24, Module 7 and Module 16. Notebook: open in Colab · notebooks/25-how-the-field-is-organised.ipynb

Why this

In Module 24 you learned to check one paper. But papers do not hang in a vacuum: they have a pipeline that decides what you even see, and benchmarks that decide what counts as progress. To understand this machinery is to understand where, inside it, the truth is lost.

The path of a paper

idea preprint arXiv submit review proceedings claim, not fact filter, not proof

arXiv is a preprint server: a paper is posted with no review and instantly. This speeds up exchange, but it means the main thing: a preprint is a claim, not a verified fact. Half of the loud results live only on arXiv and go no further.

Peer review — two to four people read the paper and decide whether to take it to a conference. It catches obvious holes: a missing baseline, an unwarranted conclusion, a broken setup. But it almost never reproduces: a reviewer does not run the code, does not recompute the numbers, does not catch a fitted seed. Review is a filter on the quality of the writing, not a proof of correctness. A paper that passed review still needs the check from Module 24.

Benchmarks and leaderboards

To compare methods, the field sets up a benchmark — a shared test set and a metric. Results are lined up into a leaderboard — a ranking. This is convenient, and it is a poor source of truth, for several reasons at once.

ceiling fast growth saturation time · number of submitted methods score
  • The public test leaks through repeated submissions. Everyone tunes on the same set, and it gradually stops measuring generalization — this is Goodhart from Module 16: a benchmark that becomes a target stops being a measure.
  • First place is often "the best of many". The race for SOTA is the selection from Module 24 at the scale of a community: out of a hundred attempts the top goes to the luckiest, not the most reliable.
  • A single number hides per-slice failures. A leaderboard ranks by the average, and the average hides a regression exactly as in Module 23.
  • Saturation. When everyone is at the ceiling, the number stops telling methods apart: +0.1% at the top means nothing.

The order falls apart

The main consequence is in the picture. Take a leaderboard and run the same methods on a fresh test that no one has seen.

On the left is the race for first place on the public benchmark. Press "show the fresh test". On new data the order falls apart: method A, first on the benchmark, drops to fourth, and method C, which was only third, comes out on top. The community overfit the public test — and the leaderboard stopped measuring what it was set up for. You can use it, but as a map, not a verdict: it shows where to look, not what is true.

A leaderboard is the Goodhart's law of Module 16 at the scale of a whole community. There one agent padded the metric; here thousands of researchers tune to the same public test at once, and it stops measuring generalization — the very holdout of Module 7, worn through by collective fitting. A measure that becomes a target for everyone at once wears out faster than it does for one.

Practice

Part 1. The notebook

Open notebooks/25-how-the-field-is-organised.ipynb. Only numpy and matplotlib.

What is inside:

  1. Benchmark leakage: the community submits hundreds of methods to one public test. The leader's score climbs, while the result on a fresh test stays put. The gap is overfitting to the benchmark.
  2. The SOTA race: out of many noisy methods, "the best on the public test" is systematically worse on a fresh one — regression to the mean.
  3. Saturation: when everyone is at the ceiling, the difference between first and tenth dissolves into noise — the ranking stops discriminating.

Part 2. Your own field

Take a benchmark from your area (a leaderboard from a site, a table from a survey).

  1. How long has it existed and how close are the leaders to the ceiling?
  2. Is the test public? How many times a year is it submitted to?
  3. Does first place release code? How much bigger than the noise is the gap to second?
  4. Is there a fresh version of the benchmark — and do the old leaders fall on it?

Assignment

  1. Model a leaderboard of N methods with a true quality plus noise and a public test on which the best is picked. Plot the leader's score and its own fresh-test result as the number of submissions grows.
  2. Show that the gap between the public and the fresh score grows with the number of attempts — that is overfitting to the benchmark.
  3. Take a saturated benchmark (everyone at the ceiling) and show that the order of places is almost entirely determined by noise.
  4. Compare ranking by the average and by the worst slice, and find a method that leads on the average and fails on a slice.
  5. Build an "honest" leaderboard with confidence intervals and show how many top places are statistically indistinguishable.

Self-check

  1. What is arXiv and why is a preprint not a fact?
  2. What does peer review catch, and what does it not?
  3. Name four reasons a leaderboard is a poor source of truth.
  4. How does the race for first place overfit the community to the public test?
  5. Why does +0.1% at the top of a saturated benchmark mean nothing?
  6. Which module does benchmark leakage rhyme with, and which one the SOTA race?
  7. How do you use a leaderboard after all?

Next

You understand where the field loses the truth. In Module 26 — what to do about it: reproduction as the only real check. Not "it passed review" and not "it is on top of the leaderboard", but "I took the paper, ran it and got the same result — or did not".

A preprint is a claim, not a fact; review is a filter on the writing, not a proof of correctness; a leaderboard ranks by a number that the community gradually overfits. You can use a leaderboard, but as a map, not a verdict: it shows where to look, not what is true.

Principle

The machine that spreads results and the machine that checks them are different machines, and the second is weaker than the first. So a place in a ranking is a reason to look closer, not a conclusion.