Skip to content

Module 24. How to read a paper

After this module you will be able to

  • Read a paper not top to bottom, but in the order that reaches the point faster.
  • Find the main claimed number in a paper and ask it the three questions from Module 1.
  • See where the weak spot hides: the comparison, the seeds, the intervals, test leakage.
  • Tell a strong paper from a weak and a retracted one by what backs the number.
  • Turn reading from taking claims on faith into checking — the same eye as the whole course.

Time: about one week. Prerequisites: Module 1, Module 7 and Module 20. Notebook: open in Colab · notebooks/24-how-to-read-a-paper.ipynb

Why this

The previous twenty-three modules were written for this one. The field moves through papers, and most of the improvements claimed in them do not survive an honest check: they dissolve under a change of seed, come from comparing a tuned method with an untuned baseline, or rest on one lucky run. Someone who reads papers on faith builds on noise.

Reading a paper is not retelling it — it is checking a claim. Exactly what Module 1 did, only now the claim is someone else's and is delivered persuasively.

The reading order

You do not read a paper top to bottom. The introduction and related work are written to persuade; the tables and the figure are there to show. You go first to what can be checked, and only then to what explains it.

abstract · claim 1 main results table 2 baseline · seeds · interval 3 method 4 what is claimed how much, on what will it survive noise how it works read from the checkable (1–3) to the explanatory (4) — not top to bottom

The abstract gives the claim. The main table shows how much. And only the third step is the important one: what is compared against, how many times it was run, is the spread shown. If it is empty here, you can stop: the method does not matter if the gain cannot be told from noise.

Three questions for a number

Any claimed "+3.2%" is not a fact but a claim, and it has three questions from Module 1.

+3.2% compared to what? how many runs? what is the spread?

Compared to what — is the baseline honest or deliberately weak (Module 7)? How many runs — one lucky seed or five (Module 1)? What is the spread — do the intervals of method and baseline separate or overlap? That is the weak spot: not in the method, but in the comparison. A tuned method against an untuned baseline gives a "gain" out of thin air.

Three papers

The same claim by form — "a gain over baseline" — lives differently depending on what backs it. Let us take apart three papers: a strong one, a weak one, and a retracted one.

Press "audit the papers". The strong paper has, under its number, an honest baseline, five seeds, an interval, a held-out test and open code: the gain survives the check. The weak one has only the test and the code, no baseline, no seeds, no interval: +1.1% drowns in the noise. The retracted one, under the largest number +9.0%, has nothing at all. Notice: the largest gain turned out to be the emptiest — a big number with no evidence is more suspicious than a modest one with it.

The three questions you put to someone else's number are Module 1, turned outward. There you asked them of your own result: what it was compared against, how many times it was run, what the spread was. Here the same three go to someone else's paper — and the weak spot is the same every time, the comparison: a tuned method against an untuned baseline (Module 7) gives a gain out of thin air. To read a paper is to withhold trust from its number until it answers the questions you asked of your own.

Practice

Part 1. The notebook

Open notebooks/24-how-to-read-a-paper.ipynb. Only numpy and matplotlib.

What is inside:

  1. We simulate three papers with the same claim "+3% over baseline": one with an honest baseline and five seeds, one with a single lucky seed, and one whose result is fitted by the choice of seed. We show what survives a change of seed and what falls apart.
  2. A tuned method against an untuned baseline: how a "gain" appears out of an unequal comparison alone, with no method at all.
  3. Seed cherry-picking: why out of ten runs there is always a pretty one, and why pass@1 is more honest than "the best result".

Part 2. Your own paper

Take any paper with a claimed improvement — one you read for work or study.

  1. Find the main number in a minute, without reading the method. Where is it?
  2. What is compared against? Is the baseline tuned as carefully as the method?
  3. How many runs, is the spread shown? Would the gain survive a change of seed?
  4. Is there a limitations section — and is it honest, or there for show?
  5. Is the code released? What in the paper is left unsaid to the point that it could not be repeated?

Assignment

  1. Implement an "honest" and a "dishonest" claim: in the first the baseline is tuned, there are five runs, an interval is shown; in the second an untuned baseline and one seed. Plot both and show which survives the check.
  2. Model seed cherry-picking: make 10 runs of a noisy method and show that "the best of 10" systematically overstates the result.
  3. Take a table with overlapping intervals of method and baseline and show that the claimed gap cannot be told from noise.
  4. Assemble your own five-point reading checklist and run three real papers through it. Which passes?
  5. Find, in a real paper, a comparison that cannot be called honest, and state exactly what is unequal.

Self-check

  1. Why is a paper not read top to bottom, and where do you start?
  2. What three questions do you ask any claimed number?
  3. Where does the weak spot most often hide — in the method or in the comparison?
  4. Why is a large gain without evidence more suspicious than a modest one with it?
  5. What is seed cherry-picking and why does "the best of N" overstate the result?
  6. Which module does "an honest baseline" rhyme with, and which one "the gain cannot be told from noise"?
  7. What makes a paper reproducible, besides open code?

Next

You can check one paper. In Module 25 — how the field these papers come from is organized: arXiv, conferences, peer review and benchmarks, and why a leaderboard is a poor source of truth that you nonetheless have to use. And in Module 26 reading turns into action: reproducing a paper from the claim to working code.

A paper is read not top to bottom but from the checkable to the explanatory; any claimed number is checked with three questions — compared to what, how many runs, what is the spread; the weak spot hides in the comparison, not the method. A large number with no evidence is more suspicious than a modest one with it.

Principle

A paper's persuasiveness and its correctness are different things, so what you can trust is not how the result is claimed but what backs it. The largest gain with nothing behind it is not a finding but a reason to check.