Skip to content

Module 20. A/B tests and causality

After this module you will be able to

  • Explain why correlation does not prove causation, and name the role of a confounder.
  • Say what exactly randomisation fixes and why an A/B test is a causal experiment.
  • Show on a simulation how peeking at interim results produces false significance.
  • Explain why checking twenty metrics gives a "significant" one by construction.
  • Assemble an honest A/B test and list the traps that spoil it — the same as in Module 1.

Time: about two weeks. Prerequisites: Module 1 and Module 19. Notebook: open in Colab · notebooks/20-ab-tests-and-causality.ipynb

Why this

Modules 18 and 19 ran up against one question: how to know that the new version of the recommender is better, and not just spinning the loop on convenient data. The answer is a causal experiment, and it also closes the recommendation part. But the methods here are not about recommendation specifically: this is how to tell "X improved Y" from "X and Y simply travel together" at all.

Correlation is not causation

"People who have the new feed on spend more time in the app" — and one wants to say the new feed retains better. Careful. The new feed may have been rolled out first to the active users; then the cause is not the feed, but who received it.

X Y Z observed correlation who got it activity

The hidden variable \(Z\) — "the user is active" — influences both the treatment \(X\) (who the feed was rolled out to) and the outcome \(Y\) (time in the app). It is what creates the correlation, and there may be no causal arrow from \(X\) to \(Y\) at all. \(Z\) is called a confounder, and while it is there, causation cannot be pulled out of an observation.

Randomisation fixes this

The only reliable way to remove a confounder is not to measure it more cleverly, but to break the arrow from it to the treatment. Assign who gets the new feed at random — by a coin flip, without looking at activity or anything else.

X Y Z coin now the difference is causal

After random assignment, groups A and B are the same in everything but the treatment: the active, the newcomers, the night owls are split evenly between both. So any difference in the outcome can be attributed only to the treatment. This is the A/B test — a causal experiment, not an observation. Exactly for this reason it, and not a metric on logs, was named the honest test in Modules 18 and 19.

Peeking

But randomisation is only half of honesty. The other half is how you look at the result. And here the traps from Module 1 return.

The temptation: to watch the test live and stop as soon as the difference becomes "significant". This is peeking, and it breaks the statistics. Even when there is no effect at all, the difference between the groups wanders, and sooner or later it randomly jumps over the significance threshold. Stop at that moment — and you declare a win that is not there.

Here A and B are from one distribution — there is no true effect at all. The blue line is the observed difference, the band is the 95-percent interval. Press "new run": in some runs the interval randomly excludes zero along the way (the red dots). Whoever stopped at the first red dot would have declared a significant effect — on pure noise. This is "stopping on the result" from Module 1, point five.

The defence is the same: the number of observations and the moment of the check are fixed in advance. If you want to look as you go, there are sequential-analysis methods that raise the threshold for peeking. But you may not decide from the data when to stop and then pretend you did not.

Multiple comparisons

The second way to invent significance is to check many things. The new feed did not move one metric — let us look at twenty: clicks, time, returns, scroll depth… With twenty checks at level 0.05, one false significance is expected by construction, even if nothing changed.

This is the sixth way to be wrong from Module 1. The defence is a correction for multiplicity (Bonferroni and gentler) or an honest "we checked twenty metrics, here are all of them". A single main metric chosen in advance is worth twenty chosen after the fact.

Module 1 returned in full. Peeking is "stopping on the result", twenty metrics are "multiple comparisons", and the spread between users is the same noise as between seeds. The A/B test adds just one new word to Module 1's arsenal — randomisation — and the rest is exactly the same: bootstrap intervals, the overlap rule and honesty about how much you checked in total.

Practice

Part 1. The notebook

Open notebooks/20-ab-tests-and-causality.ipynb. Only numpy and matplotlib, computes in seconds.

What is inside:

  1. Correlation versus cause: a confounder gives birth to a link that is not there. The observational estimate is biased, the randomised one is not.
  2. Peeking: A equals B, but stopping at the first significance gives false positives many times more than five percent. We count, not guess.
  3. Multiple comparisons: twenty honest metrics, one "significant" by construction. The Bonferroni correction puts the level back in place.

Part 2. Your own experiment

Take any change you would like to test: in a product, a newsletter, your own habit.

  1. What here is the treatment \(X\), what the outcome \(Y\), and what confounder \(Z\) could link them without a cause?
  2. What would random assignment look like? What prevents doing it?
  3. Which one metric will you name the main one in advance?
  4. How many observations will you collect before you look at the result?

Assignment

  1. Generate data with a confounder and show that the observational estimate of the effect is biased, while the randomised one is not.
  2. Simulate peeking: A equals B, a check after every hundred observations, stopping at the first significance. Measure the fraction of false wins against a fixed \(n\).
  3. Test twenty independent metrics on data with no effect and count how often at least one is "significant". Compare with Bonferroni.
  4. Build an A/B test honestly: a fixed \(n\), a metric chosen in advance, a bootstrap interval from Module 1. State the conclusion by the overlap rule.
  5. Come up with a novelty effect: the treatment gives a spike that fades. Show how a short test overestimates it.

Self-check

  1. What is a confounder and why, because of it, does correlation not prove causation?
  2. What exactly does randomisation fix and why, after it, is the difference causal?
  3. Why does peeking give rise to false significance even without an effect?
  4. Why do twenty metrics give a "significant" one by construction?
  5. Which three traps from Module 1 repeat in an A/B test?
  6. What protects against peeking and against multiple comparisons?
  7. Why does a metric on logs not replace an A/B test?

Next

Part V is finished: from collaborative filtering to the causal experiment that alone tells an improvement from a spun-up loop. In Part VI — agents: a loop with tools and memory, retrieval with citation checking, and evaluation, where eval is a test from Module 7 and regression is again the difference between runs.

An A/B test is a causal experiment: randomisation breaks the confounder, and the difference in outcome becomes a consequence of the treatment. But peeking and twenty metrics break it just as they broke any measurement in Module 1.

The principle

Randomisation gives you the right to say "cause", but does not cancel honesty about how you looked. The number of observations and the main metric are set before the experiment, not chosen from its result.