Glossary · Éprouver

Statistical significance

Statistical significance measures the probability that an observed difference between two versions is not down to chance. It is expressed as a confidence level: at 95%, there is still a one-in-twenty chance the difference is an accident.

Tester deux versions, et ne désigner un gagnant qu’au-dessus du seuil. The forty terms of the trade

What it is for

It is the guardrail that stops you concluding on noise. On small volumes, two identical versions naturally produce differences of several points from one day to the next. The threshold is what separates “B does better” from “B got lucky this week”.

How it is calculated

By comparing the two conversion rates and the volumes behind them — a two-proportion test. What matters is not the difference alone but the difference relative to volume: two points apart on 30 clicks means nothing, two points on 3,000 is solid.

The classic mistake

Reading a confidence level as a probability of winning. 95% confidence does not mean “B has a 95% chance of being better”: it means a difference of that size would be rare if the two versions were equivalent.

A/B test

What Subtraq answers, tier by tier

< 30

Not yet

Fewer than thirty clicks on a version, or no sale.

< 95%

No difference

The test is running, the observed gap stays within the noise.

≥ 95%

A winner

The gap is clear enough to move a budget.

The test that leads to this verdict

What Subtraq does with it

Both of the product's thresholds are public: thirty clicks per version minimum, and 95% confidence. Below the first, Subtraq answers “not yet”; below the second, “no difference”. Beyond 99.9%, the figure is shown as a ceiling rather than a precision it does not have.