Skip to content
QR Code Tools logo
QR Code Tools
Sign in
Testing Tool

QR A/B Test Significance Calculator

Enter scans and conversions for two QR variants and get a real chi-square test: the chi-square statistic, the p-value, and a significant / not-significant verdict at the 5% level. Everything recomputes from your numbers.

1. Enter results per variant

Method: chi-square on the 2×2 table, χ² = Σ (observed − expected)² / expected, 1 degree of freedom. p-value = 1 − erf(√(χ²/2)), using the Abramowitz & Stegun error-function approximation. The test needs every expected cell count ≥ 5 to be trustworthy.

Result
Verdict (α = 0.05)
Not significant
p = 0.0876 · χ² = 2.918
Variant A conversion rate5.00%
Variant B conversion rate6.80%
Relative uplift (B vs A)36.0%
Leading variantB

The difference could plausibly be chance. Keep running to the planned sample size before deciding.

A Higher Rate Is Not Always a Real Win

With small samples, one variant can lead by pure luck. The chi-square test asks how surprising the gap is if both variants were actually equal. Only when the gap is too large to dismiss as chance — conventionally, a p-value under 0.05 — should you treat the winner as genuinely better. Even then, significance is about existence of a difference, not its size: a tiny but significant uplift may not justify reprinting anything.

Common traps

Peeking daily and stopping at the first significant moment, testing on tiny samples, and running many variants at once all inflate false positives. Decide the sample size before the test, run to it, and judge the winner on uplift as well as significance.

Frequently Asked Questions

What test does this calculator run?

A chi-square test of independence on the 2×2 table (variant × converted / not converted), with 1 degree of freedom. The p-value is the probability of seeing a difference at least this large if both variants truly converted at the same rate.

What does "significant at 5%" mean?

It means the p-value is below 0.05: the observed difference would be unusual (under 1-in-20) if there were no real difference. It is not proof the winner is better, and it says nothing about how big the improvement is — check the uplift too.

Why does it warn about small expected counts?

The chi-square approximation is unreliable when any expected cell count is below 5. With tiny samples or very rare conversions, collect more data (or use an exact test) instead of trusting the p-value.

Can I peek at the result early and stop?

Repeated peeking inflates the false-positive rate: checking daily and stopping at the first significant moment makes "significant" far more common than 5% under no real difference. Decide the sample size in advance and run to it.