Statistical significance calculator

How to use the statistical significance calculator

1. Enter visitors and conversions for control (A) and variant (B), from your testing tool or analytics. Use the same date range and the same definitions for both groups.
2. Choose the significance level you set before the test started. 95% is the default.
3. Choose the test type. Two-sided is the default and also catches a variant that is worse. One-sided only asks whether B is higher.
4. Read three things, in order: whether the p-value is below your alpha, the confidence interval for the difference, and any warning under the result.

The calculator assumes each visitor was randomly assigned to one group and counted once, each visitor either converted or did not, and the test ran to a sample size you planned in advance (a fixed-horizon test). It also needs enough data: if a group has fewer than about 5 expected conversions or non-conversions, it shows a warning, because the normal approximation behind the test breaks down.

What does statistical significance mean in an A/B test?

Statistical significance in an A/B test means the gap between control and variant is larger than random variation would usually produce if the change had no effect at all. You set the bar in advance, the significance level, usually 5%. If the p-value falls below it, the result is called statistically significant.

By Kurt Philip, Founder. Updated October 9, 2026

The logic runs backward from how most people think about it. The test starts by assuming the variant makes no difference (the null hypothesis). It then asks how likely a gap this large would be under that assumption. If the answer is "rarely", you reject the assumption and treat the difference as real. Significance is a filter against being fooled by noise, not a measure of how good the change is.

Two kinds of mistake are possible. A false positive (type I error) is calling a winner when the change did nothing; the significance level caps how often that happens. A false negative (type II error) is missing a change that really works; you reduce that with a larger sample, which is what statistical power is about.

What is a p-value, and what is it not?

A p-value is the probability of seeing a difference at least as large as the one in your test if control and variant truly converted at the same rate, assuming the test was run correctly. A small p-value means your data would be unusual under no difference. It is evidence against "no difference", not the chance that your variant wins.

What is the difference between the significance level and the confidence level?

The significance level (alpha) and the confidence level are the same threshold seen from two sides: the confidence level is 1 minus alpha. A 95% confidence level means alpha is 0.05, so you accept a 5% false positive rate when the change has no effect. That is why "significant at 95%" and "significant at the 5% level" mean the same thing.

The level also sets the critical z value: how many standard errors apart the two rates must be before the result counts as significant. These are the values the calculator uses.

Pick 95% unless you have a reason not to. Use 99% when a wrong call would be expensive or hard to reverse, such as a pricing or checkout change, and accept that it needs more visitors. Pick the level before the test starts; choosing it after you see the p-value turns the threshold into a formality.

How do you read the confidence interval for the lift?

The confidence interval for the lift is the range of true differences between variant and control that your data are consistent with, at the level you chose. The calculator reports it in percentage points. If the whole interval sits above zero, the variant is significantly better; if it includes zero, the test has not ruled out "no difference".

In the default example the observed difference is 0.0050 (+25.0% relative), and the 95% interval runs from +0.09 to +0.91 percentage points. Divided by the control rate of 2.00%, that is roughly +4.4% to +45.6% in relative terms. Roughly, because the control rate is itself an estimate. A significant result, then, is still compatible with a lift far smaller than the one you saw.

Two habits help. Report the interval, not only the point estimate, when you share a result. And compare its lower end with the smallest lift that would be worth shipping: if even the low end clears that bar, you have a result you can act on with some comfort.

The formula: a two-proportion z-test

The test statistic uses the pooled standard error, which assumes no difference between the groups. The confidence interval uses the unpooled standard error, which uses each group's own rate.

• r1 and r2 are the conversion rates of control and variant.
• q is the pooled rate: all conversions divided by all visitors.
• z is how many standard errors apart the two rates are.
• p comes from the standard normal distribution: twice the tail area beyond |z| for a two-sided test, the upper tail beyond z for a one-sided test.
• The confidence interval is the observed difference plus or minus Zc standard errors, where Zc is 1.96 for 95%. For a one-sided test the calculator reports a one-sided lower bound instead.

Because the test and the interval use slightly different standard errors, a result right on the edge can be significant while its interval just touches zero, or the other way round. Away from the edge they agree.

Worked example

Control: 200 conversions from 10,000 visitors. Variant: 250 conversions from 10,000 visitors. Two-sided test at 95%. These are the default values in the calculator above.

1. Rates: r1 = 200 / 10,000 = 0.0200 (2.00%) and r2 = 250 / 10,000 = 0.0250 (2.50%). Difference = 0.0050, a relative lift of +25.0%.
2. Pooled rate: q = (200 + 250) / (10,000 + 10,000) = 450 / 20,000 = 0.0225.
3. Standard error: SE = sqrt(0.0225 × 0.9775 × 0.0002) = sqrt(0.0000043988) = 0.0020973.
4. z = 0.0050 / 0.0020973 = 2.3840.
5. p = 2 × (1 - Phi(2.3840)) = 2 × 0.008563 = 0.017126, which rounds to 0.0171.
6. 0.0171 is below 0.05, so the result is statistically significant at the 95% level.
7. Confidence interval: SEu = sqrt(0.0200 × 0.9800 / 10,000 + 0.0250 × 0.9750 / 10,000) = sqrt(0.0000043975) = 0.0020970. Margin = 1.959964 × 0.0020970 = 0.0041101. Interval = 0.0050 ± 0.0041101 = 0.0008899 to 0.0091101, or +0.09 to +0.91 percentage points.

So the variant's observed lift is +25.0%, and the data are consistent with a true difference anywhere from about +0.09 to +0.91 percentage points. The real lift could be much smaller than the one observed.

Six results compared: what changes the verdict

The table below runs the same calculator on six sets of inputs at the 95% level. Each row changes one thing from the default example, so you can see what moves a result in or out of significance: the amount of traffic, the test type, and the direction of the difference.

• Traffic decides more than the lift does. The same +25.0% lift is significant with 10,000 visitors per group (p = 0.0171) and nowhere near it with 1,000 per group (p = 0.4509), where the interval runs from -0.800 to +1.800 points.
• One-sided halves the p-value when the variant is ahead (0.0171 becomes 0.0086), which is why it is tempting to switch after the fact, and why you should not.
• A huge test can make a small lift significant. A +3.0% relative lift is significant (p = 0.0334) with 500,000 visitors per group, but the low end of its interval is +0.005 percentage points. Whether that is worth shipping is a business question, not a statistical one.
• A one-sided test cannot see a loss. The variant below control is significant two-sided (p = 0.0171), but a one-sided test for an increase gives p = 0.9914: it was never looking that way.

Should you use a one-sided or a two-sided test?

Use a two-sided test unless you decided on one-sided before the test began. A two-sided test asks whether the variant differs from control in either direction, so it also flags a variant that hurts conversions. A one-sided test only asks whether the variant is higher, which gives a smaller p-value for a winner and no warning for a loser.

A one-sided test is defensible when a worse result and no result lead to the same action, for example you will keep the control either way, and you wrote that down before launch. It is not defensible as a way to rescue a result that just missed significance two-sided: that doubles your real false positive rate.

What is sample ratio mismatch, and how do you check for it?

Sample ratio mismatch (SRM) means the visitors in each group do not match the split you set, beyond what chance explains. If you set 50/50 and got a clearly uneven split, something in assignment or tracking is broken, and the significance result cannot be trusted, however small its p-value. Check the split before reading the result.

The check is a chi-square goodness-of-fit test on the visitor counts. With two groups it reduces to a z-test on the split. Example: you set 50/50 and recorded 10,000 visitors in control and 10,500 in the variant (48.8% in control).

N        = 10,000 + 10,500 = 20,500
expected = N × 0.5 = 10,250 per group
SD       = sqrt(N × 0.5 × 0.5) = 71.59
z        = (10,000 - 10,250) / 71.59 = -3.49
chi²     = z² = 12.20
p        = 2 × Phi(-|z|) = 0.0005

A p-value of 0.0005 says a split this uneven would be very unlikely from random assignment alone, so this test has a sample ratio mismatch. Because the check is run on every test, many teams use a strict threshold for it, such as 0.001, to avoid false alarms. The calculator above runs this check on your visitor counts every time: set the expected split if your test does not divide traffic 50/50, and it shows a sample ratio mismatch warning under the result when the p-value is below 0.001.

Common causes of SRM:

• A redirect or a slower-loading variant, so some visitors leave before the variant records them.
• Bots or internal traffic filtered from one group but not the other.
• The tracking tag firing on a different event, page or condition in each group.
• Assignment that depends on something else, such as a cookie banner choice, a logged-in state or a cached page.
• Changing the traffic split, or pausing a variant, partway through the test.

If you find SRM, fix the cause and rerun the test. Do not try to correct the numbers afterward.

Can you stop an A/B test as soon as it reaches significance?

No, not with a fixed-horizon test like this one. Checking a running test again and again and stopping at the first p-value below 0.05 is called peeking, and it makes false positives far more likely than the 5% you set. The p-value naturally wanders up and down while data come in, and given enough looks it will dip below the line by chance.

Three ways to stop a test honestly:

1. Fix the sample size in advance with the A/B test sample size calculator, run until you reach it, and read the result once. This is what this calculator assumes.
2. Run whole weeks. Visitors on a Monday morning rarely behave like visitors on a Saturday night. Ending a test partway through a weekly cycle can bias it toward whichever days happened to be included, even when the sample size is reached.
3. Use a method built for monitoring. Sequential tests, such as group sequential designs or the always-valid p-values some testing tools offer, are designed to be checked as data arrive. They adjust the threshold for each look. Use their own result, not this calculator, when you test that way.

Looking at a running test to catch a broken variant or a tracking failure is fine. Deciding the winner from those looks is the problem.

What is the difference between statistical and practical significance?

Statistical significance says a difference is probably not noise. Practical significance says the difference is big enough to matter to the business. They are separate questions: a very large test can make a lift too small to be worth the work statistically significant, and a small test can miss a lift that would be worth a lot.

Before the test, write down the smallest lift that would justify shipping the change, given what it costs to build and maintain. After the test, compare the confidence interval with it:

Can you test revenue or average order value with a significance calculator?

Not with this one. A statistical significance calculator for conversion rates treats every visitor as converted or not converted. Revenue per visitor and average order value are continuous numbers, usually skewed by a few large orders, so they need a test built for them: Welch's t-test on a large sample, or a bootstrap, often with extreme orders capped first.

It matters because the conversion rate and revenue can move in different directions. A variant that gets more people to buy can also push them toward cheaper items, and a discount banner can raise orders while lowering margin. Decide before the test which metric decides it, and check the others for harm. Convertica's case studies report more than one metric for exactly this reason. The dScryb test, for example, measured paid membership sign-ups as the main metric and revenue per visitor alongside it:

+49.4%

What happens when you test several variants or metrics?

Every extra comparison is another chance for a false positive. With one comparison at alpha 0.05, a change that does nothing comes out significant 5% of the time. Compare three variants with control, or check ten metrics, and the chance that at least one crosses the line by luck alone is much higher. This is the multiple comparisons problem.

The figures are 1 minus 0.95 to the power of the number of comparisons. Real metrics are often correlated, so the true figure is usually lower, but the direction holds. Three ways to stay honest:

• Name one primary metric before the test starts and decide on that. Treat the rest as guardrails or ideas for the next test.
• Correct for several variants. The Bonferroni correction tests each comparison at alpha divided by the number of comparisons. It is strict but simple. The sample size calculator has a Bonferroni option so you can plan for it; then use the matching level here. With five comparisons, for example, Bonferroni gives 0.01, which is the 99% setting.
• Treat segment wins with suspicion. "It won on mobile in Canada" after a flat overall result is usually a false positive. Retest the segment on its own if it matters.

Bayesian vs frequentist A/B testing

This calculator is frequentist: it asks how surprising your data would be if there were no difference. Bayesian testing tools start from a prior belief about likely lifts and report the probability that the variant is better, and the expected loss if you pick it. That answer is closer to what most people want to know, but it depends on the prior chosen, and it is not immune to peeking or multiple comparisons just because it is phrased as a probability. With sensible priors and enough data, the two approaches usually agree on which variant to ship. The bigger risks are the ones above: stopping early, broken splits and testing too many things at once.

Checklist: before you call an A/B test

1. The test reached the sample size you planned before it started.
2. It ran for whole weeks, covering your normal business cycle.
3. The visitor split matches the split you set (no sample ratio mismatch).
4. Both groups were counted the same way: same events, same dates, visitors or sessions in both.
5. You are reading the primary metric you chose in advance, at the level you chose in advance.
6. The p-value is below your alpha.
7. The confidence interval's lower end is above the smallest lift worth shipping, or you have accepted that it might not be.
8. Guardrail metrics, such as revenue per visitor or refunds, did not get worse.

Two more mistakes to avoid

• Treating the observed lift as the lift you will get. Significant results from small tests tend to overstate the true effect, because only the lucky high readings clear the bar. Plan on the low end of the interval, not the point estimate.
• Testing a page that has no clear problem. A test only answers the question you ask it. The most useful tests start from evidence about where visitors get stuck, which is what a CRO audit is for.

Frequently asked questions