A/B Test Significance Calculator

A/B Test Significance Calculator

Visitors and conversions
A (control) Baseline
Whole number above zero
Whole number, not more than visitors
B (challenger) Test
Whole number above zero
Whole number, not more than visitors
C (second challenger)
Whole number above zero
Whole number, not more than visitors
Test settings
Threshold for calling a result significant
Test direction
Two-tailed is the safer default

Conversion rates with confidence intervals

Breakdown by variant

VariantVisitorsConversionsRateConfidence interval

Copy the verdict and figures to paste into a report or ticket.

What you want to detect
%
Your control rate today
%
A 10% lift on 3.2% means a target of 3.52%
Leave blank to skip the days estimate
How sure you want to be
Chance of a false alarm you accept
Chance of catching a real lift
Traffic is split evenly between them
Test direction
Should match the test you will run

What other lift sizes would cost

Relative liftTarget ratePer variantAll variantsDays

Copy the plan to share with your team before the test starts.

What This Calculator Tests

An A/B test shows two versions of something - a landing page, an email subject line, a checkout button - to randomly split groups of visitors, then compares how many people in each group completed the goal, such as signing up or buying. The raw gap between the two conversion rates is only part of the story, because a small sample can produce a large gap purely by chance.

This tool runs a two-proportion z-test. It pools both groups to estimate the shared rate you would expect if the versions were identical, measures how far the observed difference sits from that expectation, and converts the distance into a p-value. A result counts as significant when the p-value is at or below 1 minus your chosen confidence level - so at 95% confidence, a p-value of 0.05 or less.

How to Use It

1. Enter the visitorsType how many people saw each variant during the test. Counts must be whole numbers above zero.
2. Enter the conversionsType how many of those visitors completed the goal. This can never be higher than the visitor count.
3. Pick the settingsChoose a confidence level and a one-tailed or two-tailed test. Set these before you look at the result.
4. Plan before you testUse the sample size planner to find how many visitors each variant needs before the experiment starts.

Practical Use Cases

Landing Page HeadlinesCheck whether a new headline lifts sign-ups over the current one before rolling it out to all visitors.
Checkout ChangesTest a shorter checkout form and see whether the extra completed orders are real or simply noise.
Email Subject LinesCompare open or click rates for two subject lines sent to comparable lists, with a confidence interval for the gap.

Limits and Notes

SituationWhat to keep in mind
Very few conversionsWhen a variant has fewer than about 5 conversions or non-conversions, the normal approximation is unreliable. The calculator warns you in that case.
Stopping earlyChecking the result repeatedly and stopping the moment it turns significant inflates false positives. Decide the sample size in advance with the planner.
Three variantsEach challenger is tested against the control separately. Testing several comparisons raises the chance that one looks significant by luck.
Unit of countingEnter visitors and conversions on the same basis, such as unique users or sessions. Mixing the two gives meaningless rates.

A significant result tells you a difference is unlikely to be pure chance under this model. It does not tell you how big the effect will be over the long run, or whether it is worth the cost of the change.

Frequently Asked Questions

Is my test data sent to any server?

No. The visitor and conversion counts you enter are processed by JavaScript in your own browser and nothing is uploaded. The figures are saved in this browser's local storage only, so they are still there when you come back to the page.

Why is a result not significant even though one variant looks better?

A higher conversion rate is not enough on its own. The test asks whether a gap of this size would be unlikely if both versions truly performed the same. With small samples, ordinary random variation can produce a large gap, so the p-value stays above the threshold until enough visitors have been counted.

What is the difference between a one-tailed and a two-tailed test?

A two-tailed test checks whether the variant differs in either direction, which is the safer default. A one-tailed test only checks for a change in one direction and reaches the threshold with less data, but it cannot tell you that the variant is worse. Choose the direction before you look at the data.

How many visitors do I need before I can trust a result?

Use the sample size planner. Enter your current conversion rate, the smallest relative lift you care about, the significance level and the power. The planner returns the number of visitors each variant needs, and converts that into days if you enter your daily traffic.

Please cross-verify the result yourself. This calculator can make calculation mistakes, for example from rounding, unusual inputs or a formula that does not match your case. Recalculate important figures by hand or with a second trusted tool before you rely on them for money, tax, health or other decisions.