A/B Test Significance Calculator for Email
Compare two email variants with a two-proportion z-test: lift, confidence, and whether the result is significant. Free, no sign-up.
Free with no sign-up. Everything runs in your browser: nothing is uploaded or stored.
Variant data
Result
Variant A rate
5.00%
Variant B rate
8.00%
Relative lift
+60.0%
Confidence
99.35%
Variant A
Variant B
Two-proportion z-test, two-tailed, at a 95 percent significance threshold. A significant result would occur by chance less than one time in twenty.
How it works
- Enter how many messages each variant went to and how many converted.
- The tool computes each rate and the relative lift of B over A.
- A two-proportion z-test returns the confidence that the gap is real, not chance.
- A result above 95 percent confidence is flagged as a winner; below that, keep collecting data.
Running a fair test
Change one thing at a time. If the subject and the opening line both change, the result tells you nothing about either. Split the list randomly, keep both variants otherwise identical, and let the test run at least one full business week so weekday and weekend behavior both enter the sample.
Peeking at the numbers and stopping at the first strong result inflates false positives; decide the sample size or end date first. Use subjects worth testing from the subject line analyzer, and compare the winning rate with published benchmarks in the benchmark calculator.
Small samples produce wide confidence intervals. The calculator warns when a variant has fewer than 100 sends or fewer than 5 conversions.
Frequently asked questions
What does statistically significant mean here?
It means the observed difference between variants is unlikely to be random chance alone. A 95 percent confidence result would occur by chance about one time in twenty, so teams usually act on 95 percent or higher and keep running the test below that.
How long should an A/B test run?
Until it reaches significance or a planned end date, and at least one full business week to cover weekday behavior. Peeking constantly and stopping at the first good result inflates false positives.
Can I test a subject line with this?
Yes. Use sent as the sample size and opens as conversions, but remember open tracking is distorted by Apple Mail Privacy Protection. Replies or clicks are more trustworthy outcomes.
Answer every email. Approve every send.
Connect Gmail, name your labels, and approve your first AI draft today.
14-day free trial
No credit card
Nothing sends without approval