Small experiment tool

Did variant B actually win?

Enter the users and conversions from each group. The calculator focuses on the difference between Treatment and Control—not on whether two separate confidence intervals happen to overlap.

Four inputs

Enter your result

A / B

Use unique users as the sample unit. Conversion counts cannot be larger than user counts.

Control · A

The version users would have seen before.

Treatment · B

The version you are testing.

This is a two-sided two-proportion z-test with a 95% confidence interval. It is a quick explainer, not a replacement for a complete experiment review.

Delta first Calculating

Treatment is ahead

Update the inputs to see how clear the difference is.

Control rate —
Treatment rate —
Absolute lift —
Relative lift —
P-value —
95% delta CI —
Conversion rate
Control —
Treatment —
Look at the delta, not the overlap.

The important interval is Treatment minus Control. If that interval includes zero, the direction is not yet clear at the 5% level.

Term guide

Plain English
Control · A
The original version, used as the baseline for comparison.
Treatment · B
The new version or change being tested against Control.
User / sample
One unit in the experiment. Here, it means one unique user.
Conversion
The outcome you are counting, such as a signup or purchase.
Conversion rate
Conversions divided by users. For example, 100 / 1,000 = 10%.
Delta
Treatment rate minus Control rate. A positive delta means B is higher.
Absolute lift
The delta expressed in percentage points, such as +2 percentage points.
Relative lift
The delta divided by Control. A move from 10% to 12% is a 20% relative lift.
Confidence interval · CI
A range showing plausible values for an estimate, given this sample.
95% delta CI
The uncertainty range for Treatment minus Control. If it includes 0, the direction is uncertain at this level.
P-value
How surprising this difference would be if the two versions truly had the same rate. Smaller values mean more evidence against that assumption.
Statistically significant
A shorthand for a result whose two-sided p-value is below 0.05, or whose 95% delta CI does not include 0.
Two-sided test
A test that checks for either an increase or a decrease, rather than only looking for a win.
Two-proportion z-test
The approximate statistical method used here to compare two conversion rates.
Sample size
The number of users included in a group. Total sample size is Control users plus Treatment users.
5% level
The threshold used by this quick calculator. It corresponds to a 95% confidence interval.

Results are calculated locally in your browser. No experiment data is sent anywhere.