Kohavi-inspired experiment review

A/B Experiment Review

A statistically significant number is only one part of a trustworthy A/B test. Enter result data and this local reviewer checks the data, the OEC, the guardrails, and the conditions around your decision. Inputs stay in this browser.

Review inputs

Describe the test

Four gates

Use unique users as the sample unit. The demo values are local to this page; nothing is uploaded.

1 / The result

Experiment name

Control · A

Users

Conversions

The version users would have seen before.

Treatment · B

Users

Conversions

The version you are testing.

2 / Data quality

Planned A share

50 means a planned 50 / 50 split.

Counting

A ratio mismatch is only a true SRM at the assignment layer.

SRM alpha

Stricter than 0.05 because this check runs constantly.

Days observed

SRM (sample ratio mismatch) is a validity gate. A surprising 49 / 51 split can be a symptom of missing users, bad assignment, exposure bugs, or filtering, rather than harmless noise. The check is a chi-square goodness-of-fit test on the observed counts against the planned split. Because it is re-run on every experiment, every day, and every segment, a 0.05 alpha drowns real signals in false alarms, so 0.0005 is the usual working threshold.

3 / Decision context

Planned duration

Guardrail rollup

Choose the strongest guardrail signal. A core metric win does not cancel out a serious latency, crash, or abandonment regression.

4 / Trust checklist

The OEC was agreed before seeing this result.Assignment, exposure, and logging were validated.Date slices were checked for novelty or primacy.The stopping rule accounts for early peeking.

This is a decision aid, not a full experimentation platform. It uses a two-sided pooled two-proportion z-test and a 95% Newcombe/Wilson delta interval for a binary OEC. It requires at least five pooled-null expected successes and failures in each arm before making a categorical evidence or ship recommendation. For ratio, revenue, retention, sequential, or clustered metrics, use a method designed for that metric.

Decision, after trustCalculating

Reviewing the evidence

Update the inputs to see whether this result is clear enough to act on.

Recommendation—

The recommendation combines the four review gates below.

Control rate: —
Treatment rate: —
Absolute lift: —
Relative lift: —
P-value: —
95% delta CI: —

Conversion rate

Control—

Treatment—

Data quality

Checking the observed allocation.

—

Sample ratio detail

Chi-square: —
p-value: —
Degrees of freedom: 1

Group	Expected	Observed	Diff	Diff %
Control	—	—	—	—
Treatment	—	—	—	—

OEC evidence

Checking the Treatment minus Control delta.

—

Guardrails

Checking for unintended regressions.

—

Readiness

Checking duration and the trust checklist.

—

Why this is not just a significance calculator

The pooled p-value and Wilson-based Treatment minus Control interval must agree before this reviewer calls a delta clear. A clear delta still needs trustworthy assignment, a meaningful OEC, safe guardrails, and a pre-agreed stopping rule.

Term guide

Plain English

OEC: The Overall Evaluation Criterion: the primary, pre-agreed measure of product or customer value.
Guardrail: A metric that should not materially regress, such as latency, crashes, or abandonment.
SRM: Sample Ratio Mismatch: observed group sizes differ from the planned allocation enough to question validity.
Control · A: The original version, used as the baseline for comparison.
Treatment · B: The new version or change being tested against Control.
Conversion rate: Conversions divided by users. For example, 100 / 1,000 = 10%.
Delta: Treatment rate minus Control rate. A positive delta means B is higher.
Absolute lift: The delta expressed in percentage points, such as +2 pp.
Relative lift: The delta divided by Control. A move from 10% to 12% is a 20% relative lift.
95% delta CI: A Wilson-based uncertainty range for Treatment minus Control, shown in percentage points. It remains useful at rates of 0% and 100%.
P-value: How surprising this difference would be if the two versions truly had the same rate. Here it comes from a two-sided pooled z-test; the reviewer reports both it and the separately constructed interval.
Two-proportion z-test: The approximate statistical method used here to compare two binary conversion rates.
Novelty / primacy: A treatment effect that changes over time because users learn, adapt, or react to a new experience.
Early peeking: Looking repeatedly and stopping when a result crosses a threshold without accounting for those repeated looks.

Results are calculated locally in your browser. No experiment data is sent anywhere. Inspired by Ron Kohavi's work on trustworthy online controlled experiments.