Kohavi-inspired experiment review

Can you ship this experiment?

A statistically significant number is only one part of a trustworthy A/B test. Enter a result snapshot and this local reviewer checks the data, the OEC, the guardrails, and the conditions around your decision.

Review inputs

Describe the test

Four gates

Use unique users as the sample unit. The demo values are local to this page; nothing is uploaded.

1 / The result

Experiment name

Control · A

Users

Conversions

The version users would have seen before.

Treatment · B

Users

Conversions

The version you are testing.

2 / Data quality

Planned A share

50 means a planned 50 / 50 split.

Days observed

SRM (sample ratio mismatch) is a validity gate. A surprising 49 / 51 split can be a symptom of missing users, bad assignment, exposure bugs, or filtering—not harmless noise.

3 / Decision context

Planned duration

Guardrail rollup

Choose the strongest guardrail signal. A core metric win does not cancel out a serious latency, crash, or abandonment regression.

4 / Trust checklist

The OEC was agreed before seeing this result. Assignment, exposure, and logging were validated. Date slices were checked for novelty or primacy. The stopping rule accounts for early peeking.

This is a decision aid, not a full experimentation platform. It uses a two-sided two-proportion z-test and a 95% delta interval for a binary OEC. For ratio, revenue, retention, sequential, or clustered metrics, use a method designed for that metric.

Decision, after trust Calculating

Reviewing the evidence

Update the inputs to see whether this result is clear enough to act on.

Recommendation —

The recommendation combines the four review gates below.

Control rate: —
Treatment rate: —
Absolute lift: —
Relative lift: —
P-value: —
95% delta CI: —

Conversion rate

Control —

Treatment —

Data quality

Checking the observed allocation.

—

OEC evidence

Checking the Treatment minus Control delta.

—

Guardrails

Checking for unintended regressions.

—

Readiness

Checking duration and the trust checklist.

—

Why this is not just a significance calculator

The important interval is Treatment minus Control. But a clear delta still needs trustworthy assignment, a meaningful OEC, safe guardrails, and a pre-agreed stopping rule.

Term guide

Plain English

OEC: The Overall Evaluation Criterion: the primary, pre-agreed measure of product or customer value.
Guardrail: A metric that should not materially regress, such as latency, crashes, or abandonment.
SRM: Sample Ratio Mismatch: observed group sizes differ from the planned allocation enough to question validity.
Control · A: The original version, used as the baseline for comparison.
Treatment · B: The new version or change being tested against Control.
Conversion rate: Conversions divided by users. For example, 100 / 1,000 = 10%.
Delta: Treatment rate minus Control rate. A positive delta means B is higher.
Absolute lift: The delta expressed in percentage points, such as +2 percentage points.
Relative lift: The delta divided by Control. A move from 10% to 12% is a 20% relative lift.
95% delta CI: The uncertainty range for Treatment minus Control. If it includes 0, the direction is uncertain at this level.
P-value: How surprising this difference would be if the two versions truly had the same rate.
Two-proportion z-test: The approximate statistical method used here to compare two binary conversion rates.
Novelty / primacy: A treatment effect that changes over time because users learn, adapt, or react to a new experience.
Early peeking: Looking repeatedly and stopping when a result crosses a threshold without accounting for those repeated looks.

Results are calculated locally in your browser. No experiment data is sent anywhere. Inspired by Ron Kohavi's work on trustworthy online controlled experiments.