Statistical Foundations
Understanding basic statistics prevents misinterpreting experiment results.
Table of Contents
- Key concepts
- Sample size calculation
- Common statistical mistakes
- Interpreting results
- Alternative approaches
- When to trust results
Key Concepts
Statistical Significance
A measure of whether observed differences are likely real or due to chance.
- p-value < 0.05: "Statistically significant" at 95% confidence
- Means: If there were no real difference, there's less than a 5% chance of seeing results this extreme
- Does NOT mean: The change is important or meaningful
- Common misconception: The p-value is NOT "the probability the result is due to chance." It's the probability of observing data this extreme assuming the null hypothesis is true.
Confidence Interval
A range of plausible values for the true effect.
Example: "Conversion rate increased by 5% (95% CI: 2% to 8%)"
- Best estimate: 5% improvement
- Could be as low as 2% or as high as 8%
- Narrower intervals = more certainty
Statistical Power
The ability to detect a real effect when it exists.
- Standard: 80% power
- Higher power = larger sample size needed
- Low power = might miss real improvements
Minimum Detectable Effect (MDE)
The smallest improvement worth detecting.
- Smaller MDE = larger sample size needed
- Be realistic: Can you act on a 0.5% improvement?
Sample Size Calculation
Before running a test, calculate required sample size:
Required per variant = 16 × σ² / MDE²
Where:
- σ² = variance (for conversion rate: p × (1-p))
- MDE = minimum detectable effect (absolute)For a 5% baseline conversion rate, detecting a 1% absolute lift (5% → 6%):
- σ² = 0.05 × 0.95 = 0.0475
- MDE² = 0.01² = 0.0001
- n = 16 × 0.0475 / 0.0001 = 7,600 per variant
- Total: ~15,200 visitors minimum
Common Statistical Mistakes
Multiple Comparisons Problem
Testing 10 variants increases false positive rate.
Solution: Adjust significance threshold (Bonferroni correction) or use sequential testing methods.
Peeking Problem
Checking results daily and stopping when significant.
Why it's wrong: Significance fluctuates. Early "winners" often regress.
Solution: Pre-commit to sample size and duration. Use sequential testing if you must peek.
Simpson's Paradox
Overall results hide segmented truths.
Example:
- Overall: Variant B wins
- Mobile users: Variant A wins
- Desktop users: Variant A wins
- How? Different traffic mix per variant
Solution: Always segment by major factors (device, traffic source).
Survivorship Bias
Only analyzing users who completed the funnel.
Solution: Include all visitors, not just converters.
Interpreting Results
Significant + Meaningful
Clear win. Implement the change.
Significant + Trivial
Statistically different but tiny effect. Consider if worth the complexity.
Not Significant + Large Effect
Might be real but underpowered. Extend the test or accept uncertainty.
Not Significant + Small Effect
No detectable difference. Either no real effect or test was underpowered.
Alternative Approaches
Bayesian A/B Testing
An alternative to traditional (frequentist) hypothesis testing. Bayesian methods provide:
- Direct probability statements: "There's a 95% probability Variant B is better" (more intuitive than p-values)
- No peeking problem: Continuous monitoring is built in — you can check results at any time
- Credible intervals: Directly interpretable as "the true value falls in this range with X% probability"
Bayesian methods are offered by platforms like VWO and are useful when you need to make decisions with limited traffic or want more intuitive reporting for stakeholders.
Multi-Armed Bandits
Dynamically allocate more traffic to winning variants while still learning:
- Thompson Sampling: Balances exploration (learning) with exploitation (serving the best variant)
- Best for: Ongoing optimization where you want to minimize regret during the test
- Trade-off: Faster convergence to the winner, but less statistical rigor than fixed-allocation A/B tests
Consider bandits for content recommendations, personalization, or situations where the cost of showing a losing variant is high.
Sequential Testing
For teams that need to monitor experiments continuously:
- Group sequential designs (O'Brien-Fleming, Lan-DeMets) allow pre-planned interim analyses
- Always-valid p-values let you check results at any time without inflating false positive rates
- Use when you must balance the peeking problem with business pressure to act on results quickly
When to Trust Results
Checklist before declaring a winner:
- Reached pre-calculated sample size
- Ran for full business cycle (1-2 weeks minimum)
- p-value < 0.05 (or your chosen threshold)
- Effect size is meaningful for business
- Results consistent across major segments
- No external factors contaminated results