Experiment Design Principles
Well-designed experiments produce actionable insights. Poorly designed ones waste time and can mislead.
The Experiment Framework
1. Hypothesis
State what you believe and why.
Bad: "Let's test a new headline" Good: "We believe a benefit-focused headline will increase signup rate by 10% because users are currently confused about our value proposition"
Structure: "We believe [change] will [impact metric] because [reasoning]"
2. Success Metric
Define primary and guardrail metrics.
Primary metric: The main thing you're trying to improve (conversion rate, engagement time) Guardrail metrics: Things that shouldn't get worse (bounce rate, page load time)
3. Sample Size
Calculate required sample size before starting.
Factors:
- Baseline conversion rate
- Minimum detectable effect (MDE)
- Statistical significance level (usually 95%)
- Statistical power (usually 80%)
Use calculators like Evan Miller's.
4. Duration
Run tests for full business cycles.
- Minimum: 1-2 weeks (capture weekly patterns)
- Include weekends
- Avoid holidays and major events
- Don't stop early when you see "winning" results
What to Test
High-Impact Areas
- Headlines and value propositions
- Call-to-action text and placement
- Form length and fields
- Pricing presentation
- Social proof placement
Lower-Impact (Usually)
- Button colors
- Minor copy tweaks
- Image variations (unless hero)
- Footer changes
Test Priority Matrix
| Impact | Effort | Priority |
|---|---|---|
| High | Low | Do first |
| High | High | Plan carefully |
| Low | Low | Quick wins |
| Low | High | Avoid |
Sanity Integration Pattern
// Experiment variant schema
defineType({
name: 'experimentVariant',
type: 'object',
fields: [
defineField({ name: 'name', type: 'string' }),
defineField({ name: 'weight', type: 'number', description: 'Traffic allocation (0-100)' }),
defineField({ name: 'content', type: 'reference', to: [{ type: 'page' }] }),
]
})
// Experiment document
defineType({
name: 'experiment',
type: 'document',
fields: [
defineField({ name: 'name', type: 'string' }),
defineField({ name: 'hypothesis', type: 'text' }),
defineField({ name: 'status', type: 'string', options: {
list: ['draft', 'running', 'concluded']
}}),
defineField({ name: 'variants', type: 'array', of: [{ type: 'experimentVariant' }] }),
defineField({ name: 'startDate', type: 'datetime' }),
defineField({ name: 'endDate', type: 'datetime' }),
defineField({ name: 'winner', type: 'string' }),
defineField({ name: 'learnings', type: 'text' }),
]
})Avoiding Common Mistakes
Don't peek and stop early
Statistical significance can fluctuate. Commit to your sample size.
Don't test too many things at once
Each variable multiplies required sample size.
Don't ignore segmentation
Winners may differ by device, traffic source, or user type.
Document everything
Future you (and your team) will thank you.