Index of every page on this site: llms.txt
All skills
robotostudio avatar

/content-experimentation-best-practices

@c0b29e4
Run it yourself

Content experimentation and A/B testing guidance covering experiment design, hypotheses, metrics, sample size, statistical foundations, CMS-managed variants, and common analysis pitfalls. Use this skill when planning experiments, setting up variants, choosing success metrics, interpreting statistical results, or building experimentation workflows in a CMS or frontend stack.

  • 5 files
  • 22 KB
  • Updated 6 months ago
  • GitHub
Use this Skill: https://skilld.dev/gh/robotostudio/turbo-start-sanity/content-experimentation-best-practices

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesexperiment-design.md

≈782 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Experiment Design Principles

Well-designed experiments produce actionable insights. Poorly designed ones waste time and can mislead.

The Experiment Framework

1. Hypothesis

State what you believe and why.

Bad: "Let's test a new headline" Good: "We believe a benefit-focused headline will increase signup rate by 10% because users are currently confused about our value proposition"

Structure: "We believe [change] will [impact metric] because [reasoning]"

2. Success Metric

Define primary and guardrail metrics.

Primary metric: The main thing you're trying to improve (conversion rate, engagement time) Guardrail metrics: Things that shouldn't get worse (bounce rate, page load time)

3. Sample Size

Calculate required sample size before starting.

Factors:

  • Baseline conversion rate
  • Minimum detectable effect (MDE)
  • Statistical significance level (usually 95%)
  • Statistical power (usually 80%)

Use calculators like Evan Miller's.

4. Duration

Run tests for full business cycles.

  • Minimum: 1-2 weeks (capture weekly patterns)
  • Include weekends
  • Avoid holidays and major events
  • Don't stop early when you see "winning" results

What to Test

High-Impact Areas

  • Headlines and value propositions
  • Call-to-action text and placement
  • Form length and fields
  • Pricing presentation
  • Social proof placement

Lower-Impact (Usually)

  • Button colors
  • Minor copy tweaks
  • Image variations (unless hero)
  • Footer changes

Test Priority Matrix

Impact Effort Priority
High Low Do first
High High Plan carefully
Low Low Quick wins
Low High Avoid

Sanity Integration Pattern

// Experiment variant schema
defineType({
  name: 'experimentVariant',
  type: 'object',
  fields: [
    defineField({ name: 'name', type: 'string' }),
    defineField({ name: 'weight', type: 'number', description: 'Traffic allocation (0-100)' }),
    defineField({ name: 'content', type: 'reference', to: [{ type: 'page' }] }),
  ]
})

// Experiment document
defineType({
  name: 'experiment',
  type: 'document',
  fields: [
    defineField({ name: 'name', type: 'string' }),
    defineField({ name: 'hypothesis', type: 'text' }),
    defineField({ name: 'status', type: 'string', options: { 
      list: ['draft', 'running', 'concluded'] 
    }}),
    defineField({ name: 'variants', type: 'array', of: [{ type: 'experimentVariant' }] }),
    defineField({ name: 'startDate', type: 'datetime' }),
    defineField({ name: 'endDate', type: 'datetime' }),
    defineField({ name: 'winner', type: 'string' }),
    defineField({ name: 'learnings', type: 'text' }),
  ]
})

Avoiding Common Mistakes

Don't peek and stop early

Statistical significance can fluctuate. Commit to your sample size.

Don't test too many things at once

Each variable multiplies required sample size.

Don't ignore segmentation

Winners may differ by device, traffic source, or user type.

Document everything

Future you (and your team) will thank you.

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at c0b29e4. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 6 months ago

README badge

README badge for robotostudio/turbo-start-sanity/content-experimentation-best-practices