Synthetic Demand Calibration
Purpose: Validate synthetic user demands against real user data to measure and improve generation accuracy.
Calibration Framework
Why Calibrate
Synthetic demands generated by Echo[demand] are hypotheses, not facts. Without calibration against real-world data, they risk:
- Over-representing articulate user segments
- Missing culturally-specific needs
- Generating technically-flavored demands that real users would never express
- Confirming existing team biases rather than challenging them
Calibration Sources
| Source | Agent | Signal Quality | Use When |
|---|---|---|---|
| Real user feedback | Voice | High | Post-launch, ongoing product |
| Session replay patterns | Trace | High | Behavioral evidence available |
| Interview transcripts | Field | High | Research cycle completed |
| Support tickets | Voice | Medium | Support data accessible |
| NPS/CSAT comments | Pulse | Medium | Survey data available |
Calibration Process
- Generate: Echo[demand] produces demand set for target feature area
- Match: Compare synthetic demands against real data from Voice/Trace/Field
- Score: Rate each demand on the calibration matrix
- Tag: Apply confidence tags based on match results
- Adjust: Refine persona channeling for low-match segments
Calibration Matrix
| Match Level | Tag | Meaning | Action |
|---|---|---|---|
| Direct match | [validated] |
Real users voiced the same demand | High confidence, proceed |
| Partial match | [supported] |
Real data implies similar need | Medium confidence, refine wording |
| No match (plausible) | [hypothesis] |
No real data yet, but realistic | Flag for research validation |
| No match (implausible) | [synthetic-only] |
Likely an artifact of AI generation | Review for removal or revision |
Calibration Thresholds
- Healthy: ≥60% of demands are
[validated]or[supported] - Acceptable: 40-59% match rate — increase Field/Voice input
- Low confidence: <40% match rate — trigger recalibration with fresh Voice/Trace data
Recalibration Triggers
- New Voice feedback batch available
- Trace detects behavioral shift ≥5%
- Product pivot or major feature launch
30 days since last calibration
Integration with Cast
When calibration reveals persona accuracy issues:
- Feed match results back to Cast via FUSE mode
- Low-match personas may need Cast EVOLVE or RETIRE
- High-match personas strengthen Cast confidence scores