Preview Review Guide
Mindset
Quality is statistical, not per-record. Fix systemic issues that affect many records; don't chase cosmetic flaws in individual ones. But don't stop early — clear patterns of broken data or ignored instructions are worth fixing.
Reading Sample Records
Load dataset.parquet from the preview results directory (printed as Results path: by the preview command, or the most recent artifacts/preview_results_*/ directory). Use pandas to load the parquet file and print the records in a compact, reviewable format.
What to Look For
The specifics depend on the dataset and its intended use. The categories below are common starting points — adapt based on what matters for this dataset.
Diversity
- Mode collapse: are records clustering around the same patterns, topics, or phrasings?
- Sampler effectiveness: are samplers being used effectively to steer diversity in the dataset?
- Structural monotony: do LLM-generated columns follow the same template across records?
Data Quality
- Instruction compliance: does generated content follow prompt constraints (step counts, format requirements, allowed values)?
- Internal consistency: does data within a record agree with itself?
- Encoding integrity: no garbled encoding, mojibake, or broken unicode.
- Plausibility: do examples look like they could come from the real domain, or are they obviously synthetic?
- Judge calibration (if applicable): are scores consistent across similar-quality records? Does the judge catch visible problems?
Design Choices
Are the right Data Designer features being used? For example:
- A text column that consistently produces structured data or code might be better as a specialized column type.
- Values drawn from a fixed set or known distribution could use a sampler instead of an LLM column.