/microsoft-foundry

Dataset Curation — Human-in-the-Loop Review

Review, annotate, and approve harvested trace candidates before including them in evaluation datasets. This ensures dataset quality by adding a human review gate between raw trace extraction and finalized test cases.

Workflow Overview

Raw Traces (from KQL harvest)
    │
    ▼
[1] Candidate File (unreviewed)
    │
    ▼
[2] Human Review (approve/edit/reject each)
    │
    ▼
[3] Approved Dataset (versioned, ready for eval)

Step 1 — Generate Candidate File

After running a trace harvest, save candidates with a status field:

.foundry/datasets/<agent-name>-traces-candidates-<date>.jsonl

Each line includes a review status:

{"query": "How do I reset my password?", "response": "...", "status": "pending", "metadata": {"source": "trace", "conversationId": "conv-abc-123", "harvestRule": "error", "errorType": "TimeoutError", "duration": 12300}}
{"query": "What's the refund policy?", "response": "...", "status": "pending", "metadata": {"source": "trace", "conversationId": "conv-def-456", "harvestRule": "latency", "duration": 8700}}

Step 2 — Present for Review

Show candidates in a review table:

Review Actions

For each candidate, the user can:

Batch Operations

• "Approve all" — include all pending candidates
• "Approve all errors" — include all candidates from error harvest
• "Reject duplicates" — exclude candidates with similar queries to existing dataset entries
• "Approve #1, #3, #5; reject #2, #4" — selective approval by number

Step 3 — Finalize Dataset

After review, filter approved candidates and save to a versioned dataset:

1. Read .foundry/datasets/manifest.json to find the latest version number
2. Filter candidates where status == "approved"
3. Remove the status field from the output
4. Save to .foundry/datasets/<agent-name>-<source>-v<N>.jsonl
5. Update .foundry/datasets/manifest.json with metadata

Update Candidate Status

Mark the candidate file with final statuses:

{"query": "How do I reset my password?", "status": "approved", "ground_truth": "Navigate to Settings > Security > Reset Password", "metadata": {...}}
{"query": "What's the refund policy?", "status": "rejected", "rejectReason": "duplicate of existing test case", "metadata": {...}}
{"query": "Can you help me...", "status": "approved", "metadata": {...}}

💡 Tip: Keep candidate files as an audit trail. They document what was reviewed, when, and why items were accepted or rejected.

Quality Checks

Before finalizing, verify dataset quality:

Next Steps

• Version the approved dataset → Dataset Versioning
• Organize into splits → Dataset Organization
• Run evaluation → observe skill Step 2