/agentic-workflow-guide

Agent Evaluation

A framework for measuring agent workflow effectiveness based on vscode-ai-toolkit best practices.

Table of Contents

• Overview - Why evaluation matters
• Three Core Metrics - Intent Resolution, Tool Call Accuracy, Task Adherence
• Evaluation Methodology - How to evaluate
• Evaluation Best Practices - Guidelines for effective evaluation
• Common Evaluation Pitfalls - Anti-patterns to avoid
• Evaluation Checklist - Quick check template

Overview

Effective agent evaluation goes beyond simple success/fail metrics. It requires assessing the quality of reasoning, tool usage, and task adherence across diverse scenarios.

Three Core Metrics

1. Intent Resolution

Does the agent correctly understand and fulfill user intent?

Evaluation Questions

• Did the agent clarify ambiguous requests?
• Did it address the underlying goal or just the explicit ask?
• Were edge cases and constraints considered?
• Did the output match user expectations?

Example Scenarios

2. Tool Call Accuracy

Does the agent use tools correctly and efficiently?

Evaluation Questions

• Did the agent use available tools (not reimplementing)?
• Were tool parameters correct and complete?
• Was the tool sequence logical and efficient?
• Were errors handled gracefully?

Common Tool Usage Mistakes

Tool Call Metrics

**Accuracy:** (Correct Tool Calls) / (Total Tool Calls)
**Efficiency:** (Optimal Tool Count) / (Actual Tool Count)
**Coverage:** (Tools Used) / (Relevant Tools Available)

3. Task Adherence

Does the agent follow instructions and constraints?

Evaluation Questions

• Did the agent respect scope boundaries?
• Were constraints (time, resources, safety) followed?
• Did it avoid making unauthorized changes?
• Were deliverables complete per specification?

Adherence Violations

Evaluation Methodology

1. Test Data Generation

Create diverse, realistic test scenarios

Scenario Categories

Test Data Generation Process

Step 1: Identify Agent's Domain
  - What tasks does it handle?
  - What tools does it use?

Step 2: Create Baseline Scenarios
  - 10-20 typical use cases
  - Cover main functionality

Step 3: Add Variation
  - Ambiguous phrasing
  - Missing information
  - Conflicting requirements

Step 4: Include Edge Cases
  - Boundary values (empty, max size)
  - Unusual file types or formats
  - Concurrent requests

Step 5: Simulate Failures
  - Tool unavailability
  - Partial results
  - Timeout scenarios

Example Test Set (Code Review Agent)

test_cases:
  - id: "happy-001"
    input: "Review src/auth.py for security issues"
    expected_tools: ["read", "grep"]
    expected_output_type: "security_report"
    success_criteria:
      - Uses #tool:read to access file
      - Identifies security patterns (SQL injection, XSS, etc.)
      - Provides severity ratings

  - id: "ambiguous-001"
    input: "Check this"
    expected_behavior: "Request clarification"
    success_criteria:
      - Asks what to check
      - Offers options (code, tests, docs)
      - Doesn't make assumptions

  - id: "edge-001"
    input: "Review all Python files"
    context: "Repository has 1000+ .py files"
    expected_behavior: "Plan-First approach"
    success_criteria:
      - Creates plan before execution
      - Proposes batching or sampling
      - Estimates time required

  - id: "error-001"
    input: "Review src/nonexistent.py"
    expected_behavior: "Graceful error handling"
    success_criteria:
      - Detects file doesn't exist
      - Informs user clearly
      - Suggests alternatives (similar files, search)

2. Evaluation Rubric

Score each dimension on a scale

Intent Resolution Rubric

Tool Call Accuracy Rubric

Task Adherence Rubric

3. Automated Testing

Implement programmatic evaluation where possible

What Can Be Automated

Prefer Stable Assertions

Programmatic evaluation is strongest when assertions survive renames, workspace changes, and normal output variation.

Prefer:

• schema or heading checks over long exact-string matches
• tool-usage checks over replaying one transcript verbatim
• stable identifiers (file basenames, declared section names, output schema keys)

Avoid:

• temporary workspace paths
• machine-specific absolute paths
• exact output assertions copied from a single recorded run

What Requires Human Review

Automation Example (Python)

def evaluate_tool_calls(log_file, expected_tools):
    """
    Evaluate tool call accuracy from execution logs.
    """
    with open(log_file) as f:
        logs = json.load(f)

    tool_calls = [entry for entry in logs if entry["type"] == "tool_call"]

    metrics = {
        "total_calls": len(tool_calls),
        "unique_tools": len(set(call["tool_name"] for call in tool_calls)),
        "hallucinated": sum(1 for call in tool_calls
                           if call["tool_name"] not in expected_tools),
        "failed": sum(1 for call in tool_calls
                     if call["status"] == "error"),
    }

    metrics["accuracy"] = 1 - (metrics["hallucinated"] / metrics["total_calls"])

    return metrics

Evaluation Best Practices

1. Establish Baselines

Before making changes, measure current performance:

Baseline Evaluation (v1.0)

- Intent Resolution: 3.8/5.0
- Tool Call Accuracy: 4.2/5.0
- Task Adherence: 4.5/5.0

After Optimization (v1.1)

- Intent Resolution: 4.3/5.0 ✅ +13%
- Tool Call Accuracy: 4.6/5.0 ✅ +9.5%
- Task Adherence: 4.4/5.0 ⚠️ -2.2%

2. Diverse Test Set

3. Iterative Improvement

graph LR
    A[Baseline] --> B[Identify Weaknesses]
    B --> C[Implement Fix]
    C --> D[Re-evaluate]
    D --> E{Improved?}
    E -->|Yes| F[Deploy]
    E -->|No| G[Different Approach]
    G --> C

4. Log Everything

Comprehensive logging enables debugging and improvement

{
  "test_id": "happy-001",
  "timestamp": "2024-01-15T10:30:00Z",
  "user_input": "Review src/auth.py",
  "agent_plan": "1. Read file 2. Scan for patterns 3. Report",
  "tool_calls": [
    {
      "tool": "read",
      "params": { "file": "src/auth.py" },
      "status": "success"
    },
    { "tool": "grep", "params": { "pattern": "eval\\(" }, "status": "success" }
  ],
  "output": "Found 2 security issues...",
  "scores": {
    "intent_resolution": 5,
    "tool_accuracy": 5,
    "task_adherence": 5
  },
  "evaluator_notes": "Excellent handling, proactive security scan"
}

Logs are valuable inputs for designing evals, but they should not become the assertion wholesale. Use logs to discover robust checks, then reduce them to stable criteria.

5. Continuous Monitoring

Evaluation isn't one-time; monitor in production

Common Evaluation Pitfalls

1. Overfitting to Test Set

Problem: Agent performs well on tests but poorly on real tasks.

Solution:

• Use held-out test set
• Regularly refresh test scenarios
• Include user-reported issues

2. Ignoring User Feedback

Problem: Metrics look good, users are unhappy.

Solution:

• Collect qualitative feedback
• Review actual usage logs
• User interviews/surveys

3. Binary Pass/Fail

Problem: Doesn't capture nuance or areas for improvement.

Solution:

• Use rubrics with gradations
• Track multiple dimensions
• Identify specific failure modes

4. Testing Only Happy Paths

Problem: Agent fails when things go wrong.

Solution:

• Systematically test error conditions
• Simulate timeouts, missing resources
• Test with malformed inputs

5. Lack of Regression Testing

Problem: New features break existing functionality.

Solution:

• Maintain test suite
• Run before each release
• Automate where possible

6. Brittle Recorded Evaluations

Problem: A recorded test only passes for the original session because it encodes one temporary path or one exact output.

Solution:

• Rewrite assertions around structure, tool behavior, or schema
• Replace workspace-specific paths with stable identifiers
• Re-run evaluation after renames to confirm the checks still hold

Evaluation Checklist

Before deploying an agent workflow:

- [ ] Created diverse test set (happy, ambiguous, edge, error)
- [ ] Defined success criteria for each test
- [ ] Established baseline metrics
- [ ] Tested all major workflows
- [ ] Verified tool call accuracy
- [ ] Confirmed constraint adherence
- [ ] Assertions rely on stable structure or tool behavior, not one recorded run
- [ ] Logged evaluation results
- [ ] Identified improvement areas
- [ ] Documented known limitations
- [ ] Set up monitoring for production

References

• vscode-ai-toolkit Agent Evaluation
• Building Effective Agents - Anthropic
• Prompt Engineering Guide - OpenAI