Built-in Evaluators Reference

Complete reference for Microsoft Foundry's built-in evaluators using the azure-ai-projects SDK.

Discovering Evaluators

List All Built-in Evaluators

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]

with (
    DefaultAzureCredential() as credential,
    AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
):
    evaluators = project_client.evaluators.list_latest_versions(type="builtin")
    for e in evaluators:
        print(f"{e.name}: {e.description}")
        print(f"  Categories: {e.categories}")

Get Evaluator Schema

Before using an evaluator, query its schema to discover required inputs:

evaluator = project_client.evaluators.get_version(
    name="builtin.task_adherence",
    version="latest"
)
print(f"Init Parameters: {evaluator.definition.init_parameters}")
print(f"Data Schema: {evaluator.definition.data_schema}")
print(f"Metrics: {evaluator.definition.metrics}")

Using Built-in Evaluators

All built-in evaluators use the azure_ai_evaluator type with builtin. prefix:

testing_criteria = [
    {
        "type": "azure_ai_evaluator",
        "name": "my_coherence_check",          # Your custom name for results
        "evaluator_name": "builtin.coherence", # The actual evaluator
        "data_mapping": {
            "query": "{{item.query}}",
            "response": "{{item.response}}"
        },
        "initialization_parameters": {
            "deployment_name": "gpt-4o-mini"   # Required for LLM-based evaluators
        }
    }
]

Quality Evaluators

builtin.coherence

Measures logical flow and consistency of the response.

{
    "type": "azure_ai_evaluator",
    "name": "coherence",
    "evaluator_name": "builtin.coherence",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response
Output: Score 1-5 (5 = highly coherent)

builtin.fluency

Measures grammatical correctness and natural language quality.

{
    "type": "azure_ai_evaluator",
    "name": "fluency",
    "evaluator_name": "builtin.fluency",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response
Output: Score 1-5 (5 = perfectly fluent)

builtin.relevance

Measures how well the response addresses the query given context.

{
    "type": "azure_ai_evaluator",
    "name": "relevance",
    "evaluator_name": "builtin.relevance",
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{item.response}}",
        "context": "{{item.context}}"
    },
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response, context
Output: Score 1-5 (5 = highly relevant)

builtin.groundedness

Measures whether the response is factually grounded in the provided context.

{
    "type": "azure_ai_evaluator",
    "name": "groundedness",
    "evaluator_name": "builtin.groundedness",
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{item.response}}",
        "context": "{{item.context}}"
    },
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response, context
Output: Score 1-5 (5 = fully grounded)

builtin.response_completeness

Measures whether the response fully addresses all aspects of the query.

{
    "type": "azure_ai_evaluator",
    "name": "response_completeness",
    "evaluator_name": "builtin.response_completeness",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Inputs: query, response
Output: Score 1-5

Safety Evaluators

Safety evaluators detect harmful content. They don't require deployment_name.

builtin.violence

Detects violent content.

{
    "type": "azure_ai_evaluator",
    "name": "violence",
    "evaluator_name": "builtin.violence",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}

Inputs: query, response
Output: pass/fail with severity score

builtin.sexual

Detects inappropriate sexual content.

{
    "type": "azure_ai_evaluator",
    "name": "sexual",
    "evaluator_name": "builtin.sexual",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}

builtin.self_harm

Detects content promoting or describing self-harm.

{
    "type": "azure_ai_evaluator",
    "name": "self_harm",
    "evaluator_name": "builtin.self_harm",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}

builtin.hate_unfairness

Detects biased or hateful content.

{
    "type": "azure_ai_evaluator",
    "name": "hate_unfairness",
    "evaluator_name": "builtin.hate_unfairness",
    "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"}
}

Agent Evaluators

Agent evaluators assess AI agent behavior and tool usage.

builtin.task_adherence

Evaluates whether the agent follows its system instructions.

{
    "type": "azure_ai_evaluator",
    "name": "task_adherence",
    "evaluator_name": "builtin.task_adherence",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Note: Use {{sample.output_items}} for agent responses to include tool call information.

builtin.intent_resolution

Evaluates whether the agent correctly understood user intent.

{
    "type": "azure_ai_evaluator",
    "name": "intent_resolution",
    "evaluator_name": "builtin.intent_resolution",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_text}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

builtin.task_completion

Evaluates whether the agent completed the task end-to-end.

{
    "type": "azure_ai_evaluator",
    "name": "task_completion",
    "evaluator_name": "builtin.task_completion",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

builtin.tool_call_accuracy

Evaluates whether tool calls are correct (selection + parameters).

{
    "type": "azure_ai_evaluator",
    "name": "tool_call_accuracy",
    "evaluator_name": "builtin.tool_call_accuracy",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

builtin.tool_call_success

Evaluates whether tool calls executed without failures.

{
    "type": "azure_ai_evaluator",
    "name": "tool_call_success",
    "evaluator_name": "builtin.tool_call_success",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"}
}

builtin.tool_selection

Evaluates whether the correct tools were selected.

{
    "type": "azure_ai_evaluator",
    "name": "tool_selection",
    "evaluator_name": "builtin.tool_selection",
    "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_items}}"},
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

NLP Evaluators

NLP evaluators compare responses to ground truth without requiring an LLM.

builtin.f1_score

Token-level F1 score between response and ground truth.

{
    "type": "azure_ai_evaluator",
    "name": "f1",
    "evaluator_name": "builtin.f1_score",
    "data_mapping": {"response": "{{item.response}}", "ground_truth": "{{item.ground_truth}}"}
}

Output: Score 0-1

builtin.bleu_score

BLEU score for generation quality.

{
    "type": "azure_ai_evaluator",
    "name": "bleu",
    "evaluator_name": "builtin.bleu_score",
    "data_mapping": {"response": "{{item.response}}", "ground_truth": "{{item.ground_truth}}"}
}

builtin.rouge_score

ROUGE score for summarization quality.

{
    "type": "azure_ai_evaluator",
    "name": "rouge",
    "evaluator_name": "builtin.rouge_score",
    "data_mapping": {"response": "{{item.response}}", "ground_truth": "{{item.ground_truth}}"}
}

builtin.similarity

Semantic similarity between response and ground truth.

{
    "type": "azure_ai_evaluator",
    "name": "similarity",
    "evaluator_name": "builtin.similarity",
    "data_mapping": {
        "query": "{{item.query}}",
        "response": "{{item.response}}",
        "ground_truth": "{{item.ground_truth}}"
    },
    "initialization_parameters": {"deployment_name": "gpt-4o-mini"}
}

Evaluator Sets by Use Case

Quick Health Check

testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "coherence", "evaluator_name": "builtin.coherence", ...},
    {"type": "azure_ai_evaluator", "name": "fluency", "evaluator_name": "builtin.fluency", ...},
    {"type": "azure_ai_evaluator", "name": "violence", "evaluator_name": "builtin.violence", ...},
]

Safety Audit

testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "violence", "evaluator_name": "builtin.violence", ...},
    {"type": "azure_ai_evaluator", "name": "sexual", "evaluator_name": "builtin.sexual", ...},
    {"type": "azure_ai_evaluator", "name": "self_harm", "evaluator_name": "builtin.self_harm", ...},
    {"type": "azure_ai_evaluator", "name": "hate_unfairness", "evaluator_name": "builtin.hate_unfairness", ...},
]

Agent Evaluation

testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "task_adherence", "evaluator_name": "builtin.task_adherence", ...},
    {"type": "azure_ai_evaluator", "name": "intent_resolution", "evaluator_name": "builtin.intent_resolution", ...},
    {"type": "azure_ai_evaluator", "name": "tool_call_accuracy", "evaluator_name": "builtin.tool_call_accuracy", ...},
]

RAG Evaluation

testing_criteria = [
    {"type": "azure_ai_evaluator", "name": "groundedness", "evaluator_name": "builtin.groundedness", ...},
    {"type": "azure_ai_evaluator", "name": "relevance", "evaluator_name": "builtin.relevance", ...},
    {"type": "azure_ai_evaluator", "name": "response_completeness", "evaluator_name": "builtin.response_completeness", ...},
]

Data Mapping Reference

Data Source	Response Mapping	Use Case
JSONL dataset	`{{item.response}}`	Pre-recorded query/response pairs
Agent target	`{{sample.output_text}}`	Plain text response
Agent target	`{{sample.output_items}}`	Structured JSON with tool calls

When to use sample.output_items:

Tool-related evaluators (tool_call_accuracy, tool_selection, etc.)
Task adherence evaluator
Any evaluator needing tool call context

/azure-ai-projects-py

referencesbuilt-in-evaluators.md