/microsoft-foundry

eval.yaml Guidance

Create eval.yaml directly when the conversation or .foundry/agent-metadata*.yaml already selected the dataset/evaluators. Otherwise ask whether to run azd ai agent eval generate or let optimize use built-in defaults.

Include

name: <suite-or-optimization-name>
agent:
  name: <agent-name>
  kind: hosted
  version: "<agent-version>"
  model: <baseline-model-deployment-name>
  config: .agent_configs/baseline/metadata.yaml
dataset:
  local_uri: <path-to-jsonl>
  # name: <foundry-dataset-name>
  # version: "<dataset-version>"
# validation_dataset:
#   name: <validation-dataset-name>
#   version: "<validation-version>"
evaluators:
  - <evaluator-name>
  - name: <custom-evaluator-name>
    version: "<evaluator-version>"
    local_uri: <local-evaluator-json>
options:
  eval_model: <existing-chat-model-deployment-name>
  optimization_model: <allowed-optimizer-model-deployment-name>
  max_candidates: 4
  optimization_config:
    model_search_space:
      - <target-model-deployment-name>

Use existing model deployments for agent.model and options.eval_model; do not assume gpt-4o.

For options.optimization_model, first verify that the target Foundry project has a deployment whose name is in this allowlist:

• GPT-5
• GPT-5.1
• GPT-5.2
• GPT-5.4
• GPT-5.5
• DeepSeek-V4-Pro
• DeepSeek-V-3.2

If none exist, ask the user to deploy one before configuring optimization. Use options.optimization_config.model_search_space only for target model candidates that exist in the project; it may include the baseline model when the user wants it compared.

Generate evals when inputs are missing

Prefer eval generate over older init flows:

azd ai agent eval generate --dataset <path-to-jsonl>
azd ai agent eval generate --reset-defaults

After generation, run azd ai agent optimize --optimize-model <allowed-optimizer-model-deployment-name> from the azd project; optimize auto-detects the generated eval.yaml.

Skip

Do not add these fields unless the user explicitly asks and understands the tradeoff:

• target_attributes
• budget
• min_improvement
• pass_threshold
• keep_versions
• generation_instruction
• max_samples
• trace_days
• legacy dataset_file, dataset_reference, or validation_reference when writing a new file

Keep target_attributes omitted so azd can auto-detect optimizable attributes.

Source mapping

Treat older dataset_file, dataset_reference, validation_reference, max_iterations, and optimization_config.model as legacy inputs when reading existing files, but write new files with the current contract above.