/google-agents-cli-eval

Evaluation Metrics Reference

File paths below reference the scaffolded layout (tests/eval/eval_config.yaml or .json). Adjust for your project structure if not using google-agents-cli-scaffold.

Python projects scaffold the eval datasets in the tests/eval/ directory; Go projects scaffold in the eval/ directory. Examples below use the Python convention.

Managed (Built-in) Metrics Reference

Run agents-cli eval metric list for the live set. Single-turn only below means the metric 400s on a trace with 2+ turns (Single-turn metric '<name>_v1' received agent_eval_data with N turns). The single-turn adaptive-rubric metrics grade a case's own rubric_groups instead of generating their own when it supplies them (see Managed Metric Parameters).

Agent metrics (adaptive rubrics)

multi_turn_general_quality and multi_turn_text_quality need a conversation_history field that eval generate does not produce, and 400 on agent traces. Use multi_turn_task_success or multi_turn_trajectory_quality.

General quality metrics (adaptive rubrics, single-turn only)

Static rubric metrics (fixed criteria, single-turn only)

Custom Metrics

Custom metrics are declared in eval_config.yaml (or .json) under custom_metrics. See SKILL.md's Evaluation Configuration Schema section for how metrics_to_run selects from the pool. The schema below defines the per-entry fields.

Code-based metrics default to local in-process execution (no GCP project or region required); opt into the Vertex AI sandbox with execution: "remote". The metric functions are written in Python, regardless of the project's language.

Scaffolded default metric. The scaffolded eval_config.yaml ships custom_response_quality as a local LLM-judge in tests/eval/response_quality.py (referenced via custom_function_file, run in-process via google-genai). It grades on either backend — genai.Client() uses GEMINI_API_KEY (AI Studio) or ADC (Vertex) — and reads each case's reference (ground truth) when present. To grade with the managed Vertex eval service instead, replace it with a built-in metric or an LLMMetric (prompt_template).

Example

metrics_to_run:
  - multi_turn_trajectory_quality
  - project_response_rubric
  - agent_turn_count

custom_metrics:
  - name: project_response_rubric
    prompt_template: |
      Rate the agent's response 1-5 for helpfulness and accuracy.
      Prompt: {prompt}
      Final response: {response}
      Full trace (for tool-call and reasoning context): {agent_data}
      Return JSON: {"score": <1|2|3|4|5>, "explanation": "<reason>"}
    judge_model_sampling_count: 3

  - name: agent_turn_count
    custom_function: |
      def evaluate(instance):
          turns = (instance.get("agent_data") or {}).get("turns", [])
          return {'score': len(turns)}

  - name: tool_call_count
    execution: remote
    custom_function: |
      def evaluate(instance):
          n = 0
          for turn in (instance.get("agent_data") or {}).get("turns", []):
              for event in turn.get("events", []):
                  for part in (event.get("content") or {}).get("parts", []):
                      if "function_call" in part:
                          n += 1
          return {'score': n}

Metrics receive the eval case's {prompt}, {response}, and {agent_data} (and {reference} / {context} when the case populates them) — see SKILL.md's Evaluation Configuration Schema → Agent trace field model for details.

Schema reference

Each entry in custom_metrics must conform to one of two Agent Platform evaluation metric schemas. custom_function or custom_function_file selects the code-based schema (in-process by default, CodeExecutionMetric with execution: remote); otherwise it's LLMMetric. An entry that carries neither, and whose name is a built-in metric, is a managed metric parameterization instead (see below).

Code Execution Metric (CodeExecutionMetric)

Evaluates responses using custom Python code.

Local metrics grade cases concurrently. Build the client once per thread, never inside evaluate(), which re-does ADC and the TLS handshake every case. Per thread rather than per process: one client cannot serve several threads behind a client certificate. tests/eval/response_quality.py has the shape to copy.

evaluate() can run up to five times for one case, since transient failures are retried. Avoid side effects that must happen exactly once.

Minimal custom_function_file example — point the metric at a sibling .py file instead of an inline blob:

# tests/eval/eval_config.yaml
metrics_to_run:
  - turn_count
custom_metrics:
  - name: turn_count
    custom_function_file: metrics.py   # resolved next to this config file

# tests/eval/metrics.py  (same directory as the config)
def evaluate(instance):
    turns = (instance.get("agent_data") or {}).get("turns", [])
    return {"score": len(turns)}

Run with agents-cli eval run --config tests/eval/eval_config.yaml.

LLM judge in a custom function: the way to combine your own judge prompt with per-case criteria, and how to grade multi-turn rubric_groups:

# tests/eval/rubric_judge.py: keep execution local: the remote sandbox has no network
import json

from google import genai


def evaluate(instance):
    rubrics = [
        r["content"]["property"]["description"]
        for g in (instance.get("rubric_groups") or {}).values()
        for r in g["rubrics"]
    ]
    prompt = (
        f"Criteria: {rubrics}\nConversation: {json.dumps(instance['agent_data'])}\n"
        'Return JSON: {"score": <fraction of criteria met>, "explanation": "<what failed>"}'
    )
    out = genai.Client().models.generate_content(
        model="gemini-3.8-flash",
        contents=prompt,
        config={"response_mime_type": "application/json"},
    )
    return json.loads(out.text)

LLM-as-a-Judge Metric (LLMMetric)

Evaluates responses using an LLM judge driven by a prompt template.

Managed Metric Parameters (metric_spec_parameters)

Parameters for a built-in metric, passed through a custom_metrics entry that has no prompt_template and no custom_function. Use a metric ID from the tables above. rubric_group_key picks which of the case's rubric_groups to grade against, and is required only when a case defines more than one.

metrics_to_run:
  - final_response_quality
custom_metrics:
  - name: final_response_quality
    metric_spec_parameters:
      rubric_group_key: case_criteria