/google-agents-cli-eval

Evaluating Agents with google_search and Built-in Tools

ADK naming. Tool classes and code snippets below are ADK's. The behavior generalizes: in any framework, Gemini's built-in grounding, retrieval, and code-execution tools run inside the model and never surface as function calls in a trace.

google_search Behavior (IMPORTANT)

google_search is NOT a regular tool — it's a model-internal grounding feature.

Key behavior:

• Custom tools (save_preferences, save_feedback) → appear as function_call in trajectory
• google_search → NEVER appears in trajectory (happens inside the model)

How google_search works internally:

llm_request.config.tools.append(
    types.Tool(google_search=types.GoogleSearch())  # Injected into model config
)

Search results come back as grounding_metadata, not function call/response events. But the evaluator STILL detects it at the session level:

{
  "error_code": "UNEXPECTED_TOOL_CALL",
  "error_message": "Unexpected tool call: google_search"
}

This causes multi_turn_tool_use_quality to fail for agents whose only tool is google_search — the evaluator flags an unexpected tool call it can never see in the trace. For agents that also call function tools, the metric still scores those function-tool calls (see metric compatibility below).

Metric compatibility for google_search agents:

Dataset best practices for google_search agents:

{
  "eval_cases": [
    {
      "eval_case_id": "news_digest_test",
      "prompt": {
        "role": "user",
        "parts": [{"text": "Give me my news digest."}]
      }
      // NO trajectory criteria for google_search - it won't appear in the trace anyway
    }
  ]
}

For agents that mix google_search with custom function tools, grade the custom tool usage with multi_turn_tool_use_quality — it judges the tool calls in the generated trace, so you don't hand-author expected calls. Optionally add a reference response for reference-based matching:

{
  "eval_case_id": "news_digest_feedback",
  "prompt": {
    "role": "user",
    "parts": [{"text": "Great, save my positive feedback."}]
  },
  "reference": {
    "response": {
      "role": "model",
      "parts": [{"text": "Feedback saved!"}]
    }
  }
}

The google_search invocation still won't appear in the trace, so multi_turn_tool_use_quality only assesses the function-tool calls (e.g., save_feedback).

Config for google_search agents (eval_config.yaml):

metrics_to_run:
  - final_response_quality

The built-in final_response_quality is sufficient for most google_search agents; it auto-generates a content-based rubric. Define a custom override in custom_metrics only if you need project-specific judge instructions — see SKILL.md's Evaluation Configuration Schema for the override pattern.

Bottom line: google_search is a model feature, not a function tool. You cannot test it with trajectory matching. Use final_response_quality to verify the agent produces grounded, cited responses.

ADK Built-in Tools: Trajectory Behavior Reference

Model-Internal Tools (DON'T appear in trajectory):

These inject into llm_request.config.tools as model capabilities:

types.Tool(google_search=types.GoogleSearch())
types.Tool(code_execution=types.ToolCodeExecution())
types.Tool(retrieval=types.Retrieval(...))

Function-Based Tools (DO appear in trajectory):

These generate function_call and function_response events:

types.Tool(function_declarations=[...])

Quick Reference — Can I use multi_turn_tool_use_quality?

• google_search → NO (model-internal)
• code_executor → NO (model-internal)
• VertexAiSearchTool → NO (model-internal)
• url_context → NO (model-internal)
• load_web_page → YES (FunctionTool)
• Custom functions → YES (FunctionTool)

When mixing both types (e.g., google_search + save_preferences):

1. Rely on final_response_quality for overall quality, OR
2. Keep multi_turn_tool_use_quality — it assesses the function-tool calls that do appear in the trace, accepting that the google_search step is invisible to it

Rule of Thumb:

• If a tool provides grounding/retrieval/execution capabilities built into Gemini → model-internal, won't appear in trajectory
• If it's a Python function you can call → appears in trajectory

Model thinking mode may bypass tools

Models with "thinking" enabled may decide they have sufficient information and skip tool calls. Force tool usage through the model's tool-choice config (ADK: tool_config with mode="ANY"), or switch to a non-thinking model for predictable tool calling.

Mock mode for external APIs

When your agent calls external APIs, add mock mode so evals can run without real credentials:

def call_external_api(query: str) -> dict:
    api_key = os.environ.get("EXTERNAL_API_KEY", "")
    if not api_key or api_key == "dummy_key":
        return {"status": "success", "data": "mock_response"}
    # Real API call here