skilld
All skills
google avatar

/google-agents-cli-eval

@2c39459
by googlegoogle/agents-cli6k stars
686

This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Agent Platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).

Requires /google-agents-cli-scaffold /google-agents-cli-workflow /google-agents-cli-adk-code /google-agents-cli-deploy /google-agents-cli-observability

Ask your Agent

Use this Skill: https://skilld.dev/gh/google/agents-cli/google-agents-cli-eval

This session only. Nothing lands on disk.

referenceslive-eval.md

≈671 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Evaluating Live agents

--mode adk_live runs an eval over the agent's /run_live WebSocket instead of /run_sse. Pass it on its own for a local autoboot, or alongside the agent's https:// --url when deployed (for Agent Runtime, the full engine URL). The dataset, the trace output, and the eval grade step are unchanged.

# Local autoboot
agents-cli eval generate --mode adk_live

# Deployed agent
agents-cli eval generate --mode adk_live --url https://my-live-agent.run.app --app-name app

# Chain generate + grade
agents-cli eval run --mode adk_live --url https://my-live-agent.run.app --app-name app \
  --metrics final_response_quality

What gets graded

Live replies are audio, and are transcribed by default, so the transcript is what gets graded. Raw audio and video bytes are dropped from the trace.

How cases are played

Each user turn is sent as text over one persistent socket, in order. A single-turn case (prompt) is just a one-turn conversation. History lives in the live session rather than being seeded over HTTP, so the model conditions on the real running conversation, and multi-turn cases produce a trajectory that multi_turn_* metrics can score.

Author user-only turns. The agent generates every reply over the live session, so pre-authored agent turns are ignored (the CLI warns) and are not seeded as history.

Two failures that look like transport bugs

Both let the socket connect and then fail inside the session, so the error points at /run_live when the cause is agent configuration.

Cause Fix
The agent is not on a Live model. --mode adk_live changes only the transport; it cannot make a non-Live agent bidi. The scaffold default is not a Live model, so a fresh project fails with WebSocket code 1011. Switch the agent to a Live model.
On Vertex, the model's region is not pinned. Live models are served from a regional endpoint such as us-central1, not global. Unpinned, the model falls back to GOOGLE_CLOUD_LOCATION, which agents-cli deploy sets to global on Agent Runtime. Pin the region on the model itself rather than steering GOOGLE_CLOUD_LOCATION, which is shared with sessions, telemetry, and grading.

Grading is unaffected by either: eval grade passes its own location (--region, default global), so eval run --mode adk_live chains both steps correctly in one command.

ADK projects. Current Live model IDs, and the code to pin a model's region, are in /google-agents-cli-adk-code (references/adk-python-live.md, "Models"). Transport mechanics are on the same page under "Events" and "Serving and testing a Live agent".

Source: SKILL.md on GitHub

Third-party checks

No alertstoday3 checks · Risk SAFE
  • Gen Agent Trust Hubtoday

    This skill provides comprehensive guidance on using the Agent Platform evaluation framework. It includes security considerations regarding local code execution for custom metrics and data processing, which are managed through documented best practices and platform guardrails.

  • Sockettoday

    No alerts

  • Snyktoday

    Risk: LOW · No issues

Provenance

Signed by skilld at 2c39459. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub 2 days ago.

Activeupdated 2 days ago

Capability

Other metadata
metadata
{
  "author": "Google",
  "license": "Apache-2.0",
  "version": "1.8.0",
  "requires": {
    "bins": [
      "agents-cli"
    ],
    "install": "uv tool install google-agents-cli"
  }
}

README badge

README badge for google/agents-cli/google-agents-cli-eval