[!IMPORTANT] Gemini Enterprise Agent Platform was previously named "Vertex AI"; the API service endpoint remains
aiplatform.googleapis.com.
LiveAPI Service Skill
This skill provides instructions for generating a LiveAPI client service
class that connects to the Gemini Enterprise Agent Platform Live API over
WebSockets (LlmBidiService/BidiGenerateContent), as well as diagnosing and
resolving Live API runtime issues. The generated client handles bidirectional
JSON frame streaming, bearer-token authentication via Application Default
Credentials (ADC), transparent session resumption, tool call responses,
mid-session context updates, and ClientMessage / ServerMessage proto
exchange.
The skill also produces a demo frontend + backend service so the user can interactively validate the generated client (text, audio, video, transcription, and interrupt handling).
Usage Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read and follow the instructions in
Usage Attribution.
Prerequisites
Before running the generation flow, verify or guide the user through the following prerequisites (always ask for user approval before modifying host configurations or installing system tools):
Google Cloud Project & Billing: An active Google Cloud project with a linked billing account (review Gemini Enterprise Agent Platform pricing):
gcloud config set project {project_id}API Enablement & IAM: The
aiplatform.googleapis.comAPI enabled on the project, and the caller principal granted at least Vertex AI User (roles/aiplatform.user):gcloud services enable aiplatform.googleapis.comApplication Default Credentials (ADC): Configured on the host running the generated client (see Google Cloud authentication):
gcloud auth application-default loginRegional Location: Default to
us-central1unless the user specifies another supported US or EU region (e.g.,europe-west4; see supported locations). Live API models do not support theglobalendpoint.Isolated Output Directory: A destination folder supplied by the user (e.g.,
/tmp/liveapi_out) where generated code, virtual environments, and the demo service will be written. Never mutate the host's system Python environment.Implementation Language: Python is the default and reference language when the user does not specify another language.
Supported Live API Models
gemini-3.8-live(GA — Recommended default): Native audio I/O, VAD, affective dialog and proactive audio always enabled by default,NON_BLOCKINGandBLOCKINGfunction calling, mid-sessioncontext_update, and custom transcription vocabulary.gemini-live-2.5-flash-native-audio(GA): Native audio I/O, transcriptions, VAD, optionalgeneration_config.enable_affective_dialogandproactivity.proactive_audio, and function calling.gemini-3.5-transcribe-live-preview(Preview): Dedicated low-latency speech-to-text streaming (input_transcriptionandinterim_input_transcription) withVERBATIMandSMARTmodes and custom vocabulary.
Reference Files
Load these files from references/ on demand (do not treat them as
standalone skills):
- client_server_messages.md: Wire
protocol reference for
ClientMessage/ServerMessage, model-specificgemini-3.8-liverules, and JSON examples. - client_server_messages.proto:
Proto3 schema derived from the public Live API documentation for generating
typed
ClientMessageandServerMessageclasses. - session_manager.md: Specification for
connection lifecycle, 1-based message buffering,
go_awayhandling, and transparent session resumption. - Audio, Voice, Accent, and Quality Troubleshooting: Prebuilt voice gender table, accent/locale fixes, amplitude collapse, tonal humming, and jitter buffering.
- Turns, VAD, Barge-In, and Transcriptions Troubleshooting:
generationCompletevs.turnComplete,contextWindowCompressionreadback settings, client stall nudges, backchannel gating, telephony VAD padding, and transcription/instruction leakage guards. - Tool Calling, Safety Filters, Lifecycle, Latency, and Cost Troubleshooting:
Typed schemas,
NON_BLOCKINGscheduling, state-machine polarity, pre-tool filler handling, safety filters, WebSocket1008/1011andgoAwayrecovery,TTFTtriage, and cost optimization.
Steps
Step 1: Copy the reference files
Copy client_server_messages.md, client_server_messages.proto, and
session_manager.md from this skill's references/ folder into the user's
destination output folder. These files become the local source of truth for the
generated client.
Step 2: Reconcile and compile the proto definitions
Check the public documentation linked in
client_server_messages.md. If any new
server fields are needed for the user's task, update the copied
client_server_messages.md and client_server_messages.proto in the
destination folder, and compile client_server_messages.proto (e.g., via
grpc_tools.protoc or protoc) into the destination folder.
Step 3: Implement the client class and SessionManager
Implement a client service class in the user's chosen language that:
- Imports the local compiled
client_server_messages.prototypes (ClientMessage,ServerMessage). - Connects over WebSockets to:
wss://{location}-aiplatform.googleapis.com/ws/google.cloud.aiplatform.{api_version}.LlmBidiService/BidiGenerateContentwhereapi_versiondefaults to"v1"(allow overriding to"v1beta1"for preview-only features such as Vertex RAG Engine). - Formats
setup.modelasprojects/{project_id}/locations/{location}/publishers/google/models/{model_id}. - Serializes outgoing
ClientMessageinstances as UTF-8 JSON text frames (e.g.,google.protobuf.json_format.MessageToJson) and parses incoming JSON text frames intoServerMessagewith unknown fields ignored (ignore_unknown_fields=True). Never send binary protobuf bytes. - Implements transparent session resumption per
session_manager.md (including
context_window_compression.sliding_windowwhen sessions may exceed 15 minutes).
For languages that require an isolated runtime (e.g., Python), create an
isolated environment (e.g., venv) inside the destination folder along with
a requirements.txt pinning exact compatible versions within tested bounds
(websockets>=13.0,<16.0, protobuf>=5.29.0,<7.0.0,
google-auth>=2.35.0,<3.0.0, requests>=2.32.0,<3.0.0,
grpcio-tools>=1.68.0,<2.0.0) and a runnable setup.sh script that creates the
venv, runs pip install -r requirements.txt (explicitly referencing
requirements.txt), and compiles client_server_messages.proto. Never
install into the system interpreter or user-global site-packages, and never
instruct the user to run sudo pip install.
Initialization parameters
Accept at construction time:
project_id: Google Cloud project ID.location: Regional location (e.g.,"us-central1"; reject"global").model_id: Live API model ID (e.g.,"gemini-3.8-live").config: AClientMessagewith thesetupfield populated.api_version(optional): Defaults to"v1"(supports"v1beta1").
Authentication
Obtain an access token via Application Default Credentials with scope
https://www.googleapis.com/auth/cloud-platform, attach it to the WebSocket
handshake header as Authorization: Bearer <token>, refresh it before or upon
expiry, and attach the valid/refreshed token on every reconnection (including
go_away and close codes 1000, 1006, or 1011). Do not hard-code a
long-lived API key as the only auth mechanism.
Public async API
Expose the following async methods, all gated on receipt of a setup_complete
ServerMessage before sending:
send_realtime_data(data: ClientMessage): Send continuous low-latency media/text.dataMUST be typed asClientMessagecarryingrealtime_input.send_client_content(data: ClientMessage): Send discrete, history-bearing turns.dataMUST be typed asClientMessagecarryingclient_content. (Ongemini-3.8-live, setsetup.history_config.initial_history_in_client_content = truewhen seeding prior history, and never userole="system"inclient_content.)send_tool_response(data: ClientMessage): Reply to a servertool_call.dataMUST be typed asClientMessagecarryingtool_response. (Ongemini-3.8-live, return informative status objects such as{"status": "ok"|"not_found", "retryable": false, ...}instead of empty{}to prevent retry sweeps.)send_context_update(data: ClientMessage): Updatesystem_instructionand/ortoolsmid-session (gemini-3.8-live).dataMUST be typed asClientMessagecarryingcontext_update.receive() -> AsyncIterator[ServerMessage]: Async generator yieldingServerMessageinstances parsed from the WebSocket stream.
Do not expose synchronous blocking variants as the primary API surface.
Step 4: Write a test file
Generate a test file in the destination folder that initializes the client and
exercises text, audio (16 kHz PCM), video (sampled JPEG/PNG frames),
send_tool_response, and send_context_update, verifying ServerMessage
responses. Ask the user for any runtime inputs needed (project_id, model, or
sample media).
Step 5: Generate how_to_run.md
Write how_to_run.md in the destination folder documenting the generated client
class, setup scripts, and complete examples building ClientMessage payloads
for realtime audio/video/text, client_content turns (including seeded
history), tool_response (async scheduling and structured responses), and
context_update.
Step 6: Generate a demo frontend + backend service
Create a frontend UI and backend service in the destination folder that reuse
the local ClientMessage / ServerMessage protos from Step 1. Through the UI
the user must be able to:
- Open and close a Live API session.
- Select the model (defaulting to
gemini-3.8-live). - Stream microphone audio (16 kHz mono PCM) and camera or screen frames (~1 fps JPEG) to the model.
- Send typed text messages.
- Play back 24 kHz model audio and view interleaved user and model transcriptions.
Follow Live API best practices for audio capture, echo cancellation, and playback buffering.
Handling the interrupt signal
When server_content arrives with interrupted: true, the UI MUST:
- Keep played audio and rendered transcription time-aligned.
- Immediately stop currently playing model audio and stop appending to the in-progress model transcription bubble.
- Clear unplayed audio buffers and queued unrendered transcription text so stale content never bleeds into the next turn.
- Start fresh chat bubbles for subsequent user and model turns.
Handling the transcription finished signal
For streamed input_transcription and output_transcription chunks (and
optional interim_input_transcription previews), append text to the active
bubble while finished is unset/false, and close that bubble to start a fresh
one when finished: true arrives. Route input_transcription to user-role
bubbles and output_transcription to model-role bubbles.
Step 7: Generate how_to_test_with_ui.md
Write how_to_test_with_ui.md in the destination folder with:
- Exact shell command(s) to start the backend service.
- Exact shell command(s) to start the frontend UI.
- Browser URL (
http://localhost:PORT). - Walkthrough for starting a session, selecting a model, streaming mic/camera/ screen inputs, sending text, and verifying audio + transcription + barge-in.
Validation Checklist
Verify every item before completing generation:
-
client_server_messages.md,client_server_messages.proto, andsession_manager.mdwere copied into the destination folder. - The generated client compiles and imports the local
ClientMessageandServerMessageproto types, and exchanges UTF-8 JSON text frames (MessageToJson/Parsewith unknown fields ignored). - The client connects to
wss://{location}-aiplatform.googleapis.com/ws/google.cloud.aiplatform.{api_version}.LlmBidiService/BidiGenerateContentdefaultingapi_versionto"v1"on a regional endpoint (neverglobal), and formatssetup.modelasprojects/{project_id}/locations/{location}/publishers/google/models/{model_id}. - Authentication uses ADC bearer tokens (
Authorization: Bearer <token>), refreshes before expiry, and reattaches the valid token on every reconnect. - Public async methods
send_realtime_data,send_client_content,send_tool_response,send_context_update, andreceiveare present, typed withClientMessage/ServerMessage, and gated onsetup_complete. - Transparent session resumption sets
session_resumption.transparent = true, tracksnew_handle, indexes buffered messages starting at1, prunes vialast_consumed_client_message_index, and replays from index1on reconnect (go_awayand WebSocket codes1000/1006/1011). - For Python, an isolated
venv,setup.sh, andrequirements.txtlive inside the destination folder with zero changes to system/global Python. -
how_to_run.mdandhow_to_test_with_ui.mdexist, and the demo UI implementsinterrupted: trueandTranscription.finishedhandling. - The client does not target
generativelanguage.googleapis.comand does not pass API keys in URL query strings.
Troubleshooting Live API Issues
When debugging an existing Live API integration or runtime failure (rather than scaffolding a new client from scratch), follow this diagnostic workflow before making prompt tweaks.
Diagnostic Workflow
- Verify environment and model currency: Confirm a regional endpoint
(for example,
us-central1;globalis not supported for Live API WebSockets) and active billing (Google Cloud Pricing). Legacygemini-2.0-*models andgemini-live-2.5-flash(half-cascade) are deprecated or shut down; test whether the symptom resolves ongemini-3.8-liveorgemini-live-2.5-flash-native-audio. - Capture the full serialized wire payload: Log the exact
BidiGenerateContentSetupJSON,clientContent,realtimeInput,toolResponse, andcontextUpdateframes actually sent over the WebSocket. Frameworks often inject extra session state or tool declarations absent from source prompt files. - Audit against Pre-Flight Configuration Rules: Fix primary structural
defects (gender mismatch, missing regional
languageCode, lowtemperaturegenerationConfig.enableAffectiveDialog, bloatedtoolsorsystemInstruction) first.
- Isolate optional flags and tools: On
gemini-live-2.5-flash-native-audio, temporarily disablesetup.proactivity.proactiveAudioandsetup.generationConfig.enableAffectiveDialog(both are built-in by default ongemini-3.8-liveand should be omitted there), and stub outtoolswhen testing minimal prompt behavior. - Reproduce and verify A/B: Replay the failing turn sequence before and
after 1-2 targeted changes, checking audio duration, PCM amplitude,
output_transcription, andturnCompletesignals.
Quick Symptom Triage Matrix
| Symptom | Primary Suspect | Immediate Fix | Reference |
|---|---|---|---|
| Voice flips gender (Turn 1 or mid-call) | Prebuilt voice gender mismatches SI persona name/pronouns | Match voiceName gender to SI persona |
Audio & Voice |
Accent reverts to en-US, pitch jumps, or code-mix fails |
Omitted speechConfig.languageCode or long single-script SI |
Set supported BCP-47 languageCode (e.g., en-IN) and concise SI |
Audio & Voice |
| Output mimics user's voice timbre | Native audio role leakage from user audio in context | Upgrade to gemini-3.8-live; clarify role boundaries; reconnect |
Audio & Voice |
| Gradual voice/tone drift over long calls | Audio-token conditioning drift as history grows | Enable contextWindowCompression and rotate long sessions |
Audio & Voice |
Fast speech (240+ WPM) or raw /IPA/ leaks |
Long single-breath clauses, 2.5 affective dialog, or TTS markup in SI |
Shorten clauses, strip TTS headers/IPA, or use gemini-3.8-live |
Audio & Voice |
Near-zero audio volume (-34 dBFS), intact text |
Tool schemas >40k tokens or rolling text logs in SI |
Prune tools to active turn/step, drop rolling text logs, keep SI <2,000 tokens |
Audio & Voice |
150-700 Hz tonal hum or endless greeting |
temperature <= 0.3 with 2.5 generationConfig.enableAffectiveDialog |
Set temperature >= 0.7, disable 2.5 affective dialog, open fresh session |
Audio & Voice |
Choppy playback with 0.1-0.5s gaps |
Small variable WebSocket audio chunks played raw | Add a ~200 ms client-side jitter buffer |
Audio & Voice |
Premature generationComplete (~0.75s) or missing turnComplete |
Advancing on generationComplete, <verbatim> tags, or history bloat |
Wait for turnComplete, strip <verbatim> tags, add contextWindowCompression |
Turns & Transcripts |
0 audio chunks after toolResponse (>14k tokens) |
XML/JSON bloat diluting voice conditioning | Use prose SI, clean JSON tools, and contextWindowCompression |
Turns & Transcripts |
Empty turns (output_audio=0) and 12-30s wedges |
2.5 proactivity.proactiveAudio or client 3s stall nudges |
Disable proactivity.proactiveAudio on 2.5; remove 3s clientContent nudges |
Turns & Transcripts |
Turn-1 output_transcription dropped or silence placeholder leak |
realtimeInput.text kickoff under TURN_INCLUDES_ALL_INPUT |
Use TURN_INCLUDES_ONLY_ACTIVITY and clientContent(turnComplete=True) |
Turns & Transcripts |
Unspoken Markdown or code in output_transcription |
Post-speech text flush into UI / persisted transcripts | Set languageCode, ban Markdown in SI, strip trailing code blocks in UI |
Turns & Transcripts |
| Audible instruction leak pre-tool without transcript | Spoken pre-tool filler in >70k-token context |
Call tools silently, strip "speak..." wrappers, compress context |
Turns & Transcripts |
Model speaks <no speech>, {pause}, or raw JSON |
NON_BLOCKING missing scheduling="SILENT", echo, or routing trap |
Set scheduling="SILENT", enable AEC, cap unclear turns |
Turns & Transcripts |
Short "Mm-hmm" cuts off speech; double interrupt stalls |
Acoustic barge-in firing on brief backchannels | Filter <500 ms backchannels client-side before streaming |
Turns & Transcripts |
Final digits clipped ("98058" -> "98050") |
Manual VAD closing too fast on 8 kHz PSTN audio | Add 400-500 ms hangover padding; confirm critical digits |
Turns & Transcripts |
| New turn plays over old turn after barge-in | Server finished fast without emitting interrupted: true |
Flush client playback buffer on new turn start and on interrupted |
Turns & Transcripts |
MALFORMED_FUNCTION_CALL on complex args |
Single STRING parameter used as mini-DSL |
Use typed OBJECT, ARRAY, ENUM, BOOLEAN properties |
Tools & Lifecycle |
Tool loops or duplicate reply on NON_BLOCKING tool |
State machine in SI or missing scheduling="SILENT" |
Move state machine to app layer; set scheduling="SILENT" |
Tools & Lifecycle |
| Unreliable pre-tool filler on fast vs. slow tools | Per-latency-tier natural-language filler rules in SI | Call tools silently and play a client-side UI/audio cue while waiting | Tools & Lifecycle |
| Minimal test prompt ignored or agent invents data | Production tools left attached or framework context leak |
Stub tools in tests; audit full serialized wire JSON |
Tools & Lifecycle |
| Stall on search grounding or ignored SI update | Undeclared googleSearch (incompatible with custom functions) or clientContent SI update |
Use googleSearch alone or custom search tool; use contextUpdate on 3.8-live |
Tools & Lifecycle |
Safety refusal on custom_vocabulary |
Synthetic concatenated single-token compounds distorting ASR | Keep multi-word phrases space-delimited in custom_vocabulary |
Tools & Lifecycle |
WebSocket 1008 / 1011 / goAway or 429 |
Unsupported generationConfig, timeout, or burst traffic |
Audit setup fields, enable sessionResumption, backoff on 429 |
Tools & Lifecycle |
6-12s TTFT spike or high multi-turn audio cost |
Client 3P tool delay, regional queue, or 1:1 audio token retention | Log per-stage timestamps; compress/rotate sessions; check pricing | Tools & Lifecycle |
Pre-Flight Configuration Rules
- Align voice gender with persona gender: Match prebuilt
voiceNamegender withsystemInstructionpersona gender so acoustic and text conditioning do not compete. - Set
speechConfig.languageCodewhen pinning a regional accent or default language: While optional for dynamic multilingual auto-detection, omitting it defaults the initial voice reference toen-USand overrides text accent instructions. Use a supported BCP-47 code (such asen-INores-US). - Keep
temperature >= 0.7(default1.0): Values<= 0.3(especially when paired withgenerationConfig.enableAffectiveDialogon2.5models) trap native-audio sampling in tonal humming loops. - Keep
systemInstruction< 2,000tokens and prunetoolsper turn: Rolling text transcripts in SI and>40,000-token tool declarations dilute voice conditioning, causing-34 dBFSamplitude collapse and0-audio turns. - Write plain-spoken prose without markup: Strip
#Markdown headers,<verbatim>/<taskflow>XML, Gemini-TTS# Director's Notes, raw IPA/slashes/, and"""code delimiters so unspoken code or markup does not leak intooutput_transcription. - Separate prompt template blocks with
\n: Avoid concatenating adjacent template sections onto a single run-on line without newline separators. - Declare typed tool schemas: Use
OBJECT,ARRAY,ENUM, andBOOLEANinstead of packing mini-DSLs into a singleSTRINGargument. - Set
scheduling="SILENT"on transactionalNON_BLOCKINGresponses: Prevents duplicate speech or<no speech>leaks after async tool results (note:gemini-3.8-live-extended-thinking-previewdoes not support explicitFunctionResponse.scheduling). - Wait for
turnComplete, notgenerationComplete: Prevents sequentialclientContentturns from self-interrupting active playback pacing. - Enable
contextWindowCompressionandsessionResumption: Prevents long-session audio drop/drift, bounds multi-turn audio token retention cost, and recovers cleanly acrossgoAway,1000,1006, and1011disconnects.