Tool Calling, Safety Filters, Lifecycle, Latency, and Cost
Root causes and fixes for malformed or looping tool calls, duplicate
NON_BLOCKING speech, early-acknowledgement ("filler") misfires, prompt/context
contamination, input safety false positives, WebSocket 1008/1011/goAway
drops, 429 throttling, latency (TTFT) triage, and multi-turn audio token
cost optimization.
Table of Contents
- MALFORMED_FUNCTION_CALL on Complex Tool Arguments: Lines 24-58
- Redundant Tool Loops and Duplicate NON_BLOCKING Speech: Lines 60-121
- State-Machine Polarity Traps and Unbounded Routing Loops: Lines 123-149
- Pre-Tool Early Acknowledgement (Filler) Misfires: Lines 151-172
- Test Prompt Ignored or Model Invents Unseen Data: Lines 174-201
- Undeclared Google Search Deadlock and Mid-Session SI Updates: Lines 203-239
- Safety Refusals on custom_vocabulary: Lines 241-258
- WebSocket 1008, 1011, goAway Drops, and 429 Throttling: Lines 260-290
- Latency (TTFT) Triage: Lines 292-315
- Multi-Turn Audio Token Cost Optimization: Lines 317-336
MALFORMED_FUNCTION_CALL on Complex Tool Arguments
Symptom: Tool calls fail with MALFORMED_FUNCTION_CALL or dropped quotes /
closing brackets, sometimes accompanied by stray reasoning text.
Root Cause: Declaring a tool with a single STRING parameter (for example,
query_dsl: STRING or payload: STRING) and instructing the model inside
system_instruction to format a quote-heavy JSON or key-value mini-DSL string
causes the model to drop closing quotes or delimiters during real-time speech
generation.
Fix
Always declare explicit typed schema properties (OBJECT, ARRAY, ENUM,
BOOLEAN, INTEGER) in functionDeclarations[].parameters (or
parametersJsonSchema) instead of encoding a mini-DSL inside a STRING.
{
"name": "search_flights",
"description": "Search available flights for a route and cabin.",
"parameters": {
"type": "OBJECT",
"properties": {
"origin": { "type": "STRING" },
"destination": { "type": "STRING" },
"cabin": { "type": "STRING", "enum": ["ECONOMY", "BUSINESS", "FIRST"] },
"nonstop_only": { "type": "BOOLEAN" }
},
"required": ["origin", "destination"]
}
}Redundant Tool Loops and Duplicate NON_BLOCKING Speech
Symptom:
- The model invokes the same tool repeatedly with identical arguments in a loop.
- With
NON_BLOCKINGtools, the assistant speaks a response while the tool runs and then speaks a second, duplicate answer whentoolResponsearrives.
Root Cause:
- State-machine bloat & contradictory directives: Multi-hop deterministic
state machines inside large
system_instructionprompts combined with conflicting rules ("ALWAYS call tools" vs. "Only one tool call per turn") push real-time native-audio models into repetitive tool loops. - Default
FunctionResponsescheduling: Returning aNON_BLOCKINGFunctionResponsewithoutscheduling="SILENT"instructs the server to trigger a new model generation turn when the response arrives, causing a duplicate spoken reply.
Fix
- Move deterministic state transitions and retry counters into application code, and expose only the tools valid for the current conversation step.
- Set
scheduling="SILENT"on transactional or backgroundFunctionResponsepayloads when the model has already acknowledged the action and only needs the result stored in context. (Note:gemini-3.8-live-extended-thinking-previewdoes not support explicitFunctionResponse.scheduling.)
{
"toolResponse": {
"functionResponses": [{
"id": "call_1",
"name": "save_preference",
"scheduling": "SILENT",
"response": { "status": "saved", "retryable": false }
}]
}
}# Equivalent Python google-genai SDK call
await session.send_tool_response(
function_responses=[
types.FunctionResponse(
id=call.id,
name=call.name,
response={"status": "saved", "retryable": false},
scheduling=types.FunctionResponseScheduling.SILENT,
)
]
)- For small reference lists (
<= 50items), include a compact lookup table insystem_instructionor resolve entities via client-side retrieval before injecting results.
State-Machine Polarity Traps and Unbounded Routing Loops
Symptom: In branching voice workflows (such as triage or confirmation
flows), the assistant misroutes when the user says "No", loops indefinitely
on ambiguous questions or unintelligible fragments, or leaks raw JSON routing
blocks.
Root Cause:
- Question polarity inversion: Asking
"Do you want to cancel your order?"("No"= keep order) followed immediately by"To confirm, you want to keep your order, correct?"("No"= cancel order) without explicitly mapping"No"on the confirmation step causes the model to carry over the initial"No"intent. - Unbounded Q&A / forbidden same-state routing: Prohibiting a routing tool from targeting the current state without a turn cap causes infinite loops or raw JSON fallback dumps when callers stay off-topic.
Fix
- Keep confirmation question polarity consistent across turns, or explicitly specify both affirmative and negative branch targets in the tool response instructions.
- Enforce a bounded retry limit (for example, re-prompt once on unclear input, then escalate or transfer on the second consecutive unclear turn).
Pre-Tool Early Acknowledgement (Filler) Misfires
Symptom: Attempting to use natural-language system_instruction rules to
make the model speak a filler phrase ("Let me look that up") before slow
tools (3-5s) while remaining silent before fast utility tools (< 1s) causes
frequent false-positive fillers on fast tools, silent execution on slow tools,
or double spoken sentences (empathy + filler) before a single tool call.
Root Cause: Natural-language filler rules inside large system_instruction
prompts cannot reliably discriminate tool latency tiers at decode time.
Fix
- Silent tools + client UI cue (recommended): Instruct the model to
invoke tools immediately without spoken filler, and play a lightweight
client-side chime or visual indicator while awaiting
toolResponse. - Uniform single-sentence acknowledgement: If spoken acknowledgement is
required before slow operations, keep it uniform rather than writing
per-tool latency rules in
system_instruction, and combine any empathy and waiting phrase into a single short sentence.
Test Prompt Ignored or Model Invents Unseen Data
Symptom:
- Replacing
system_instructionwith a 1-line echo test prompt still causes the assistant to run the production workflow. - The assistant states user details (account tiers, dates, names) that are not in the prompt template and were never spoken by the user.
Root Cause:
- Latent tool schemas: Leaving production
toolsattached inBidiGenerateContentSetupacts as latent workflow instructions that outweigh a minimal 1-linesystem_instruction. - Client framework context contamination: Agent frameworks or middleware
often serialize extra session-state attributes (templates, script buffers,
user profile objects) into the assembled setup or
clientContentpayload without appearing in the raw prompt file.
Fix
- Remove or stub out
toolswhen testing minimal or echo prompts in isolation. - Log and inspect the complete serialized wire JSON (
setup,clientContent,toolResponse) actually sent over the WebSocket rather than auditing only source prompt templates.
Undeclared Google Search Deadlock and Mid-Session SI Updates
Symptom:
- Sessions stall for
8-10suntil idle timeout when the prompt demands search grounding. - Mid-session
clientContentattempts to update instructions fail, trigger an unsolicited model reply, or erase earlier rules.
Root Cause:
- Instructing "Ground answers to Google Search ONLY" in
system_instructionwithout declaring{"googleSearch": {}}insetup.toolscauses the model to attempt an undeclared search call and stall waiting for a response. Note that on Live API,googleSearchcannot be combined withfunctionDeclarationsin the samesetup.toolsconfiguration. gemini-3.8-liverejectsrole="system"inclientContent(mid-session instruction and tool updates must usecontextUpdate, which replacessetup.systemInstructionin full). On2.5models, sendingturnComplete: trueon a context update immediately triggers an unsolicited spoken turn rather than quietly updating instructions for the next user utterance.
Fix
- Either declare
{"googleSearch": {}}alone insetup.tools(when nofunctionDeclarationsare used), remove the Google Search mandate fromsystem_instruction, or expose search behind a custom server-executed function declaration when combining search with other custom tools. - On
gemini-3.8-live, send acontextUpdate(BidiGenerateContentContextUpdate) frame carrying the complete replacementsystemInstruction(and/or updatedtools). On2.5models, reconnect or resume a session with the updatedsetup.systemInstructionat a turn boundary rather than sendingrole="system"orturnComplete: trueinclientContent.
Safety Refusals on custom_vocabulary
Symptom: Spoken queries containing domain terms registered in
inputAudioTranscription.customVocabulary trigger an unexpected canned safety
refusal ("I'm just a language model and can't help with that."), while the
exact same audio passes when customVocabulary is omitted.
Root Cause: Concatenating multi-word domain phrases into synthetic
single-token compounds in customVocabulary can distort speech-recognition
subword tokenization and trigger false-positive input safety blocks.
Fix
Keep multi-word phrases in inputAudioTranscription.customVocabulary
space-delimited as natural words rather than concatenating synthetic
single-token compounds.
WebSocket 1008, 1011, goAway Drops, and 429 Throttling
Symptom: Sessions terminate mid-call with WebSocket close codes 1008
(Policy Violation), 1011 (Internal Error), goAway, or fail to connect during
traffic spikes with HTTP 429 RESOURCE_EXHAUSTED.
Root Cause:
1008on setup: Caused by unsupportedgenerationConfigfields in Live API (responseSchema,responseMimeType,responseLogprobs,logprobs,stopSequence,routingConfig,audioTimestamp), an unknown or case-mismatchedvoiceName, or invalidtoolResponsepayloads.1008/goAwaymid-call: Hitting the per-connection Live session duration limit (~10-15 minutes) without session resumption enabled.1011/429: Transient upstream disconnects without resumption, or unsmoothed outbound dialing / connection bursts exceeding regional QPM/RPM.
Fix
- Audit
setup.generationConfigagainst supported Live API fields, and verify case-sensitivevoiceNamespelling. - Enable transparent session resumption (
sessionResumption: { transparent: true }), tracksessionResumptionUpdate.newHandleandlastConsumedClientMessageIndex, and reconnect automatically ongoAwayand close codes1000,1006, or1011. Follow the Session Manager reference (references/session_manager.md) linked fromSKILL.mdfor the full buffering and replay state machine. - Smooth connection bursts with jittered exponential backoff on
429, and evaluate Provisioned Throughput for predictable high-concurrency workloads.
Latency (TTFT) Triage
Symptom: Time-to-first-audio-byte (TTFT) spikes from < 1.5s to 6-12s,
causing idle timeouts.
Root Cause: Multi-second delays fall into three distinct buckets:
- Client 3P tool latency: Gap between receiving
toolCalland sendingtoolResponse. - User barge-in overlap: Caller resumes speaking before the server turn finishes.
- Regional backend / prefix-cache latency: Delay between
activityEnd/turnComplete(ortoolResponse) and the firstserverContentchunk with no pending client tool calls.
Fix
Log monotonic timestamps at activityEnd (or clientContent), toolCall,
toolResponse, and first serverContent audio chunk. Optimize slow client 3P
APIs (or mark non-critical calls NON_BLOCKING with scheduling="SILENT"). If
server latency spikes at flat QPS with no client tool delay, test an alternate
supported region and share session IDs + UTC timestamps with Google Cloud
Support.
Multi-Turn Audio Token Cost Optimization
Symptom: Multi-turn Live API call costs grow superlinearly as session duration increases.
Root Cause: Live API retains prior model audio output tokens 1-for-1 as
audio input context across subsequent turns in the same session, and streams
continuous background audio when TURN_INCLUDES_ALL_INPUT is active.
Fix
- Configure
contextWindowCompressionsliding windows to cap retained audio history. - Rotate/reset long sessions at natural conversation boundaries.
- Set
realtimeInputConfig.turnCoverage = "TURN_INCLUDES_ONLY_ACTIVITY"so long silent intervals are not retained in turn context. - Offload pure text/data post-processing (summarization, classification) to
non-audio
generateContentcalls. - Review current rates and calculators on the Google Cloud Pricing page.