Index of every page on this site: llms.txt
All skills
firebase avatar

/firebase-ai-logic-basics

@99954a3 official
by firebasefirebase/agent-skills520 stars
116
Run it yourself

Integrates Firebase AI Logic (formerly Vertex AI for Firebase) into mobile and web apps (Android, iOS, Web, and Flutter) to call Gemini models from client code. Use when provisioning or initializing Firebase AI Logic, or adding Gemini features to an app: text generation, multimodal input, chat, streaming, structured output, hybrid on-device inference, App Check protection, or switching models with Remote Config.

Use this Skill: https://skilld.dev/gh/firebase/agent-skills/firebase-ai-logic-basics

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencestts_generation.md

≈5.6k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Client-Side Text-to-Speech (TTS) Generation with Gemini

Firebase AI Logic enables client-side Text-to-Speech (TTS) generation directly from your mobile and web applications without maintaining custom backend speech services. Using specialized Gemini TTS models, apps can synthesize natural, expressive audio with custom voice personas, multi-speaker dialogues, and in-flight audio streaming.


Supported Models

Always specify a dedicated Gemini TTS model when generating audio:

Model ID Description
gemini-3.1-flash-tts-preview Low-latency speech synthesis (preview); supports single- and multi-speaker output.

[!WARNING] Always verify currently supported model availability in the Firebase AI Logic Models documentation.


Core Concepts & Configurations

1. Response Modality

To instruct Gemini to synthesize and return audio, configure responseModalities in the generation configuration to include audio:

  • Swift: generationConfig = GenerationConfig(responseModalities: [.audio])
  • Kotlin: generationConfig { responseModalities = listOf(ResponseModality.AUDIO) }
  • Web (JS/TS): generationConfig: { responseModalities: ["AUDIO"] }

2. Single-Speaker SpeechConfig

Configure voice persona and optional language code (languageCode, e.g. "en-US", "es-ES", "ja-JP") using SpeechConfig. Voice names map to Google prebuilt neural voices (for example: Puck, Charon, Kore, Fenrir, Aoede):

  • Swift:
    let speechConfig = SpeechConfig(
        voiceConfig: VoiceConfig(
            prebuiltVoiceConfig: PrebuiltVoiceConfig(voiceName: "Puck")
        ),
        languageCode: "en-US"
    )
  • Kotlin:
    val speechConfig = speechConfig {
        voiceConfig = voiceConfig {
            prebuiltVoiceConfig = prebuiltVoiceConfig {
                voiceName = "Puck"
            }
        }
        languageCode = "en-US"
    }
  • Web:
    speechConfig: {
      voiceConfig: {
        prebuiltVoiceConfig: {
          voiceName: "Puck"
        }
      },
      languageCode: "en-US"
    }

3. Multi-Speaker MultiSpeakerVoiceConfig (2-Speaker Dialogues)

For scripts with two distinct characters or roles (e.g., host and guest, narrator and character), configure MultiSpeakerVoiceConfig with speakerVoiceConfigs linking speaker names to specific voices:

  • Swift:
    let multiSpeakerConfig = SpeechConfig(
        multiSpeakerVoiceConfig: MultiSpeakerVoiceConfig(
            speakerVoiceConfigs: [
                SpeakerVoiceConfig(
                    speaker: "Host",
                    voiceConfig: VoiceConfig(
                        prebuiltVoiceConfig: PrebuiltVoiceConfig(voiceName: "Puck")
                    )
                ),
                SpeakerVoiceConfig(
                    speaker: "Guest",
                    voiceConfig: VoiceConfig(
                        prebuiltVoiceConfig: PrebuiltVoiceConfig(voiceName: "Aoede")
                    )
                )
            ]
        )
    )
  • Kotlin:
    val multiSpeakerConfig = speechConfig {
        multiSpeakerVoiceConfig = multiSpeakerVoiceConfig {
            speakerVoiceConfigs = listOf(
                speakerVoiceConfig {
                    speaker = "Host"
                    voiceConfig = voiceConfig {
                        prebuiltVoiceConfig = prebuiltVoiceConfig { voiceName = "Puck" }
                    }
                },
                speakerVoiceConfig {
                    speaker = "Guest"
                    voiceConfig = voiceConfig {
                        prebuiltVoiceConfig = prebuiltVoiceConfig { voiceName = "Aoede" }
                    }
                }
            )
        }
    }
  • Web:
    speechConfig: {
      multiSpeakerVoiceConfig: {
        speakerVoiceConfigs: [
          {
            speaker: "Host",
            voiceConfig: { prebuiltVoiceConfig: { voiceName: "Puck" } }
          },
          {
            speaker: "Guest",
            voiceConfig: { prebuiltVoiceConfig: { voiceName: "Aoede" } }
          }
        ]
      }
    }

Audio Directives and Emotional Tags

Gemini TTS models accept natural-language directives and inline tags in prompts to control tone, cadence, pacing, and emotional expression.

Directives

Include directives at the start of your prompt to set the overall tone and scene:

  • [Audio Profile: Warm, conversational podcast co-host with energetic delivery]
  • [Scene: Quiet bedtime story in a calm room]
  • [Director's Note: Speak deliberately, pausing for reflection after major questions]

Emotional Tags & Delivery Cues

Embed delivery markers inline within dialogue or narration:

  • [whispers] — Soft, confidential whisper.
  • [laughs] / [chuckles] — Lighthearted laughter during delivery.
  • [sighs] — Expressive exhale.
  • [slowly] — Reduced tempo for emphasis or dramatic effect.
  • [excited] — Higher energy and faster pacing.
  • [pause] — Deliberate silence.

Multi-Speaker Prompt Example

[Scene: A lively coffee-shop tech discussion]
[Director's Note: Host is upbeat and curious; Guest is thoughtful and explanatory]

Host: Welcome back! [excited] Today we are diving into client-side audio.
Guest: [laughs] It's about time! No servers in between means virtually zero latency.
Host: [whispers] Tell us the secret... how does it stream?
Guest: [slowly] Pure linear PCM, chunk by chunk, directly to your device's audio engine.

Audio Decoding and Playback

Format Specifications

Gemini TTS audio format depends on whether the request is unary or streaming:

  • Unary Requests (generateContent): Returns complete WAV (audio/wav) audio with a standard 44-byte RIFF header included by default (24 kHz, mono, 16-bit signed little-endian PCM). These bytes can be passed directly to standard media players (AVAudioPlayer, MediaPlayer, <audio>).

  • Streaming Requests (generateContentStream): Returns headerless raw Linear PCM (audio/l16; rate=24000; channels=1) chunks (24 kHz, mono, 16-bit signed little-endian PCM) so chunks can be streamed or concatenated continuously without per-chunk container headers.

  • Audio Parameters:

    • Sample Rate: 24,000 Hz (24 kHz)
    • Channels: 1 channel (mono)
    • Bit Depth: 16-bit linear PCM (little-endian signed integer)
    • MIME Types: audio/l16 (streaming chunks) or audio/wav / audio/x-wav (unary)

Playback Approaches

  1. Direct Low-Latency PCM Streaming:
    • Stream raw PCM chunks directly into low-level audio renderers without waiting for the full response:
      • iOS: AVAudioEngine + AVAudioPlayerNode scheduling AVAudioPCMBuffer.
      • Android: AudioTrack configured with ENCODING_PCM_16BIT and CHANNEL_OUT_MONO.
      • Web: Web Audio API (AudioContext) scheduling PCM chunks into AudioBufferSourceNode.
  2. WAV Container Format:
    • Standard media players (AVAudioPlayer on iOS, MediaPlayer on Android, <audio> on Web) expect a container header (RIFF/WAV).
    • For unary responses, the RIFF header is already present. For assembled streaming PCM chunks, prepend a standard 44-byte RIFF header to make the complete audio immediately playable or saveable as .wav.
Standard 44-Byte WAV (RIFF) Header Construction
Offset  Size  Field              Value
0       4     ChunkID            "RIFF"
4       4     ChunkSize          36 + Subchunk2Size (file size - 8)
8       4     Format             "WAVE"
12      4     Subchunk1ID        "fmt "
16      4     Subchunk1Size      16 (for PCM)
20      2     AudioFormat        1 (PCM linear)
22      2     NumChannels        1 (Mono)
24      4     SampleRate         24000
28      4     ByteRate           48000 (SampleRate * NumChannels * BitsPerSample / 8)
32      2     BlockAlign         2 (NumChannels * BitsPerSample / 8)
34      2     BitsPerSample      16
36      4     Subchunk2ID        "data"
40      4     Subchunk2Size      Number of PCM bytes
44+     N     Data               Raw PCM bytes

Platform Code Snippets

Swift (iOS)

1. Streaming Playback with AVAudioEngine
import AVFoundation
import FirebaseAILogic

@MainActor
final class SpeechStreamingManager {
    private let audioEngine = AVAudioEngine()
    private let playerNode = AVAudioPlayerNode()
    
    // 24 kHz, 16-bit Mono PCM
    private let audioFormat = AVAudioFormat(
        commonFormat: .pcmFormatInt16,
        sampleRate: 24000,
        channels: 1,
        interleaved: false
    )!

    init() {
        audioEngine.attach(playerNode)
        audioEngine.connect(playerNode, to: audioEngine.mainMixerNode, format: audioFormat)
    }

    func streamSpeech(prompt: String) async throws {
        let ai = FirebaseAI.firebaseAI()
        
        let speechConfig = SpeechConfig(
            voiceConfig: VoiceConfig(
                prebuiltVoiceConfig: PrebuiltVoiceConfig(voiceName: "Puck")
            )
        )
        
        let genConfig = GenerationConfig(
            responseModalities: [.audio],
            speechConfig: speechConfig
        )
        
        let model = ai.generativeModel(
            modelName: "gemini-3.1-flash-tts-preview",
            generationConfig: genConfig
        )
        
        if !audioEngine.isRunning {
            try audioEngine.start()
        }
        playerNode.play()

        let stream = model.generateContentStream(prompt)
        for try await chunk in stream {
            for part in chunk.candidates.first?.content.parts ?? [] {
                if let inlineData = part as? InlineDataPart,
                   inlineData.mimeType.contains("audio/pcm") || inlineData.mimeType.contains("audio/l16") {
                    schedulePCMBuffer(data: inlineData.data)
                }
            }
        }
    }

    private func schedulePCMBuffer(data: Data) {
        let frameCount = UInt32(data.count) / audioFormat.streamDescription.pointee.mBytesPerFrame
        guard frameCount > 0,
              let buffer = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity: frameCount) else {
            return
        }
        buffer.frameLength = frameCount
        
        data.withUnsafeBytes { rawBufferPointer in
            guard let src = rawBufferPointer.baseAddress else { return }
            if let dest = buffer.int16ChannelData?[0] {
                UnsafeMutableRawPointer(dest).copyMemory(from: src, byteCount: data.count)
            }
        }
        
        playerNode.scheduleBuffer(buffer)
    }

    func stop() {
        playerNode.stop()
        audioEngine.stop()
    }
}
2. WAV Converter for AVAudioPlayer
func addWavHeader(to pcmData: Data, sampleRate: Int = 24000, channels: Int = 1, bitDepth: Int = 16) -> Data {
    var header = Data()
    let byteRate = sampleRate * channels * bitDepth / 8
    let blockAlign = channels * bitDepth / 8
    let totalDataLen = Int32(pcmData.count)
    let totalLength = totalDataLen + 36

    header.append(contentsOf: "RIFF".utf8)
    header.append(Data(from: totalLength.littleEndian))
    header.append(contentsOf: "WAVE".utf8)
    header.append(contentsOf: "fmt ".utf8)
    header.append(Data(from: Int32(16).littleEndian))       // Subchunk1Size for PCM
    header.append(Data(from: Int16(1).littleEndian))        // AudioFormat 1 = PCM
    header.append(Data(from: Int16(channels).littleEndian))
    header.append(Data(from: Int32(sampleRate).littleEndian))
    header.append(Data(from: Int32(byteRate).littleEndian))
    header.append(Data(from: Int16(blockAlign).littleEndian))
    header.append(Data(from: Int16(bitDepth).littleEndian))
    header.append(contentsOf: "data".utf8)
    header.append(Data(from: totalDataLen.littleEndian))

    return header + pcmData
}

private extension Data {
    init<T>(from value: T) {
        var val = value
        self = Swift.withUnsafeBytes(of: &val) { Data($0) }
    }
}

Kotlin (Android)

Streaming Playback with AudioTrack
import android.media.AudioAttributes
import android.media.AudioFormat
import android.media.AudioTrack
import com.google.firebase.ai.FirebaseAI
import com.google.firebase.ai.type.ResponseModality
import com.google.firebase.ai.type.generationConfig
import com.google.firebase.ai.type.prebuiltVoiceConfig
import com.google.firebase.ai.type.speechConfig
import com.google.firebase.ai.type.voiceConfig
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext

class SpeechStreamingManager {
    private val sampleRate = 24000
    private val minBufferSize = AudioTrack.getMinBufferSize(
        sampleRate,
        AudioFormat.CHANNEL_OUT_MONO,
        AudioFormat.ENCODING_PCM_16BIT
    )

    private val audioTrack = AudioTrack.Builder()
        .setAudioAttributes(
            AudioAttributes.Builder()
                .setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
                .setUsage(AudioAttributes.USAGE_MEDIA)
                .build()
        )
        .setAudioFormat(
            AudioFormat.Builder()
                .setEncoding(AudioFormat.ENCODING_PCM_16BIT)
                .setSampleRate(sampleRate)
                .setChannelMask(AudioFormat.CHANNEL_OUT_MONO)
                .build()
        )
        .setBufferSizeInBytes(minBufferSize * 2)
        .setTransferMode(AudioTrack.MODE_STREAM)
        .build()

    suspend fun streamSpeech(prompt: String) = withContext(Dispatchers.IO) {
        val config = generationConfig {
            responseModalities = listOf(ResponseModality.AUDIO)
            speechConfig = speechConfig {
                voiceConfig = voiceConfig {
                    prebuiltVoiceConfig = prebuiltVoiceConfig {
                        voiceName = "Puck"
                    }
                }
            }
        }

        val generativeModel = FirebaseAI.getInstance().getGenerativeModel(
            modelName = "gemini-3.1-flash-tts-preview",
            generationConfig = config
        )

        audioTrack.play()
        try {
            val responseStream = generativeModel.generateContentStream(prompt)
            responseStream.collect { chunk ->
                chunk.candidates.firstOrNull()?.content?.parts?.forEach { part ->
                    if (part is com.google.firebase.ai.type.InlineDataPart) {
                        val audioBytes = part.data
                        audioTrack.write(audioBytes, 0, audioBytes.size)
                    }
                }
            }
        } finally {
            audioTrack.stop()
        }
    }

    fun release() {
        audioTrack.release()
    }
}
2. Convert PCM to WAV for MediaPlayer
import android.content.Context
import android.media.MediaPlayer
import java.io.File
import java.io.FileOutputStream
import java.nio.ByteBuffer
import java.nio.ByteOrder

fun addWavHeader(
    pcmBytes: ByteArray,
    sampleRate: Int = 24000,
    channels: Short = 1,
    bitDepth: Short = 16
): ByteArray {
    val totalDataLen = pcmBytes.size
    val totalLength = totalDataLen + 36
    val byteRate = sampleRate * channels * bitDepth / 8
    val blockAlign = (channels * bitDepth / 8).toShort()

    val header = ByteBuffer.allocate(44).order(ByteOrder.LITTLE_ENDIAN).apply {
        put("RIFF".toByteArray())
        putInt(totalLength)
        put("WAVE".toByteArray())
        put("fmt ".toByteArray())
        putInt(16) // Subchunk1Size
        putShort(1) // AudioFormat 1 = PCM
        putShort(channels)
        putInt(sampleRate)
        putInt(byteRate)
        putShort(blockAlign)
        putShort(bitDepth)
        put("data".toByteArray())
        putInt(totalDataLen)
    }.array()

    return header + pcmBytes
}

fun playWavAudio(context: Context, wavBytes: ByteArray) {
    val tempFile = File.createTempFile("tts_", ".wav", context.cacheDir).apply {
        deleteOnExit()
        FileOutputStream(this).use { it.write(wavBytes) }
    }

    val mediaPlayer = MediaPlayer()
    try {
        mediaPlayer.setDataSource(tempFile.absolutePath)
        mediaPlayer.prepare()
        mediaPlayer.start()
        mediaPlayer.setOnCompletionListener {
            it.release()
            tempFile.delete()
        }
    } catch (e: Exception) {
        mediaPlayer.release()
        tempFile.delete()
        throw e
    }
}

Web (JavaScript/TypeScript)

1. Streaming Playback with Web Audio API
import { initializeApp } from "firebase/app";
import { getAI, getGenerativeModel, GoogleAIBackend } from "firebase/ai";

const firebaseConfig = {
  // your firebase configuration
};

const app = initializeApp(firebaseConfig);
const ai = getAI(app, { backend: new GoogleAIBackend() });

const model = getGenerativeModel(ai, {
  model: "gemini-3.1-flash-tts-preview",
  generationConfig: {
    responseModalities: ["AUDIO"],
    speechConfig: {
      voiceConfig: {
        prebuiltVoiceConfig: {
          voiceName: "Puck",
        },
      },
    },
  },
});

export class WebSpeechPlayer {
  private audioCtx: AudioContext | null = null;
  private nextStartTime: number = 0;

  private getAudioContext(): AudioContext {
    if (!this.audioCtx) {
      if (typeof window === "undefined") {
        throw new Error("AudioContext is only available in browser environments.");
      }
      const AudioContextClass =
        window.AudioContext ||
        (window as unknown as { webkitAudioContext: typeof AudioContext })
          .webkitAudioContext;
      this.audioCtx = new AudioContextClass({ sampleRate: 24000 });
    }
    return this.audioCtx;
  }

  async playSpeechStream(prompt: string) {
    const ctx = this.getAudioContext();
    if (ctx.state === "suspended") {
      await ctx.resume();
    }
    this.nextStartTime = ctx.currentTime;

    const responseStream = await model.generateContentStream(prompt);

    for await (const chunk of responseStream.stream) {
      const candidates = chunk.candidates || [];
      for (const candidate of candidates) {
        for (const part of candidate.content?.parts || []) {
          if ("inlineData" in part && part.inlineData?.data) {
            const rawPcm = this.base64ToArrayBuffer(part.inlineData.data);
            this.queuePcmChunk(rawPcm, ctx);
          }
        }
      }
    }
  }

  private queuePcmChunk(pcmData: ArrayBuffer, ctx: AudioContext) {
    const int16Array = new Int16Array(pcmData);
    const float32Array = new Float32Array(int16Array.length);

    // Convert 16-bit PCM integer samples to -1.0 .. 1.0 float samples
    for (let i = 0; i < int16Array.length; i++) {
      float32Array[i] = int16Array[i] / 32768.0;
    }

    const audioBuffer = ctx.createBuffer(1, float32Array.length, 24000);
    audioBuffer.copyToChannel(float32Array, 0);

    const source = ctx.createBufferSource();
    source.buffer = audioBuffer;
    source.connect(ctx.destination);

    const startTime = Math.max(this.nextStartTime, ctx.currentTime);
    source.start(startTime);
    this.nextStartTime = startTime + audioBuffer.duration;
  }

  private base64ToArrayBuffer(base64: string): ArrayBuffer {
    const binaryString = window.atob(base64);
    const len = binaryString.length;
    const bytes = new Uint8Array(len);
    for (let i = 0; i < len; i++) {
      bytes[i] = binaryString.charCodeAt(i);
    }
    return bytes.buffer;
  }
}
2. Convert PCM Base64 to Playable WAV Blob for <audio> Elements
export function pcmToWavBlob(pcmBytes: Uint8Array, sampleRate = 24000): Blob {
  const header = new ArrayBuffer(44);
  const view = new DataView(header);

  const numChannels = 1;
  const bitsPerSample = 16;
  const byteRate = (sampleRate * numChannels * bitsPerSample) / 8;
  const blockAlign = (numChannels * bitsPerSample) / 8;
  const dataSize = pcmBytes.byteLength;

  // RIFF identifier
  writeString(view, 0, "RIFF");
  view.setUint32(4, 36 + dataSize, true);
  writeString(view, 8, "WAVE");

  // fmt subchunk
  writeString(view, 12, "fmt ");
  view.setUint32(16, 16, true);             // Subchunk1Size
  view.setUint16(20, 1, true);              // AudioFormat (PCM = 1)
  view.setUint16(22, numChannels, true);
  view.setUint32(24, sampleRate, true);
  view.setUint32(28, byteRate, true);
  view.setUint16(32, blockAlign, true);
  view.setUint16(34, bitsPerSample, true);

  // data subchunk
  writeString(view, 36, "data");
  view.setUint32(40, dataSize, true);

  return new Blob([header, pcmBytes], { type: "audio/wav" });
}

function writeString(view: DataView, offset: number, string: string) {
  for (let i = 0; i < string.length; i++) {
    view.setUint8(offset + i, string.charCodeAt(i));
  }
}

Best Practices & Troubleshooting

  • Audio Session & Permissions: On iOS and Android, make sure the app's audio session/category is configured to allow playback (e.g., AVAudioSessionCategoryPlayback).
  • Buffering & Jitter Prevention: When streaming audio chunks via generateContentStream, queue buffers sequentially with exact timestamps or frame offsets to avoid audio stutter or gaps.
  • Model Compatibility: TTS features require Gemini models with dedicated audio synthesis capabilities (gemini-3.1-flash-tts-preview). Do not use standard text-only model IDs.
  • App Check Protection: Generating audio consumes quota. Protect your API endpoints by enforcing App Check on every client build.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts22d3 checks · Risk SAFE
  • Gen Agent Trust Hub22d

    This skill provides safe and well-documented instructions for integrating Firebase AI Logic into mobile and web applications. It follows security best practices, specifically regarding the protection of App Check debug tokens and avoiding unauthorized system modifications.

  • Socket22d

    No alerts

  • Snyk22d

    Risk: LOW · No issues

Signed by skilld at 99954a3. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub last week.

Activeupdated 3 weeks ago
version
1.0.1
metadata
{
  "author": "Google LLC",
  "category": "AiAndMachineLearning"
}

README badge

README badge for firebase/agent-skills/firebase-ai-logic-basics