Index of every page on this site: llms.txt
All skills
google avatar

/gke-inference

@5fb62c2
by googlegoogle/skills24k stars
2,026
Run it yourself

Deploys and optimizes AI/ML inference workloads on GKE, using GPUs, TPUs, and model servers. Use when deploying GKE inference servers, configuring GKE GPU resources for inference, or deploying LLMs on GKE. Don't use for migrating existing AI workloads to GKE (use google-cloud-solution-guided-gke-ai-migration), GKE RAG with Cloud SQL/AlloyDB (use google-cloud-solution-rag-enterprise-search-gke-sqldb), or batch/HPC (use gke-batch-hpc).

Use this Skill: https://skilld.dev/gh/google/skills/gke-inference

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈113 tokens for metadata: the name and description. ≈2k when used: this file. ≈470 more on demand in 1 file.

Description uses 5.7% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈453 characters, or 5.7%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

GKE AI/ML Inference

Routing Note: For migrating existing AI workloads to GKE, refer to the google-cloud-solution-guided-gke-ai-migration skill. For GKE RAG with Cloud SQL or AlloyDB (pgvector), refer to the google-cloud-solution-rag-enterprise-search-gke-sqldb skill.

This reference covers deploying AI/ML inference workloads on GKE using Google's Inference Quickstart (GIQ) and best practices for LLM serving.

MCP Tools: apply_k8s_manifest, get_k8s_resource, get_k8s_logs, get_k8s_rollout_status, describe_k8s_resource, list_k8s_events. CLI-only: gcloud container ai profiles *

Attribution

MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix gcloud with CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-inference)" and append agent-skills/1.0 (skill:gke-inference) to User-Agent headers.

When to Use

  • Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE
  • Generate optimized Kubernetes manifests for inference
  • Select GPU/TPU accelerators for model serving
  • Configure autoscaling for LLM inference

Prerequisites

  • A golden path GKE Autopilot cluster (GPU workloads are supported via ComputeClasses and NAP)
  • gcloud CLI authenticated
  • Sufficient GPU/TPU quota in the target region

Workflow

1. Discovery: Find Models and Hardware

# List all supported models
gcloud container ai profiles models list --quiet

# Find valid accelerator/server combinations for a model
gcloud container ai profiles list --model=<MODEL_NAME> --quiet

# Example: what can run Gemma 2 9B?
gcloud container ai profiles list --model=gemma-2-9b-it --quiet

2. Generate Manifest

gcloud container ai profiles manifests create \
  --model=<MODEL_NAME> \
  --model-server=<SERVER> \
  --accelerator-type=<ACCELERATOR> \
  --target-ntpot-milliseconds=<NTPOT> --quiet > inference.yaml

Parameters:

  • --model: Model ID (e.g., gemma-2-9b-it, llama-3-8b)
  • --model-server: Inference server (vllm, tgi, triton, tensorrt-llm)
  • --accelerator-type: GPU/TPU type (nvidia-l4, nvidia-tesla-a100, nvidia-h100-80gb)
  • --target-ntpot-milliseconds: Target Normalized Time Per Output Token (optional, for latency optimization)

Example:

gcloud container ai profiles manifests create \
  --model=gemma-2-9b-it \
  --model-server=vllm \
  --accelerator-type=nvidia-l4 \
  --target-ntpot-milliseconds=50 --quiet > inference.yaml

3. Review and Deploy

# Review for placeholders (HF tokens, PVCs)
cat inference.yaml

# Deploy
kubectl apply -f inference.yaml

# Monitor
kubectl get pods -w
kubectl logs -f <POD_NAME>

Some models require Hugging Face tokens. Create a Kubernetes Secret and reference it in the manifest.

GPU ComputeClass for Inference

For Autopilot clusters, create a ComputeClass to target GPU nodes:

apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: l4-inference
spec:
  priorities:
  - machineFamily: g2
    gpu:
      type: nvidia-l4
      count: 1
    minCores: 4
    minMemoryGb: 16

Accelerator Selection Guide

Accelerator Best For Memory Relative Cost
NVIDIA T4 Budget inference, 16 GB Lowest
: : lightweight legacy : : :
: : models : : :
NVIDIA L4 (G2) Small-medium model 24 GB Low
: : inference, video, : : :
: : graphics : : :
NVIDIA RTX PRO 6000 Multimodal AI, 96 GB Medium
: (G4) : high-fidelity 3D, : : :
: : fine-tuning : : :
Cloud TPU v5e Cost-effective Varies Medium
: : transformer inference : : :
Cloud TPU v5p High-performance Varies High
: : training : : :
Cloud TPU v6e High-efficiency next-gen 32 GB/chip Medium-High
: (Trillium) : training & serving : : :
Cloud TPU v7x Ultra-scale inference & 192 GB/chip High
: (Ironwood) : agentic workflows : : :
NVIDIA A100 Large model inference, 40/80 GB High
: : enterprise ML : : :
NVIDIA H100 / H200 Frontier model training, 80/141 GB Highest
: : high throughput : : :
NVIDIA B200 (A4) Blackwell-scale 192 GB Highest
: : training, FP4 precision : : :
NVIDIA GB200 (A4X) Rack-scale AI (Grace Massive Highest
: : Blackwell Superchip) : : :

Autoscaling LLM Inference

GPU-based autoscaling

Use custom metrics for GPU utilization:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-server
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: gpu_duty_cycle
      target:
        type: AverageValue
        averageValue: "80"

Best practices for inference autoscaling

  1. Use DCGM metrics: Golden path enables DCGM monitoring for GPU utilization metrics
  2. Set appropriate minReplicas: At least 1 for always-on serving; 0 for batch/on-demand
  3. Tune scale-down delay: LLM model loading is slow; use longer stabilization windows
  4. Consider queue depth: Scale on pending requests rather than pure GPU utilization for latency-sensitive workloads

Optimization Tips

  • Quantization: Use quantized models (GPTQ, AWQ) to reduce GPU memory and increase throughput
  • Batching: Configure model server batch size for throughput vs latency trade-off
  • Tensor parallelism: Split large models across multiple GPUs within a node
  • KV cache optimization: Tune --gpu-memory-utilization in vLLM for KV cache allocation

Troubleshooting

Issue Cause Fix
Invalid Unsupported tuple Re-run `gcloud container ai
: model/accelerator : : profiles list :
: combination : : --model=<MODEL>` :
GPU quota exceeded Regional quota limit Request quota increase or
: : : try a different region :
OOM on GPU Model too large for Use larger GPU, enable
: : accelerator : quantization, or use tensor :
: : : parallelism :
Slow cold start Large model loading from Use local SSD for model
: : registry : caching; pre-pull images :

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

1 warning2d3 checks · Risk SAFE
  • Gen Agent Trust Hub2d

    This skill provides standard procedures for deploying AI/ML workloads on Google Kubernetes Engine (GKE). It utilizes official gcloud CLI commands and Kubernetes manifests for infrastructure management and includes guidance on usage attribution for tracking purposes.

  • Socket2d

    No alerts

  • Snyk2d

    Risk: MEDIUM · 1 issue

Signed by skilld at 5fb62c2. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 3 days ago
metadata
{
  "version": "1.0.3",
  "category": "Containers"
}

README badge

README badge for google/skills/gke-inference