/airunway-aks-setup

Step 5 — First Model Deployment

Goal: Deploy a starter model that fits the cluster's GPU capacity and reach Ready status.

Model recommendations (based on cluster VRAM from Step 3 — see gpu-profiles.md for full sizing guide):

Present recommendation. Ask user to confirm or choose their own model.

Naming Convention

• model-name: A DNS-safe Kubernetes resource name derived from the model ID (e.g., meta-llama/Llama-3.1-8B-Instruct → llama-3-1-8b-instruct)
• model-id: The full HuggingFace model identifier (e.g., meta-llama/Llama-3.1-8B-Instruct)

Gated Models (HuggingFace Token)

Llama-family models are gated on HuggingFace and require an access token. Read the token interactively to avoid exposing it in shell history:

# Read token without echo, store in a securely permissioned temp file
hf_token_file="$(mktemp)"
trap 'rm -f "$hf_token_file"' EXIT

read -r -s -p "Enter HuggingFace token: " hf_token
printf '\n'
printf '%s' "$hf_token" > "$hf_token_file"
chmod 600 "$hf_token_file"
unset hf_token

kubectl create secret generic hf-token \
  --from-file=token="$hf_token_file" \
  -n <namespace> \
  --dry-run=client -o yaml | kubectl apply -f -

See powershell-notes.md for the PowerShell equivalent.

Apply the ModelDeployment CR

Use the appropriate manifest based on whether the model is gated.

For gated models (Llama etc. — requires HuggingFace token):

kubectl apply -f - <<EOF
apiVersion: airunway.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: <model-name>
  namespace: <namespace>
spec:
  model:
    id: <model-id>
    huggingFaceTokenSecretRef:
      name: hf-token
      key: token
  provider:
    name: <provider-name>
EOF

For non-gated models (Phi-3, Gemma etc. — no token required):

kubectl apply -f - <<EOF
apiVersion: airunway.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: <model-name>
  namespace: <namespace>
spec:
  model:
    id: <model-id>
  provider:
    name: <provider-name>
EOF

See powershell-notes.md for the PowerShell equivalent.

Monitor Deployment

kubectl get modeldeployment <model-name> -n <namespace> -w

Wait until STATUS = Ready. If not ready within 10 minutes:

• kubectl describe modeldeployment <model-name> -n <namespace> — check events
• kubectl get pods -l modeldeployment=<model-name> -n <namespace> — check pod status

Note: Large models (13B+) may take significantly longer than 10 minutes because weights must be downloaded first. For 70B models, expect 20–40+ minutes depending on network speed. Check pod logs to confirm download progress rather than assuming failure.