/digital-health-clinical-asr-finetune

Stage 4 — Fine-tune playbook (deep dive)

Companion to SKILL.md's Stage 4 sections. Use this for the why behind the recipe — hyperparameter rationale, the validated empirical numbers, and when to stop tuning. The what (split script, base-model choice, docker invocation, offline eval, riva-build/riva-deploy commands) lives in SKILL.md and is not duplicated here.

Empirical validation

The recipe in SKILL.md is verified end-to-end on a reference clinical manifest. The numbers below are the actual measurements, not estimates:

Per-category breakdown on the same manifest (cycle 1 → cycle 2):

Note the asymmetry: drug names start hardest and improve most. Conditions and procedures already had partial coverage in the base model.

Hyperparameter rationale

The hyperparameter table itself is in SKILL.md §4d. The choices below are the why — diagnostic notes for tuning, not values to copy.

• bf16-mixed precision is non-negotiable for TDT. fp32 works but is ~2× slower. fp16-mixed produces NaN losses with TDT decoders — a known TDT numerical-stability issue.
• lr=3e-4 is the upper end of the comfortable range. Below 1e-4, training barely moves on small (<100-row) manifests. Above 1e-3, you risk catastrophic forgetting of the base model's general English vocabulary — recoverable but expensive.
• warmup_steps=5 is tiny-manifest-only. At 1,000+ row scale, bump to ~500 (one epoch's worth of steps). The 5-step value exists because the reference manifest's 39 rows fit in <10 steps total at batch_size=4.
• epochs=3 is a smoke test. Production runs use 10–30 epochs with early-stopping on validation WER (patience=3). The 3-epoch verified result reflects how quickly TDT picks up clinical vocabulary once the override SSML has gotten the audio right.
• batch_size=4 fits a 16 GB VRAM GPU. On 48 GB cards (L40S, A6000), raise to 16. Effective batch size also scales via accumulate_grad_batches if you're OOM-constrained — this is the right escape hatch on 24 GB cards when bs=8 is needed but bs=8 won't fit.
• gradient_clip_val=1.0 is defensive. With this recipe + the verified base, gradients haven't been observed to explode. Keep it on — the cost is zero, and removing it makes diagnosing rare divergences harder.

When to stop tuning

A multi-cycle loop has natural stopping points. After cycle N+1, evaluate:

• You've hit a KER floor across multiple cycles. Two consecutive cycles with KER drop < 5% relative is the signal to stop tuning and either accept the model or rethink the methodology (add a new metric, extend entity_category to capture a missed dimension, etc.).
• You're past 30 epochs without improvement. TDT bases plateau by ~30 epochs on manifests under ~5,000 rows. Larger manifests merit larger budgets — but verify scaling laws empirically; don't extrapolate from the 3-epoch smoke run.
• Validation WER trends upward while training loss drops. Classic overfit. Bail to /digital-health-clinical-asr-build and grow the manifest, or add early-stopping (patience=3 on validation WER).

Brev provisioning (full walkthrough)

The condensed Brev recipe in SKILL.md §4a is enough for the happy path. This section covers the why and the corners — disk sizing, SKU selection, install-script verification, and the SSH-config step that trips first-time users.

Account + cost shape

Create an account at https://brev.dev. Brev bills per-second on top of the underlying cloud's SKU rate, so the cost shape is: (SKU $/hr) × (provision time + active time + idle time before stop). The L40S 48 GB SKU runs about $1.50/hr at the time of writing; a 3-epoch SFT run on a ~100-row manifest finishes in 15–30 minutes, so the run itself is under a dollar. The trap is forgetting to stop the instance after the run — overnight idle on an L40S is ~$36. Set a calendar reminder, or wrap the whole flow in a script that ends with brev stop.

Verifying the Linux install script

The Brev install script is hosted on raw.githubusercontent.com/brevdev/brev-cli/main/bin/install-latest.sh. The curl | sh antipattern hands arbitrary code execution to whoever controls that URL (Brev's GitHub repo + GitHub's CDN). Mitigations:

1. Download first, run second (the SKILL.md §4a recipe). curl -o install-brev.sh separates fetch from execute; shasum -a 256 and less let you verify before running.
2. Pin to a release if your org policy requires reproducibility. The script's URL with main follows HEAD; replacing main with a tag (v0.6.420 or whatever's current) freezes the binary. Find the current tag at https://github.com/brevdev/brev-cli/releases.
3. Use Homebrew on macOS. Homebrew installs from a tap with package-level integrity guarantees; prefer it over the curl path on Mac.

SKU selection

L40S 48 GB is the right default for Parakeet TDT 0.6B SFT — raises batch_size to 16 (vs. 4 on a 24 GB card) and cuts wall-clock proportionally. Step up when:

The current SKU catalog: https://docs.brev.dev/gpus.

Disk sizing

--disk 200gi is enough for: NeMo container (~12 GB), 1–2 cycles of audio (~5 GB each at 16 kHz mono on ~200-row manifests), the base .nemo (~2 GB), and the trained .nemo (~2 GB). Bump to 400 GB if you're keeping multiple cycles on the instance, or if your audio is 48 kHz.

Image choice

--image ubuntu-22-04-cuda-12-4 is the only image you should pick for this flywheel. It pre-bakes:

• NVIDIA driver compatible with CUDA 12.4
• Docker + NVIDIA Container Toolkit (covers /riva-nim-setup prereqs)
• nvidia-smi works out of the box

Vanilla Ubuntu images need driver + toolkit install before nvcr.io/nvidia/nemo:25.11.01 can use the GPU — solvable, but wasted setup time at $1.50/hr.

SSH-config + rsync (the step that trips first-timers)

Brev exposes each instance over SSH, but the connection details aren't in ~/.ssh/config by default. The fix is one command:

brev ssh-config            # writes Host entries to ~/.ssh/config
ssh digital-health-clinical-asr-sft  # or: rsync -avz ./cycle1/ digital-health-clinical-asr-sft:~/cycle1/

After brev ssh-config, the instance name works as a standard SSH host. Skip this command and rsync will fail with ssh: Could not resolve hostname digital-health-clinical-asr-sft.

Stopping vs deleting

• brev stop <name> — halts billing for compute, keeps the disk (and its $0.10/GB-month storage cost). Use between training sessions on the same cycle.
• brev delete <name> — frees everything. Use when you're done with a cycle and have rsync'd the artifacts back to your laptop.

If you have a recurring training cadence (e.g. one cycle a week), stop between sessions saves you the docker pull + re-rsync each time. If cycles are one-offs, delete is cleaner.

Container invocation (full docker-run pattern from SKILL.md §4d)

Paths are illustrative — adapt to your cycle layout. The flag set encodes the hyperparameters from the SKILL.md §4d table.

docker run --gpus all --rm -it \
  -v "$PWD:/workspace" \
  nvcr.io/nvidia/nemo:25.11.01 \
  python /opt/NeMo/examples/asr/speech_to_text_finetune.py \
    --config-path=conf \
    --config-name=speech_to_text_finetune \
    model.train_ds.manifest_filepath=/workspace/train.jsonl \
    model.validation_ds.manifest_filepath=/workspace/validation.jsonl \
    init_from_pretrained_model=nvidia/parakeet-tdt-0.6b-v2 \
    trainer.precision=bf16-mixed \
    trainer.max_epochs=3 \
    model.optim.lr=3e-4 \
    model.optim.sched.warmup_steps=5 \
    model.train_ds.batch_size=4 \
    trainer.gradient_clip_val=1.0

Related references

• Base-model selection table → SKILL.md §4c
• Stock SFT hyperparameter values → SKILL.md §4d
• Decision tree on cycle-N+1 KER → SKILL.md §4e
• riva-build / riva-deploy commands → SKILL.md §4f
• Host → container manifest path rewriting → SKILL.md "References" section (links container-paths.md from the top level)