Index of every page on this site: llms.txt
All skills
nvidia avatar

/i4h-workflow-dataset-annotate

@6b75e9e
by NVIDIA Corporationnvidia/skills4.1k stars
547
Run it yourself

Grade or filter workflow HDF5 episodes with an OpenAI-compatible vision model. Use for visual success labels; do not use for replay, policy evaluation, or recordings without frames.

Use this Skill: https://skilld.dev/gh/nvidia/skills/i4h-workflow-dataset-annotate

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈53 tokens for metadata: the name and description. ≈1.4k when used: this file. ≈3.1k more on demand in 3 files.

Description uses 2.7% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈213 characters, or 2.7%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Annotate Workflow Recordings

Purpose

Grade sampled camera frames against a natural-language success criterion while keeping VLM labels separate from simulator success.

Instructions

  1. Run the checkout resolver and select one HDF5 and one criterion.
  2. Test camera sampling and the vision endpoint.
  3. Run grading and optional filtering on every selected episode.
  4. Compare verdict and output counts.

Resolve input and criterion

export I4H_WORKFLOWS_REPO_URL="${I4H_WORKFLOWS_REPO_URL:-https://github.com/isaac-for-healthcare/i4h-workflows}"
I4H_REPO_DIR_NAME="${I4H_WORKFLOWS_REPO_URL%/}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME##*/}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME##*:}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME%.git}"
[ -n "$I4H_REPO_DIR_NAME" ] || { echo "Cannot derive a checkout name from I4H_WORKFLOWS_REPO_URL" >&2; exit 2; }
ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"
if [ ! -d "$ROOT/workflows/i4h_workflows" ]; then
  ROOT="${I4H_WORKFLOWS:-$HOME/$I4H_REPO_DIR_NAME}"
  [ -d "$ROOT/workflows/i4h_workflows" ] || git clone "$I4H_WORKFLOWS_REPO_URL" "$ROOT"
fi
export I4H_WORKFLOWS="$ROOT"
cd "$ROOT"
find runs -name '*.hdf5' -type f -printf '%T@ %p\n' | sort -nr | head

Treat the resolver above as part of the skill contract: a hosted copy may run outside the base repository, so never assume the current checkout contains workflows/i4h_workflows. I4H_WORKFLOWS_REPO_URL selects the clone source. When I4H_WORKFLOWS is unset, derive the fallback directory from that URL; set I4H_WORKFLOWS only to reuse or choose a specific destination. Never replace an existing checkout.

Use the explicit/current-chain HDF5. “All recorded episodes” means every episode in that selected file, not every historical run. Inspect it and use the user's explicit success criterion when supplied; otherwise combine the source Scene manifest instruction with the workflow's visible terminal goal semantics. Phrase placement success as the object reaching and remaining at its target, not as the robot continuing to hold it.

Resolve the endpoint

Use a caller-provided OpenAI-compatible vision endpoint/model first. Local Agent exposes that configuration as I4H_AGENT_VL_BASE_URL, I4H_AGENT_VL_MODEL, and either I4H_AGENT_VL_API_KEY or I4H_AGENT_API_KEY. Map those generic agent variables to the annotator without printing the credential:

VLM_ARGS=()
if [ -n "${I4H_AGENT_VL_BASE_URL:-}" ] && [ -n "${I4H_AGENT_VL_MODEL:-}" ]; then
  I4H_VLM_URL="${I4H_AGENT_VL_BASE_URL%/}"
  case "$I4H_VLM_URL" in */v1) ;; *) I4H_VLM_URL="$I4H_VLM_URL/v1" ;; esac
  export I4H_VLM_URL
  export OPENAI_API_KEY="${I4H_AGENT_VL_API_KEY:-${I4H_AGENT_API_KEY:-EMPTY}}"
  VLM_ARGS=(--model "$I4H_AGENT_VL_MODEL")
fi

If no caller-provided endpoint/model is available, start the repository's local service:

tools/annotator/scripts/vllm.sh ensure

Record whether this invocation started it. Do not hard-code a model name in the skill; use the CLI/service defaults unless the user supplies one.

Dry-run sampling when needed

uv run --project tools/annotator i4h-annotator \
  --task "<success criterion>" \
  --dry-run \
  offline /absolute/path/to/recording.hdf5

Use this to verify cameras and sampled frames without transmitting images.

Grade and filter

RUN_DIR="$(pwd)/runs/<workflow>/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$RUN_DIR"
uv run --project tools/annotator i4h-annotator \
  --task "<success criterion>" \
  "${VLM_ARGS[@]}" \
  offline /absolute/path/to/recording.hdf5 \
  --write

Add global --base-url, --model, --camera, or --frames only when selected. Add offline --node only for a requested segment. Add --filter "$RUN_DIR/filtered.hdf5" only when filtering was requested; a summarize-only prompt must grade all episodes without requiring at least one success. Keep credentials in environment variables; never print them.

Stop the local VLM only if this invocation started it:

tools/annotator/scripts/vllm.sh stop

Verify

Inspect the annotator summary. If filtering was requested, also inspect the filtered file:

uv run --project tools/dataset i4h-dataset inspect "$RUN_DIR/filtered.hdf5" --segments

Require a verdict for every selected episode and reconcile pass/fail counts plus filtered counts when applicable. Treat endpoint errors, absent cameras, partial writes, and unexplained zero-episode output as failure. An all-failure verdict set is a valid completed grading run for summarize-only prompts; it is not a valid filtered dataset.

Troubleshooting

Check camera sampling before endpoint/authentication errors. Never accept partial writes or a filtered file with an unexplained zero count.

Prerequisites

Require a readable workflow HDF5 with camera frames and, unless dry-running, a reachable OpenAI-compatible vision endpoint.

Limitations

Visual grading cannot recover missing frames or prove simulator state that is not visible.

Examples

  • Run annotation on all recorded episodes and summarize. → select the current HDF5, grade every episode, verify the filtered file, and report pass/fail counts.

Completion gate

Report source HDF5, selected criterion/camera/model/endpoint origin, graded pass/fail counts, filtered path/count when requested, dry-run result if used, and local-service cleanup.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

2 warnings1mo3 checks · Risk SAFE
  • Gen Agent Trust Hub1mo

    The skill is safe to use for its intended purpose of robot workflow annotation. It follows best practices for credential management and utilizes official vendor-associated repositories. It possesses a standard indirect prompt injection attack surface typical of tools that process external data through Vision-Language Models.

  • Socket1mo

    1 alert: gptAnomaly

  • Snyk1mo

    Risk: MEDIUM · 1 issue

Signed by skilld at 6b75e9e. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 months ago
Other metadata
metadata
{
  "author": "Isaac for Healthcare Team <isaac-for-healthcare-support@nvidia.com>",
  "version": "0.8.0",
  "verification-request": "2026-09-15",
  "tags": [
    "isaac-for-healthcare",
    "i4h",
    "dataset",
    "annotation",
    "vlm"
  ]
}

README badge

README badge for nvidia/skills/i4h-workflow-dataset-annotate