skilld
All skills
nvidia avatar

/digital-health-clinical-asr-eval

@1ba3403
by NVIDIA Corporationnvidia/skills3.5k stars
424

Stage 3 of Clinical ASR Flywheel. Score a NeMo manifest, produce the five-section KER leaderboard (by-ipa_source diagnostic). Not for ASR auth (/riva-asr).

Requires /digital-health-clinical-asr-build /digital-health-clinical-asr-finetune

Ask your Agent

Use this Skill: https://skilld.dev/gh/nvidia/skills/digital-health-clinical-asr-eval

This session only. Nothing lands on disk.

skill-card.md

≈909 tokens on demand. Your agent reads this file only when SKILL.md points to it.

Description: <br>

Stage 3 of Clinical ASR Flywheel. Score a NeMo manifest, produce the five-section KER leaderboard (by-ipa_source diagnostic). Not for ASR auth (/riva-asr). <br>

This skill is ready for commercial/non-commercial use. <br>

Owner

NVIDIA <br>

License/Terms of Use: <br>

Apache-2.0 <br>

Use Case: <br>

Developers and engineers evaluating clinical automatic speech recognition (ASR) systems to produce keyword-error-rate (KER) leaderboards and post-eval routing recommendations. <br>

Deployment Geography for Use: <br>

Global <br>

Known Risks and Mitigations: <br>

Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br> Mitigation: Review and scan skill before deployment. <br>

Reference(s): <br>

Skill Output: <br>

Output Type(s): [Analysis, Files, Shell commands] <br> Output Format: [Markdown with inline bash code blocks] <br> Output Parameters: [1D] <br> Other Properties Related to Output: [None] <br>

Evaluation Agents Used: <br>

  • Claude Code (claude-code) <br>
  • Codex (codex) <br>

Evaluation Tasks: <br>

Evaluated against 8 NVSkills-Eval tasks (7 positive skill-activation, 1 negative activation). Pass threshold: 50%. Overall verdict: PASS. <br>

Evaluation Metrics Used: <br>

Reported benchmark dimensions: <br>

  • Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
  • Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
  • Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
  • Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
  • Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>

Underlying evaluation signals used in this run: <br>

  • skill_execution: Verifies that the agent loaded the expected skill and workflow. <br>
  • skill_efficiency: Checks routing quality, decoy avoidance, and redundant tool usage. <br>
  • accuracy: Grades final-answer correctness against the reference answer. <br>
  • goal_accuracy: Checks whether the overall user task completed successfully. <br>
  • behavior_check: Verifies expected behavior steps, including safety expectations. <br>
  • token_efficiency: Compares token usage with and without the skill. <br>

Evaluation Results: <br>

Dimension Num claude-code codex
Security 8 84% (+10%) 92% (+44%)
Correctness 8 79% (+2%) 95% (+27%)
Discoverability 8 49% (+2%) 72% (+25%)
Effectiveness 8 85% (+6%) 92% (+28%)
Efficiency 8 44% (+6%) 57% (+10%)

Skill Version(s): <br>

1.1.0 (source: frontmatter) <br>

Ethical Considerations: <br>

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

(For Release on NVIDIA Platforms Only) <br> Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here. <br>

Source: SKILL.md on GitHub

Third-party checks

1 warning3mo3 checks · Risk SAFE
  • Gen Agent Trust Hub3mo

    This skill provides a workflow for evaluating Clinical Automatic Speech Recognition (ASR) performance. It is well-documented, follows security best practices, and communicates exclusively with official NVIDIA infrastructure for its ASR transcription tasks.

  • Socket3mo

    No alerts

  • Snyk3mo

    Risk: MEDIUM · 1 issue

Provenance

Signed by skilld at 1ba3403. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 months ago

Capability

version
1.1.0
author
Ben Randoing <brandoing@nvidia.com>
tags
[
  "clinical-asr",
  "eval",
  "ker",
  "leaderboard",
  "flywheel"
]
Other metadata
tools
[
  "Read",
  "Write",
  "Bash",
  "Skill"
]
compatibility
NVIDIA_API_KEY (required) for hosted ASR NIMs via NVCF. A NeMo-format manifest produced by /digital-health-clinical-asr-build (or an externally-provided manifest carrying the clinical-extension fields). All ASR call shapes and WER/CER/KER/SER scoring recipes are inlined — no sibling agent skill required.
metadata
{
  "author": "Ben Randoing <brandoing@nvidia.com>",
  "tags": [
    "clinical-asr",
    "flywheel",
    "eval",
    "ker",
    "leaderboard"
  ],
  "team": "healthcare-tme",
  "domain": "ai-ml",
  "stage": 3,
  "previous_skill": "digital-health-clinical-asr-build",
  "next_skill": "digital-health-clinical-asr-finetune"
}

README badge

README badge for nvidia/skills/digital-health-clinical-asr-eval