/skill

Healthcare Eval Harness

This skill checks a health app before release. It helps find risks that may harm a patient or expose private health data.

A failed critical check must block the release.

Credit: This skill was created by Health1 Super Speciality Hospitals and contributed by Dr. Keyur Patel.

Note: The examples use Jest. You may use Vitest, pytest, PHPUnit, or another test tool. Keep the same test groups and pass rules.

When to Use This Skill

Use this skill:

• Before any EMR or EHR app release
• After a CDSS rule changes
• After a drug, dose, or score rule changes
• After a patient data table changes
• After login or access rules change
• When you set up a CI release check
• After you fix a merge conflict in care code
• After an HL7, FHIR, lab, or vendor link changes
• After a safety bug or PHI leak is fixed

Safety Rules

Run the five test groups below.

The first three groups are critical. Every test must pass. One failed test blocks the release.

The last two groups are high priority. A pass rate of 95% or more is green. A lower rate needs a written review before release.

Also follow these rules:

• A test crash counts as a failure.
• A missing test group counts as a failure.
• Bad or missing test output counts as a failure.
• Do not hide failed tests with || true unless later code checks the full result.
• Do not use real patient data in tests.
• Test data must be fake and safe to share.
• Delete temp result files after each run.
• Pin tool and package versions in CI.
• Keep proof of the test run with the release.
• A human must review any result below 95%.
• A human review cannot allow a failed critical group.
• Do not treat code coverage as proof that care rules are correct.

Test Groups

1. CDSS Accuracy

Level: Critical
Pass rule: 100%

Test all care decision rules:

• Drug pairs in both orders
• Dose limits
• Age, weight, kidney, and liver rules
• Unit changes, such as mg to mcg
• Decimal and rounding rules
• Missing or bad input
• Known clinical scores
• Rules at exact limit values
• No missed danger alert
• No silent failure

npx jest --testPathPattern='tests/cdss' \
  --bail \
  --ci \
  --coverage \
  --coverageThreshold='{"global":{"branches":80,"functions":80,"lines":80}}'

A coverage score of 80% is only a base check. Set a higher score when the app needs it.

2. PHI Exposure

Level: Critical
Pass rule: 100%

Test that private health data does not leak through:

• API errors
• App logs
• Browser logs
• URLs and query text
• Browser storage
• Cache data
• File names
• Export files
• Metrics or debug tools
• Data from another clinic or tenant
• Access without login
• Access after logout
• A missing or weak service key

npx jest --testPathPattern='tests/security/phi' --bail --ci

Search for more than names. Tests should also cover birth dates, record IDs, phone numbers, email addresses, and mixed fields that can point to one person.

3. Data Integrity

Level: Critical
Pass rule: 100%

Test that care data stays correct:

• Signed or locked visits cannot change
• Audit records are complete
• Audit time and user data are correct
• Delete rules do not remove needed records
• Two users editing at once do not lose data
• Failed work rolls back as one unit
• No record is left without its parent
• Time zones do not change care dates
• Repeat requests do not make repeat orders
• A retry does not make two prescriptions

npx jest --testPathPattern='tests/data-integrity' --bail --ci

4. Clinical Workflows

Level: High
Green rule: 95% or more

Test full care steps:

• Start, update, sign, and close a visit
• Show forms and notes
• Add and remove medicine
• Search for drugs and health problems
• Make a prescription PDF
• Show red alerts
• Stop unsafe actions
• Save and restore a draft
• Handle a slow or failed service
• Keep the right user and patient in view

tmp_json=$(mktemp)
trap 'rm -f "$tmp_json"' EXIT

set +e
npx jest --testPathPattern='tests/clinical' \
  --ci \
  --json \
  --outputFile="$tmp_json"
test_exit=$?
set -e

if [ ! -s "$tmp_json" ] || ! jq -e . "$tmp_json" >/dev/null 2>&1; then
  echo "Clinical test output is missing or not valid JSON" >&2
  exit 1
fi

total=$(jq -r '.numTotalTests // 0' "$tmp_json")
passed=$(jq -r '.numPassedTests // 0' "$tmp_json")

if [ "$total" -eq 0 ]; then
  echo "No clinical tests found" >&2
  exit 1
fi

rate_x100=$((passed * 10000 / total))
printf 'Clinical pass rate: %d.%02d%% (%d/%d)\n' \
  $((rate_x100 / 100)) $((rate_x100 % 100)) "$passed" "$total"

if [ "$test_exit" -ne 0 ] && [ "$passed" -eq "$total" ]; then
  echo "The clinical test tool failed" >&2
  exit 1
fi

if [ "$rate_x100" -lt 9500 ]; then
  echo "Clinical review is required before release" >&2
fi

5. System Links

Level: High
Green rule: 95% or more

Test links to other systems:

• HL7 v2 message reading
• FHIR resource checks
• Lab result maps
• Code and unit maps
• Bad or cut-off messages
• Repeat messages
• Late messages
• Time zone changes
• Vendor timeouts
• Safe retry rules
• Unknown fields
• Wrong patient or clinic IDs

tmp_json=$(mktemp)
trap 'rm -f "$tmp_json"' EXIT

set +e
npx jest --testPathPattern='tests/integration' \
  --ci \
  --json \
  --outputFile="$tmp_json"
test_exit=$?
set -e

if [ ! -s "$tmp_json" ] || ! jq -e . "$tmp_json" >/dev/null 2>&1; then
  echo "Integration test output is missing or not valid JSON" >&2
  exit 1
fi

total=$(jq -r '.numTotalTests // 0' "$tmp_json")
passed=$(jq -r '.numPassedTests // 0' "$tmp_json")

if [ "$total" -eq 0 ]; then
  echo "No integration tests found" >&2
  exit 1
fi

rate_x100=$((passed * 10000 / total))
printf 'Integration pass rate: %d.%02d%% (%d/%d)\n' \
  $((rate_x100 / 100)) $((rate_x100 % 100)) "$passed" "$total"

if [ "$test_exit" -ne 0 ] && [ "$passed" -eq "$total" ]; then
  echo "The integration test tool failed" >&2
  exit 1
fi

if [ "$rate_x100" -lt 9500 ]; then
  echo "Integration review is required before release" >&2
fi

These scripts need jq. They use whole number math, so bc is not needed.

Release Rules

A written review should name:

• Each failed test
• The patient risk
• The person who checked the risk
• The fix or short-term guard
• The date for the full fix
• The final release choice

CI Example

This GitHub Actions job runs the three critical groups. It then checks the two high-priority groups. A low high-priority score fails the job so a person must review it.

name: Healthcare Safety Gate

on:
  push:
  pull_request:

jobs:
  safety-gate:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: npm

      - run: npm ci

      - name: Check required tools
        run: command -v jq >/dev/null

      - name: CDSS accuracy
        run: >
          npx jest
          --testPathPattern='tests/cdss'
          --bail
          --ci
          --coverage
          --coverageThreshold='{"global":{"branches":80,"functions":80,"lines":80}}'

      - name: PHI exposure
        run: >
          npx jest
          --testPathPattern='tests/security/phi'
          --bail
          --ci

      - name: Data integrity
        run: >
          npx jest
          --testPathPattern='tests/data-integrity'
          --bail
          --ci

      - name: Clinical workflows
        shell: bash
        run: |
          set -u
          TMP_JSON=$(mktemp)
          trap 'rm -f "$TMP_JSON"' EXIT

          set +e
          npx jest --testPathPattern='tests/clinical' --ci --json --outputFile="$TMP_JSON"
          TEST_EXIT=$?
          set -e

          if [ ! -s "$TMP_JSON" ] || ! jq -e . "$TMP_JSON" >/dev/null 2>&1; then
            echo "::error::Clinical test output is missing or not valid JSON"
            exit 1
          fi

          TOTAL=$(jq -r '.numTotalTests // 0' "$TMP_JSON")
          PASSED=$(jq -r '.numPassedTests // 0' "$TMP_JSON")

          if [ "$TOTAL" -eq 0 ]; then
            echo "::error::No clinical tests found"
            exit 1
          fi

          RATE_X100=$((PASSED * 10000 / TOTAL))
          printf 'Pass rate: %d.%02d%% (%d/%d)\n' \
            $((RATE_X100 / 100)) $((RATE_X100 % 100)) "$PASSED" "$TOTAL"

          if [ "$TEST_EXIT" -ne 0 ] && [ "$PASSED" -eq "$TOTAL" ]; then
            echo "::error::The clinical test tool failed"
            exit 1
          fi

          if [ "$RATE_X100" -lt 9500 ]; then
            echo "::error::Clinical pass rate is below 95%. A written review is required."
            exit 1
          fi

      - name: System links
        shell: bash
        run: |
          set -u
          TMP_JSON=$(mktemp)
          trap 'rm -f "$TMP_JSON"' EXIT

          set +e
          npx jest --testPathPattern='tests/integration' --ci --json --outputFile="$TMP_JSON"
          TEST_EXIT=$?
          set -e

          if [ ! -s "$TMP_JSON" ] || ! jq -e . "$TMP_JSON" >/dev/null 2>&1; then
            echo "::error::Integration test output is missing or not valid JSON"
            exit 1
          fi

          TOTAL=$(jq -r '.numTotalTests // 0' "$TMP_JSON")
          PASSED=$(jq -r '.numPassedTests // 0' "$TMP_JSON")

          if [ "$TOTAL" -eq 0 ]; then
            echo "::error::No integration tests found"
            exit 1
          fi

          RATE_X100=$((PASSED * 10000 / TOTAL))
          printf 'Pass rate: %d.%02d%% (%d/%d)\n' \
            $((RATE_X100 / 100)) $((RATE_X100 % 100)) "$PASSED" "$TOTAL"

          if [ "$TEST_EXIT" -ne 0 ] && [ "$PASSED" -eq "$TOTAL" ]; then
            echo "::error::The integration test tool failed"
            exit 1
          fi

          if [ "$RATE_X100" -lt 9500 ]; then
            echo "::error::Integration pass rate is below 95%. A written review is required."
            exit 1
          fi

If your Jest version does not support --testPathPattern, use the matching path option for that version. Do not change the test groups or pass rules.

Concrete Usage Example

A team changes a kidney dose rule for a drug.

1. Add tests for low, normal, and high kidney values.
2. Add tests at the exact rule limits.
3. Add tests for missing values and wrong units.
4. Run the critical groups:

npx jest --testPathPattern='tests/cdss' --bail --ci --coverage &&
npx jest --testPathPattern='tests/security/phi' --bail --ci &&
npx jest --testPathPattern='tests/data-integrity' --bail --ci

5. If one command fails, stop the release.
6. Run the clinical and system-link checks.
7. If either score is below 95%, ask for a written review.
8. Save the result with the release record.

Example result:

Healthcare Safety Check
Commit: abc1234

CDSS accuracy:       39/39, PASS
PHI exposure:         8/8, PASS
Data integrity:      12/12, PASS
Clinical workflows:  21/22, 95.45%, PASS
System links:          6/6, PASS
Coverage:             84%, PASS

Release result: PASS

If CDSS is 38/39, the release result must be BLOCKED.

Bad Practices

Do not:

• Skip CDSS tests because they passed before
• Set a critical pass rule below 100%
• turn off early stop for a critical group
• Mock the CDSS rule in a test that claims to test the real rule
• Release while a critical group is red
• Run CDSS tests without coverage
• Count a missing test group as a pass
• Hide a test crash
• Use real patient data
• Put PHI in logs, test names, images, or saved test files
• Call a low pass rate green
• Let the same person write, approve, and waive a high-risk failure without review