Index of every page on this site: llms.txt
All skills
nvidia-nemo avatar

/review-library-rail

@5112cf1
by nvidia-nemonvidia-nemo/guardrails7.5k stars
904
Run it yourself

Review a pull request that adds or modifies a rail in nemoguardrails/library/ (built-in or vendor guardrail integration). Runs the mechanical gates first, then judgment dimensions tests cannot check (privacy honesty, retry semantics, fail-closed behavior, dialect coverage, docs accuracy), then a security pass. Use when reviewing rail-integration PRs, vendor integration contributions, or changes under nemoguardrails/library/. Trigger keywords - review rail PR, rail integration review, vendor integration PR, library rail review, guardrail contribution.

  • 1 file
  • 17.5 KB
  • Apache-2
  • Updated 3 weeks ago
  • GitHub
Use this Skill: https://skilld.dev/gh/nvidia-nemo/guardrails/review-library-rail

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈144 tokens for metadata: the name and description. ≈4.3k when used: this file.

Description uses 7.2% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈578 characters, or 7.2%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Reviewing a Library Rail Contribution

This skill reviews PRs that add or change a rail under nemoguardrails/library/. The contract being reviewed against is defined by the add-library-rail skill; do not restate it, read it. Follow the repository Review Mode rules (compare against the merge base with develop, inspect tests as well as implementation, treat every finding as advisory until verified against the real code path).

The guiding fact, learned from evaluation: contributors and reviewers both comply with what is named and miss what is omitted. Work through every numbered check; do not stop at the first few findings.

Step 0: Scope and duplicates

  • Confirm the PR is a rail contribution: files under nemoguardrails/library/<name>/, usually with tests and a docs page.
  • Check for duplicate or in-flight integrations for the same vendor: gh pr list --state open --search "<vendor>" and the same for issues. If another PR covers the vendor, surface that before reviewing further.
  • Is this vendor integration plausibly wanted? If there is no linked triaged issue or maintainer signal, flag for maintainer judgment; do not review it into shape first.

Step 1: Run the mechanical gates (do not re-review what they check)

Run pre-commit FIRST, per the repository Review Mode rule: it reformats, so running it later moves the lines your findings cite.

uv run --locked pre-commit run --files <the PR's changed files>
make test TEST=tests/test_rail_skill_references.py
make test TEST="tests/rails/llm/test_builtin_rail_manifests.py tests/rails/llm/test_builtin_rail_conformance.py tests/rails/llm/test_library_flow_files.py"
make test TEST="tests/rails/llm/test_config.py tests/test_rails_config.py tests/test_runtime_flow_gate_equivalence.py"
make test TEST="tests/guardrails/test_compiled_rail.py tests/guardrails/test_guardrails.py tests/guardrails/test_rails_manager.py"
make test TEST="tests/llm/test_call_import_graph.py tests/telemetry/test_usage_reporting.py"
make test TEST=tests/http/test_library_boundary.py
make test TEST="<space-separated PR test files>"
make test TEST=tests/recorded/rails/library ARGS="--block-network -q"
make test TEST=tests/recorded/test_cassette_provenance.py
git diff <merge-base> -- pyproject.toml uv.lock   # usually empty; see below

A red gate is a finding by itself; report it and keep going. A green gate proves only that its collected cases passed. Inspect the summary for relevant skips, especially optional vendor dependencies and IORails cases; a skipped PR path is not a completed dimension. Spend judgment attention on what the gates do not encode.

Check packaging diffs against the current optional extras in pyproject.toml and nemoguardrails/manifests/manifest.py::RailRequirements. A justified new extra is valid; adding an optional vendor package to the default install path is not. The authoring skill's dependency section owns the install checklist.

Step 1b: New-rail completeness (presence, not quality)

The manifest gate detects action packages without an ancestor rail.py, but a config-only or no-action package can evade that scan. For every new rail, confirm the applicable items are present before judging quality:

  • rail.py with a RAIL manifest. For an ordinary action package the manifest gate enforces this; inspect config-only/no-action contributions explicitly.
  • For a flow-bearing rail, both-dialect unit tests: block path under Colang 1 AND under colang_version: "2.x". A v2 flow file with no v2 test is unexecuted code; inbound PRs have shipped v2 files that parse but cannot run.
  • For tool-only rails, IORails integration tests instead of Colang files and dialect tests: allow/block, argument scoping when supported, and errors. cel and tool_safety_check declare tool surfaces with empty flow-file lists.
  • For an executable provider-backed rail, an engine-level test of the action raising (vendor down, fail-closed), through Colang where applicable and through IORails for tool-only rails.
  • A recorded entry under tests/recorded/rails/library/, mandatory per nemoguardrails/library/README.md. Provider-backed rails cover allow, block, and one recordable provider error. Pure-Python rails cover applicable allow/block or transform behavior without a provider-error case. A config-only manifest covers its public configuration contract.
  • Docs page and example config.

Each missing item is a finding; request it rather than inferring it is covered elsewhere.

Step 2: Judgment dimensions (what tests cannot check)

Each item names the check, where to look, and what "wrong" looks like.

  1. Privacy honesty. Compare RailPrivacy in rail.py against what the request layer actually transmits and what the vendor's own documentation says about retention. sends_user_text/sends_bot_text must match the surfaces' bindings, including sends_retrieved_chunks for a retrieval surface and sends_tool_call_text/sends_tool_result_text for tool data sent to a remote judge or vendor; remote_services must be non-empty for any HTTP rail; data_retention must be stated if the vendor states one. A defaults-only RailPrivacy() on a vendor rail is almost always wrong.
  2. Requirements honesty. Every env var the code reads (os.environ/os.getenv) appears in RailRequirements.env_vars; every lazily imported package appears in optional_dependencies (and in extras when the package already ships in a nemoguardrails extra); the docs pip install line matches the declaration and carries the version bound, which RailRequirements cannot express. Named model selections use Binding.model_param(...) or Binding.model(...), not ordinary surface or literal bindings: the model resource drives IORails compilation checks and RailsConfig validation for global and per-tool model references.
  3. Retry semantics. For HTTP rails: a rail-owned RetryPolicy in one named module-level place, either a constant or a single config-derived builder, never inline at the call site; POST retries only via explicit retryable_methods={"POST"} carrying a comment that names one of the three permitted resend-safety mechanisms (documented idempotency, an idempotency key header, or a stateless scan endpoint), and no comment is a finding; no hand-rolled retry loop around http_call. Apply it through ensure_retrying_http_client, not by directly constructing RetryingHTTPClient(injected_client, policy): existing retry policy takes precedence, retry must sit inside instrumentation, and duplicate retry or instrumentation layers are rejected. Then verify any retry/backoff claims in docs or comments against nemoguardrails/http/retry.py: backoff is full-jitter exponential with a lower bound of zero, Retry-After is honored only within max_retry_after and is otherwise discarded rather than capped unless clamp_retry_after=True, and vendor override headers are ignored unless honor_retry_override_header is set. Contributors document this from guesswork.
  4. Fail-closed behavior, tested not asserted. Vendor errors, timeouts, and malformed payloads must surface as typed errors or blocks, never a silent allow. For an executable provider-backed rail, there must be a test of the action raising through each supported engine; a summary or comment claiming fail-closed without a test is a finding. IORails maps ordinary action exceptions to RailOutcome.failure but propagates status-bearing LLMCallException and ModelEngineError. Verify the relevant public behavior rather than expecting every error to raise or every error to become a refusal.
  5. Dialect coverage. For flow-bearing rails, the flow-files gate checks structure; behavior needs tests. Both dialects' block path must be exercised end to end in the UNIT suite (colang_version: "2.x" variant present?), and a per-rail duplicate Colang 2 RECORDED test is itself a finding when it produces the same provider traffic. Public-API v2 recordings remain valid when the runtime changes the calls, and IORails parity may reuse a sibling cassette or deliberately run a deterministic no-cassette case. If any surface declares a transform_target and exposes Colang flows, the transform path also needs a Colang 2 behavior test; the flow-gate harness is Colang 1 only, so a v2 transform can be entirely unexecuted while the block-path tests pass. Every enable_rails_exceptions branch must be either tested or absent. Check flows.co and flows.v1.co against the four equivalence items in add-library-rail Step 3 (same flow names, same action and bindings per flow, same branch structure, same asserted output); divergent semantics between dialects is a finding even when both parse.
  6. Outcome discipline. Actions return RailOutcome with neutral decision evidence. A supported metadata["refusal_message"] override may carry localized or configured refusal text, as in #2470; verify the consumer and rendering/fallback tests for each supported engine. Unsupported presentation keys or embedded exception types are findings. Flows must only read metadata keys the action actually sets (grep the action for each metadata[ subscript used in a flow). Then cross-check the config model: every field in rail_config.py must have a test at a non-default value, and decision-changing fields need both sides of the boundary. Read the comparison operator against the documented semantics; an inverted or off-by-one threshold passes every gate.
  7. Telemetry content-freeness. No logging of checked text, request or response bodies, URLs with query strings, API keys, or tokens; exception messages must not embed payloads. Check log. calls and every f-string in raised errors.
  8. Test placement and hygiene. Deterministic LLMRails tests in tests/ may inject RecordingHTTPClient (from nemoguardrails.testing) through register_action_param. IORails does not support arbitrary action-param registration; its compiled rail receives the engine-managed HTTP client. Compare the action's signature with nemoguardrails/guardrails/compiled_rail.py::CompiledRail::_request_dependencies and its manifest bindings instead of assuming LLMRails injection is portable. Recorded suite entries follow the recorded-tests skill (vcr-structured, or non-vcr with a defensible reason). Nothing enforces that split, so read every suite test in the diff that has no vcr mark and challenge it against the skill's criteria. Secrets via monkeypatch; any real-looking key in a test or cassette is a blocking finding.
  9. Docs accuracy. Every statement in the catalog page must match the code: config options, defaults, env vars, install line, the vendor's data retention period (matching RailPrivacy.data_retention), and limitations. Verify each "known limitation" is real; a false limitation is as bad as a missing one, and an uncited or unreproducible one is a finding. docs_url in the manifest is the repo-relative .mdx path of the page the PR adds; the manifest gate asserts that path resolves.
  10. Fail-open semantics. If the rail supports fail-open at all: it must default to fail-closed; the switch must live in the rail's config model where a config reviewer sees it, not in an env var (env vars are for secrets, and fail-open flips the rail's security posture); a fail-open pass must be distinguishable from a genuine vendor clear, carrying an explicit marker in the outcome metadata (as _fail_open_outcome in nemoguardrails/library/f5/actions.py does) so it is never mistaken for a real clear in logs and traces, and it must be logged. A fail-open path also needs its own test alongside the fail-closed one. Cross-check the docs against the code: if the page claims the switch covers availability only, confirm the fail-open branch actually excludes 4xx. Failing open on 401 or 403 while documenting otherwise leaves a revoked key silently unguarded, and is a finding.
  11. Cassette provenance. A cassette is a claim by its submitter, and an agent or contributor without service access can fabricate one that looks recorded. The sanitizer fixed-point gate (tests/recorded/test_cassette_provenance.py) catches naive fabrication (realistic ids, timestamps, cookies that the recorder would have scrubbed), but a sentinel-perfect forgery passes it, so for new cassettes from external contributors also ask HOW they were recorded. If the answer is unverifiable and the org holds no key to re-record, the honest options are maintainer re-recording before merge or demoting the tests to the explicit fake_cassette regime; do not merge unverifiable cassettes as recorded truth.
  12. Conformance drift (a green suite can hide this). The conformance gates pass if a contributor made them pass the WRONG way, so diff the generic infrastructure, not just the rail. Reject weakened checks in test_builtin_rail_manifests.py, test_builtin_rail_conformance.py, or test_library_flow_files.py unless the PR's purpose is the gate itself. Extending an applicable pinned inventory, such as the set of model-bound surfaces, is required; changing the rule to admit an invalid rail is not. Reject a new entry in LEGACY_UNMANIFESTED_PACKAGES or NON_PORTABLE_DECLARED_FLOWS unless it carries an explicit, reviewed design rationale (a new rail should satisfy the contract, not join the exceptions). For any change to schemas/rails_config.snapshot.json, confirm it was regenerated for a deliberate config-schema change and read the diff as a public-schema change, not a mechanical refresh.
  13. Error-path coverage (unit, exhaustive). Dimension 4 checks that errors fail closed; this checks that EVERY error branch is actually exercised. Enumerate the action's failure branches (timeout, connection error, each handled status class, 429 retried and retry-exhausted, malformed payload, missing credential, missing optional dependency, each crossed with fail-open/closed where supported) and require a unit test per branch with a mocked transport. A branch with handling code but no test is a finding. These belong in the unit layer, not recorded: a synthetic error forced into tests/recorded/ violates the placement rule, so do not accept "it's covered by the recorded suite" for anything but the recordable real error.
  14. Client composition, ownership, and error typing. These are not fully gate-checked; the HTTP boundary test is an import-only AST check, so read the code. The action must never call close or aclose on an injected http_client; only the factory= path passed to http_call may own a client, and that factory must return the fully composed closable client, since wrapping an injected client must not transfer ownership (nemoguardrails/http/request.py). Verify composition through ensure_retrying_http_client: the canonical topology is instrumentation outside retry outside transport, and an existing retry/instrumentation layer is preserved rather than compounded. Vendor failures must surface as rail-specific error types (the errs.py pattern, exemplar nemoguardrails/library/clavata/errs.py), not as raw transport or vendor SDK exceptions escaping the action.

Step 3: Security pass

  • Credential handling: key read from env only, never from config values that end up in logs or snapshots; JWT/signature construction does not log the secret; no credentials committed in tests, cassettes, or examples.
  • Outbound URL control: if the config exposes base_url or an endpoint override, note that guardrails configs are sometimes attacker-adjacent; the URL must only ever be used for the vendor call (no template injection into paths), and redirects stay disabled (transport default).
  • Input handling: user/bot text goes into the request body as data, never interpolated into URLs, headers, or shell commands; response parsing treats the payload as untrusted (no eval, no attribute access on unvalidated structures without try/except).
  • Deserialization: only response.json(); anything like pickle or yaml load on vendor bytes is blocking.

Step 4: Process conformance

  • DCO sign-off on every commit; conventional commit title; no changelog edits; AI_POLICY disclosure if applicable.
  • Docs build if docs changed: make docs-fern.
  • Every reviewer finding verified against the actual code path before it is reported (Review Mode rule); drop anything you cannot reproduce or point to a line for.

Verdict format

Report findings ranked by severity, each with file:line evidence and the dimension number it came from. State explicitly which gates ran and their results, and which dimensions came up clean; a review that only lists problems hides its own coverage. End with one of: approve, approve with nits, request changes (list the blocking subset).

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at 5112cf1. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 3 weeks ago

README badge

README badge for nvidia-nemo/guardrails/review-library-rail