Reviewing a Library Rail Contribution
This skill reviews PRs that add or change a rail under
nemoguardrails/library/. The contract being reviewed against is defined by
the add-library-rail skill; do not restate it, read it. Follow the
repository Review Mode rules (compare against the merge base with develop,
inspect tests as well as implementation, treat every finding as advisory
until verified against the real code path).
The guiding fact, learned from evaluation: contributors and reviewers both comply with what is named and miss what is omitted. Work through every numbered check; do not stop at the first few findings.
Step 0: Scope and duplicates
- Confirm the PR is a rail contribution: files under
nemoguardrails/library/<name>/, usually with tests and a docs page. - Check for duplicate or in-flight integrations for the same vendor:
gh pr list --state open --search "<vendor>"and the same for issues. If another PR covers the vendor, surface that before reviewing further. - Is this vendor integration plausibly wanted? If there is no linked triaged issue or maintainer signal, flag for maintainer judgment; do not review it into shape first.
Step 1: Run the mechanical gates (do not re-review what they check)
Run pre-commit FIRST, per the repository Review Mode rule: it reformats, so running it later moves the lines your findings cite.
uv run --locked pre-commit run --files <the PR's changed files>
make test TEST=tests/test_rail_skill_references.py
make test TEST="tests/rails/llm/test_builtin_rail_manifests.py tests/rails/llm/test_builtin_rail_conformance.py tests/rails/llm/test_library_flow_files.py"
make test TEST="tests/rails/llm/test_config.py tests/test_rails_config.py tests/test_runtime_flow_gate_equivalence.py"
make test TEST="tests/guardrails/test_compiled_rail.py tests/guardrails/test_guardrails.py tests/guardrails/test_rails_manager.py"
make test TEST="tests/llm/test_call_import_graph.py tests/telemetry/test_usage_reporting.py"
make test TEST=tests/http/test_library_boundary.py
make test TEST="<space-separated PR test files>"
make test TEST=tests/recorded/rails/library ARGS="--block-network -q"
make test TEST=tests/recorded/test_cassette_provenance.py
git diff <merge-base> -- pyproject.toml uv.lock # usually empty; see belowA red gate is a finding by itself; report it and keep going. A green gate proves only that its collected cases passed. Inspect the summary for relevant skips, especially optional vendor dependencies and IORails cases; a skipped PR path is not a completed dimension. Spend judgment attention on what the gates do not encode.
Check packaging diffs against the current optional extras in pyproject.toml
and nemoguardrails/manifests/manifest.py::RailRequirements. A justified new
extra is valid; adding an optional vendor package to the default install path
is not. The authoring skill's dependency section owns the install checklist.
Step 1b: New-rail completeness (presence, not quality)
The manifest gate detects action packages without an ancestor rail.py, but a
config-only or no-action package can evade that scan. For every new rail,
confirm the applicable items are present before judging quality:
rail.pywith aRAILmanifest. For an ordinary action package the manifest gate enforces this; inspect config-only/no-action contributions explicitly.- For a flow-bearing rail, both-dialect unit tests: block path under Colang 1
AND under
colang_version: "2.x". A v2 flow file with no v2 test is unexecuted code; inbound PRs have shipped v2 files that parse but cannot run. - For tool-only rails, IORails integration tests instead of Colang files and
dialect tests: allow/block, argument scoping when supported, and errors.
celandtool_safety_checkdeclare tool surfaces with empty flow-file lists. - For an executable provider-backed rail, an engine-level test of the action raising (vendor down, fail-closed), through Colang where applicable and through IORails for tool-only rails.
- A recorded entry under
tests/recorded/rails/library/, mandatory pernemoguardrails/library/README.md. Provider-backed rails cover allow, block, and one recordable provider error. Pure-Python rails cover applicable allow/block or transform behavior without a provider-error case. A config-only manifest covers its public configuration contract. - Docs page and example config.
Each missing item is a finding; request it rather than inferring it is covered elsewhere.
Step 2: Judgment dimensions (what tests cannot check)
Each item names the check, where to look, and what "wrong" looks like.
- Privacy honesty. Compare
RailPrivacyinrail.pyagainst what the request layer actually transmits and what the vendor's own documentation says about retention.sends_user_text/sends_bot_textmust match the surfaces' bindings, includingsends_retrieved_chunksfor a retrieval surface andsends_tool_call_text/sends_tool_result_textfor tool data sent to a remote judge or vendor;remote_servicesmust be non-empty for any HTTP rail;data_retentionmust be stated if the vendor states one. A defaults-onlyRailPrivacy()on a vendor rail is almost always wrong. - Requirements honesty. Every env var the code reads
(
os.environ/os.getenv) appears inRailRequirements.env_vars; every lazily imported package appears inoptional_dependencies(and inextraswhen the package already ships in a nemoguardrails extra); the docspip installline matches the declaration and carries the version bound, whichRailRequirementscannot express. Named model selections useBinding.model_param(...)orBinding.model(...), not ordinary surface or literal bindings: the model resource drives IORails compilation checks andRailsConfigvalidation for global and per-tool model references. - Retry semantics. For HTTP rails: a rail-owned
RetryPolicyin one named module-level place, either a constant or a single config-derived builder, never inline at the call site; POST retries only via explicitretryable_methods={"POST"}carrying a comment that names one of the three permitted resend-safety mechanisms (documented idempotency, an idempotency key header, or a stateless scan endpoint), and no comment is a finding; no hand-rolled retry loop aroundhttp_call. Apply it throughensure_retrying_http_client, not by directly constructingRetryingHTTPClient(injected_client, policy): existing retry policy takes precedence, retry must sit inside instrumentation, and duplicate retry or instrumentation layers are rejected. Then verify any retry/backoff claims in docs or comments againstnemoguardrails/http/retry.py: backoff is full-jitter exponential with a lower bound of zero,Retry-Afteris honored only withinmax_retry_afterand is otherwise discarded rather than capped unlessclamp_retry_after=True, and vendor override headers are ignored unlesshonor_retry_override_headeris set. Contributors document this from guesswork. - Fail-closed behavior, tested not asserted. Vendor errors, timeouts,
and malformed payloads must surface as typed errors or blocks, never a
silent allow. For an executable provider-backed rail, there must be a
test of the action raising through each supported engine; a summary or
comment claiming fail-closed without a test is a finding. IORails maps
ordinary action exceptions to
RailOutcome.failurebut propagates status-bearingLLMCallExceptionandModelEngineError. Verify the relevant public behavior rather than expecting every error to raise or every error to become a refusal. - Dialect coverage. For flow-bearing rails, the flow-files gate checks
structure; behavior needs tests. Both dialects' block path must be exercised
end to end in the UNIT suite (
colang_version: "2.x"variant present?), and a per-rail duplicate Colang 2 RECORDED test is itself a finding when it produces the same provider traffic. Public-API v2 recordings remain valid when the runtime changes the calls, and IORails parity may reuse a sibling cassette or deliberately run a deterministic no-cassette case. If any surface declares atransform_targetand exposes Colang flows, the transform path also needs a Colang 2 behavior test; the flow-gate harness is Colang 1 only, so a v2 transform can be entirely unexecuted while the block-path tests pass. Everyenable_rails_exceptionsbranch must be either tested or absent. Checkflows.coandflows.v1.coagainst the four equivalence items in add-library-rail Step 3 (same flow names, same action and bindings per flow, same branch structure, same asserted output); divergent semantics between dialects is a finding even when both parse. - Outcome discipline. Actions return
RailOutcomewith neutral decision evidence. A supportedmetadata["refusal_message"]override may carry localized or configured refusal text, as in #2470; verify the consumer and rendering/fallback tests for each supported engine. Unsupported presentation keys or embedded exception types are findings. Flows must only read metadata keys the action actually sets (grep the action for eachmetadata[subscript used in a flow). Then cross-check the config model: every field inrail_config.pymust have a test at a non-default value, and decision-changing fields need both sides of the boundary. Read the comparison operator against the documented semantics; an inverted or off-by-one threshold passes every gate. - Telemetry content-freeness. No logging of checked text, request or
response bodies, URLs with query strings, API keys, or tokens; exception
messages must not embed payloads. Check
log.calls and every f-string in raised errors. - Test placement and hygiene. Deterministic LLMRails tests in
tests/may injectRecordingHTTPClient(fromnemoguardrails.testing) throughregister_action_param. IORails does not support arbitrary action-param registration; its compiled rail receives the engine-managed HTTP client. Compare the action's signature withnemoguardrails/guardrails/compiled_rail.py::CompiledRail::_request_dependenciesand its manifest bindings instead of assuming LLMRails injection is portable. Recorded suite entries follow therecorded-testsskill (vcr-structured, or non-vcr with a defensible reason). Nothing enforces that split, so read every suite test in the diff that has novcrmark and challenge it against the skill's criteria. Secrets viamonkeypatch; any real-looking key in a test or cassette is a blocking finding. - Docs accuracy. Every statement in the catalog page must match the
code: config options, defaults, env vars, install line, the vendor's data
retention period (matching
RailPrivacy.data_retention), and limitations. Verify each "known limitation" is real; a false limitation is as bad as a missing one, and an uncited or unreproducible one is a finding.docs_urlin the manifest is the repo-relative.mdxpath of the page the PR adds; the manifest gate asserts that path resolves. - Fail-open semantics. If the rail supports fail-open at all: it must
default to fail-closed; the switch must live in the rail's config model
where a config reviewer sees it, not in an env var (env vars are for
secrets, and fail-open flips the rail's security posture); a fail-open
pass must be distinguishable from a genuine vendor clear, carrying an
explicit marker in the outcome metadata (as
_fail_open_outcomeinnemoguardrails/library/f5/actions.pydoes) so it is never mistaken for a real clear in logs and traces, and it must be logged. A fail-open path also needs its own test alongside the fail-closed one. Cross-check the docs against the code: if the page claims the switch covers availability only, confirm the fail-open branch actually excludes 4xx. Failing open on 401 or 403 while documenting otherwise leaves a revoked key silently unguarded, and is a finding. - Cassette provenance. A cassette is a claim by its submitter, and an
agent or contributor without service access can fabricate one that
looks recorded. The sanitizer fixed-point gate
(
tests/recorded/test_cassette_provenance.py) catches naive fabrication (realistic ids, timestamps, cookies that the recorder would have scrubbed), but a sentinel-perfect forgery passes it, so for new cassettes from external contributors also ask HOW they were recorded. If the answer is unverifiable and the org holds no key to re-record, the honest options are maintainer re-recording before merge or demoting the tests to the explicitfake_cassetteregime; do not merge unverifiable cassettes as recorded truth. - Conformance drift (a green suite can hide this). The conformance
gates pass if a contributor made them pass the WRONG way, so diff the
generic infrastructure, not just the rail. Reject weakened checks in
test_builtin_rail_manifests.py,test_builtin_rail_conformance.py, ortest_library_flow_files.pyunless the PR's purpose is the gate itself. Extending an applicable pinned inventory, such as the set of model-bound surfaces, is required; changing the rule to admit an invalid rail is not. Reject a new entry inLEGACY_UNMANIFESTED_PACKAGESorNON_PORTABLE_DECLARED_FLOWSunless it carries an explicit, reviewed design rationale (a new rail should satisfy the contract, not join the exceptions). For any change toschemas/rails_config.snapshot.json, confirm it was regenerated for a deliberate config-schema change and read the diff as a public-schema change, not a mechanical refresh. - Error-path coverage (unit, exhaustive). Dimension 4 checks that
errors fail closed; this checks that EVERY error branch is actually
exercised. Enumerate the action's failure branches (timeout, connection
error, each handled status class, 429 retried and retry-exhausted,
malformed payload, missing credential, missing optional dependency, each
crossed with fail-open/closed where supported) and require a unit test
per branch with a mocked transport. A branch with handling code but no
test is a finding. These belong in the unit layer, not recorded: a
synthetic error forced into
tests/recorded/violates the placement rule, so do not accept "it's covered by the recorded suite" for anything but the recordable real error. - Client composition, ownership, and error typing. These are not fully
gate-checked; the HTTP
boundary test is an import-only AST check, so read the code. The action
must never call
closeoracloseon an injectedhttp_client; only thefactory=path passed tohttp_callmay own a client, and that factory must return the fully composed closable client, since wrapping an injected client must not transfer ownership (nemoguardrails/http/request.py). Verify composition throughensure_retrying_http_client: the canonical topology is instrumentation outside retry outside transport, and an existing retry/instrumentation layer is preserved rather than compounded. Vendor failures must surface as rail-specific error types (theerrs.pypattern, exemplarnemoguardrails/library/clavata/errs.py), not as raw transport or vendor SDK exceptions escaping the action.
Step 3: Security pass
- Credential handling: key read from env only, never from config values that end up in logs or snapshots; JWT/signature construction does not log the secret; no credentials committed in tests, cassettes, or examples.
- Outbound URL control: if the config exposes
base_urlor an endpoint override, note that guardrails configs are sometimes attacker-adjacent; the URL must only ever be used for the vendor call (no template injection into paths), and redirects stay disabled (transport default). - Input handling: user/bot text goes into the request body as data, never interpolated into URLs, headers, or shell commands; response parsing treats the payload as untrusted (no eval, no attribute access on unvalidated structures without try/except).
- Deserialization: only
response.json(); anything like pickle or yaml load on vendor bytes is blocking.
Step 4: Process conformance
- DCO sign-off on every commit; conventional commit title; no changelog edits; AI_POLICY disclosure if applicable.
- Docs build if docs changed:
make docs-fern. - Every reviewer finding verified against the actual code path before it is reported (Review Mode rule); drop anything you cannot reproduce or point to a line for.
Verdict format
Report findings ranked by severity, each with file:line evidence and the dimension number it came from. State explicitly which gates ran and their results, and which dimensions came up clean; a review that only lists problems hides its own coverage. End with one of: approve, approve with nits, request changes (list the blocking subset).