Index of every page on this site: llms.txt
All skills
nvidia-nemo avatar

/recorded-tests

@5112cf1
by nvidia-nemonvidia-nemo/guardrails7.5k stars
904
Run it yourself

Decides when a code change requires a recorded/VCR test, whether a test belongs in tests/recorded/ or the plain unit suite under tests/, and how to structure it in either place. Use when adding, reviewing, or relocating tests that involve LLM calls, streaming, cassettes, snapshots, or provider traffic, and when changing provider clients, public API output shapes, or model-calling rails. Trigger keywords - recorded test, cassette, VCR, replay, snapshot test, tests/recorded, record-cassettes, fake cassette, TestChat, FakeLLMModel.

  • 1 file
  • 14.4 KB
  • Apache-2
  • Updated 3 weeks ago
  • GitHub
Use this Skill: https://skilld.dev/gh/nvidia-nemo/guardrails/recorded-tests

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈138 tokens for metadata: the name and description. ≈3.5k when used: this file.

Description uses 6.9% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈551 characters, or 6.9%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

Recorded Tests: When and How

tests/recorded/ is primarily a replay suite: it pins real provider wire traffic in cassettes and pins the public API outputs produced from that traffic in inline snapshots. Its value is refreshability: make rewrite-cassettes re-records against live providers and detects drift. make record-cassettes runs record mode once, so it only fills in cassettes that are missing and will not refresh an existing one. A test that records no provider traffic gets none of that value. Two repository policies still place selected non-provider cases here: public-API equivalence/validation contracts and the mandatory public entry for each library rail. Those tests do not fabricate a cassette.

Read tests/recorded/README.md before writing any test in this suite. This skill decides WHETHER and WHERE a recorded test is needed; the README defines HOW the suite works.

When a change requires a recorded test

The default criterion: a recording is required when the change alters bytes sent to a provider, the interpretation of bytes received from one, or the public output shape derived from them. A cassette-backed expectation must come from the provider rather than from us. Apply the two repository-owned non-cassette exceptions below separately.

Concretely, add or update a recorded test when:

  1. The wire path changes: nemoguardrails/llm/clients/ (request construction, SSE parsing, error mapping, usage extraction). Fakes here only encode our own assumptions about the provider dialect. Division of labor: tests/llm/clients/ with httpx.MockTransport owns synthetic wire conditions (status-code mapping, malformed bodies); tests/recorded/clients/ owns real traffic (happy path, real error shapes).
  2. The public API output contract changes over provider-driven flows: generate_async / stream_async / check output shapes, new GenerationResponse fields, usage metadata, streaming chunk framing. If such outputs change and no snapshot under tests/recorded/rails/public_api/ moved, either coverage has a hole or the change did not do what you think.
  3. A new provider integration, engine, or model type is added: minimum one happy-path recorded test plus one provider-error test (the invalid-model 404 pattern), so the integration is drift-detectable from day one.
  4. A model-calling library rail is added or its prompt/parsing changes (content safety, topic control, jailbreak detection, self check): the block/allow decision depends on real model output, so tests/recorded/rails/library/ needs the outcome triad (allow, block, error). Pure Python rails such as regex need a suite entry too, since nemoguardrails/library/README.md requires recorded e2e coverage for every new library rail; they get non-vcr suite tests with their own config directory (exemplar: tests/recorded/rails/library/test_regex.py), not zero coverage. For the full contribution checklist of a new rail, see the add-library-rail skill.
  5. A bug fix whose trigger is provider-shaped data: the bug only reproduces with a real payload (an SSE framing edge, a missing usage field, a provider error body variant). The regression test must be recorded, or a fake cassette per the README ladder if the payload is unrecordable.

A cassette-backed test is NOT required when:

  • The change is deterministic runtime logic, even when the diff touches llmrails.py (our own error envelopes, action dispatch, rail sequencing, fail-closed handling of local failures). Unit test in tests/.
  • The change is a pure refactor. The recorded suite's job in a refactor is to be RUN, not extended: replay with --block-network before and after is the equivalence proof.
  • The change is config or input validation, Colang runtime behavior, docs, or tooling, unless it changes a public-API contract intentionally pinned in the recorded suite or the mandatory library-rail entry.

File paths are a heuristic, not the rule: llmrails.py contains both wire-adjacent contract code and deterministic logic in the same file. Always apply the byte-level criterion above.

Placement checklist

Before applying the provider-traffic checklist, preserve these existing suite responsibilities:

  • tests/recorded/rails/public_api/ may pin deterministic validation and public method contracts used for engine/refactor equivalence.
  • tests/recorded/rails/library/ has one public entry per library rail by repository policy. A pure-Python rail uses a non-vcr test; IORails parity may reuse a sibling cassette or deliberately exercise a no-cassette case.

For every other scenario, answer these in order. A "no" that routes to tests/ is final.

  1. Does the scenario send at least one real provider request? Main model, a rail's model, or embeddings must actually cross the wire during the test. If nothing crosses the wire, there is nothing to record. Unless one of the two suite responsibilities above applies, write a unit test in tests/ using TestChat with llm_completions (see tests/utils.py), or FakeLLMModel from nemoguardrails.testing.fake_model directly.

    Red flags that mean the answer is "no":

    • The test passes generator= to stream_async AND no rail model or vendor call is made. With config.rails.output.streaming.enabled false, stream_async returns the generator directly and NO rail runs; when it is true only output rails run. generator= bypasses the MAIN model, so a rail whose own model or HTTP call still crosses the wire remains recorded-suite material; the library suite's own stream_with_fake_main helper is built on it.
    • The only rail action involved is a local Python action registered by the test config (a fake checker, a deliberately failing action).
    • The assertion fires before any model call (input validation, pytest.raises on config errors).
    • The effective test markers do not include vcr. That means no cassette; require one of the explicit suite responsibilities above.
  2. Is the interesting behavior in the provider response, or in our own deterministic logic? Provider response shape (streaming chunk framing, error body format, usage metadata, tool-call structure, header handling) belongs here. Deterministic library logic belongs in tests/, with the separate public library entry required above. The question to ask: "if the provider changed its behavior tomorrow, could this test's expected output change?" If it cannot, the test does not need a recording.

  3. Does an equivalent test already exist? Search both tests/ and tests/recorded/ for the behavior. Do not duplicate the same assertion unless the recorded version exercises provider-shaped behavior or the library/public-API suite contract at a distinct public layer.

  4. Is the scenario a failure that cannot occur against a live provider? Follow the negative-path ladder in tests/recorded/README.md: prefer a recordable real error (for example an invalid model name yielding a real 404), then pure runtime pytest.raises, and only as a last resort a @pytest.mark.fake_cassette with the required metadata. If the failure is injected by local test code rather than the provider, that is rule 1 territory: normally a unit test.

Colang dialects

LLMRails library recordings are dialect-single: do not duplicate a provider recording across Colang 1 and Colang 2 when both dialects produce identical wire bytes. Deterministic flow routing belongs in both-dialect unit tests. IORails parity tests are engine-equivalence tests rather than another Colang dialect; they may reuse the LLMRails cassette and may contain deterministic no-cassette cases. One separate exception exists:

  • In tests/recorded/rails/public_api/, the v2 runtime is a different generation pipeline (different prompts and LLM calls for dialog and generation surfaces), so v2 coverage there records genuinely different provider traffic and is legitimate recorded-suite material.

Non-vcr and mixed-marker suite tests

The suite intentionally hosts non-vcr, pure-runtime tests (for example StreamingNotSupportedError validation, generate_async argument validation over FakeLLMModel, and the check() contracts over local-rail configs) because public_api doubles as the public API surface contract used for refactor equivalence proofs. This exception is a convention, not an enforcement: nothing in tests/recorded/conftest.py inspects markers, so a misplaced test collects and passes silently and no tooling will tell you it is in the wrong suite. A non-vcr suite test carries the module-level pytestmark = [pytest.mark.recorded] and no vcr mark, adding pytest.mark.asyncio only when the module's tests are async (exemplar: tests/recorded/rails/library/test_regex.py for the async form, tests/recorded/test_fake_cassettes.py for the synchronous one).

That describes tests you add that exercise the library through this suite. The cassette meta-tests directly under tests/recorded/ are infrastructure for the suite itself rather than replay tests, and they do not all follow it: test_cassette_provenance.py carries no pytestmark at all. Nothing rejects that either, so do not read those modules as a template for a new suite test. Because nothing fails for you, apply these criteria yourself. Add a non-vcr suite test only when all of these hold:

  • it pins a public API contract of the same surface as the neighboring recorded tests, or is input validation on that surface,
  • it has no natural home in tests/ (no equivalent unit test exists or could express the same public contract),
  • it needs no new config directory of its own, EXCEPT for a rail under tests/recorded/rails/library/, where nemoguardrails/library/README.md mandates recorded coverage and a per-rail config directory is the usual shape. It is a convention, not a documented requirement, and the suite also holds shared multi-rail directories (configs/full_stack, openai_input_stack); reuse one before adding your own.

When in doubt, put it in tests/. Do not park a test here just because it collects; nothing is checking, so the discipline is yours.

A module-level vcr mark is a default, not proof that every test opens a cassette. A mixed-purpose parity module may deliberately route a deterministic test to an empty cassette directory. Review cassette completeness against the test's actual provider path and cassette routing, not the inherited marker alone.

Worked examples

Belongs in tests/recorded/:

  • Streaming against a real OpenAI model, pinning chunk framing and final content (test_openai_stream_async_public_contract), with usage metadata pinned separately by test_stream_async_matches_recorded_chat_completion_metadata. The chunk boundaries come from the provider's SSE stream; only a recording can pin them honestly.
  • A content-safety rail whose checker model is a real provider call, pinning what the rail blocks. The block decision depends on the real model output.
  • An invalid model name producing the provider's real 404, pinning the status-bearing LLMCallException on LLMRails and IORails. The error body is provider-shaped and refreshable. IORails converts ordinary action errors without an upstream HTTP status to a blocking failed outcome instead; synthetic error-path tests belong in tests/.

Belongs in tests/ instead:

  • A rail action that raises RuntimeError, asserting the stream yields an internal-error envelope. The action is local, the envelope is constructed by our code, and TestChat(llm_completions=...) covers it with zero cassettes. (See tests/test_streaming_internal_errors.py for the canonical pattern.)
  • Any assertion on our own JSON error format, rail ordering, or config validation.

How, once placed in tests/recorded/

Follow tests/recorded/README.md exactly. The short version:

  • Module-level pytestmark = [pytest.mark.recorded, ...]; add @pytest.mark.vcr per test when the module mixes vcr and non-vcr tests.
  • Request credentials as fixture parameters (openai_api_key, nvidia_api_key); the fixture name is what gates recording.
  • Config constants live in the suite-local configs.py; reuse an existing config directory before creating a new one.
  • Record with credentials using the first pytest command in tests/recorded/README.md, fill an empty snapshot offline with --inline-snapshot=create, then verify replay with --block-network. --inline-snapshot=fix only updates an existing value. The current make record-cassettes target runs its record pass before snapshot creation, so a brand-new empty snapshot() stops that target at the expected teardown error; use the documented explicit two-pass workflow for a new test.
  • Name negative tests test_<surface>_<failure>_<behavior> with _raises, _fails_closed, or _invalid_* suffixes.

Review checklist for diffs touching tests/recorded/

  • Test that actually sends or replays provider traffic with no committed cassette: incomplete, must be recorded. A module-level vcr mark can cover a deliberate deterministic no-cassette test, so inspect its routing.
  • New non-vcr suite test: challenge it against the exception criteria above; it usually signals a test that belongs in tests/. Nothing enforces this, so read every test in the diff that has no vcr mark.
  • New config directory for a single deterministic test: the test is in the wrong suite, unless it is the per-rail config for a new nemoguardrails/library/ rail, which is the usual shape there.
  • Same behavior asserted in both tests/ and tests/recorded/ in one change: require a provider-shaped distinction or one of the explicit public contract/library policy reasons; otherwise keep the unit test.
  • Diff changes nemoguardrails/llm/clients/ or a model-calling library rail with no change under tests/recorded/: ask where the recorded coverage is.

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at 5112cf1. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 3 weeks ago

README badge

README badge for nvidia-nemo/guardrails/recorded-tests