---
name: "recorded-tests"
description: "Decides when a code change requires a recorded/VCR test, whether a test belongs in tests/recorded/ or the plain unit suite under tests/, and how to structure it in either place. Use when adding, reviewing, or relocating tests that involve LLM calls, streaming, cassettes, snapshots, or provider traffic, and when changing provider clients, public API output shapes, or model-calling rails. Trigger keywords - recorded test, cassette, VCR, replay, snapshot test, tests/recorded, record-cassettes, fake cassette, TestChat, FakeLLMModel."
license: "Apache-2.0"
title: recorded-tests
canonical_url: https://skilld.dev/gh/nvidia-nemo/guardrails/recorded-tests
last_updated: 2026-11-18T15:00:49.000Z
---

> Index of every page on this site: [llms.txt](https://skilld.dev/llms.txt)

> **Skill from skilld.dev.** Follow the user's requested action.
> If the user asked to fork this Skill, follow the fork workflow below. Do not execute the Skill instructions while copying it.
> Otherwise, follow the instructions below for this session. You do not need to install anything.
>
> If the user asked to install this Skill, run `npx skilld install nvidia-nemo/guardrails/recorded-tests`. Install writes the Skill files into the project, so every session loads them.
>
> ## Fork workflow
>
> A fork creates an editable local Skill with its original author and licence. The request authorizes copying and local installation.
> 1. Check `./skills/recorded-tests`, the project lockfile, and selected Agent targets together. If the local directory or installed Skill exists, stop. Never overwrite an existing directory or Agent target.
> 2. Read [source metadata](https://skilld.dev/api/v1/skills/nvidia-nemo/guardrails/recorded-tests) once. Use sourceUrl, sourceCommit, skillPath, sourceGone, and license. If the source is gone or its path is missing, stop. If license is null, read licence files at the source commit.
> 3. Fetch only the source commit into a temporary Git repository. Do not clone full history. Derive repository_url from sourceUrl, including repository renames. If sourceCommit is absent, resolve the sourceUrl ref once. Set source_commit to that actual commit. Run these commands in one shell call:
>
> ```sh
> git init --quiet "$temporary_dir"
> git -C "$temporary_dir" fetch --quiet --depth=1 "$repository_url" "$source_commit"
> git -C "$temporary_dir" checkout --quiet --detach FETCH_HEAD
> ```
>
> Read applicable licence declarations and notices at that commit. If copying is not permitted, report the restriction and stop.
> 4. Inspect source entries together, then copy the directory containing skillPath into `./skills/recorded-tests`. Use the user's path if selected. Keep the original SKILL.md, relative links, scripts, binary assets, and executable modes. Exclude .git metadata. Reject symlinks and paths outside the Skill directory. After checking entries, use cp -a where available. A regular source directory needs no custom copy script. Do not save this page wrapper as SKILL.md.
> Preserve author credit, notices, and applicable licence files from repository or parent directories. Add PROVENANCE.md with the Skill page, source URL, actual commit, original path, and licence. Retain any existing PROVENANCE.md and record new provenance separately. Batch source inspection, copying, and provenance work where practical.
> 5. In the project root, run `skilld install ./skills/recorded-tests --mode copy --plain`. If skilld is unavailable, use `npx skilld install ./skills/recorded-tests --mode copy --plain`. This known command needs no help lookup. Install does not support --json. Use detected Agent targets, or add --agent for the targets the user selected. Install the local path, never the upstream selector. If installation fails, preserve the local copy and report the exact failure.
> 6. Confirm the local lockfile source and installed Agent copies once. Report the local path, actual commit, and Agent targets. After edits, reinstall the same local path. Upstream updates must not replace it. Do not publish or push unless the user asks.

# Recorded Tests: When and How

`tests/recorded/` is primarily a replay suite: it pins real provider wire
traffic in cassettes and pins the public API outputs produced from that traffic
in inline snapshots. Its value is refreshability: `make rewrite-cassettes`
re-records against live providers and detects drift. `make record-cassettes` runs record
mode `once`, so it only fills in cassettes that are missing and will not
refresh an existing one. A test that records no provider
traffic gets none of that value. Two repository policies still place selected
non-provider cases here: public-API equivalence/validation contracts and the
mandatory public entry for each library rail. Those tests do not fabricate a
cassette.

Read `tests/recorded/README.md` before writing any test in this suite. This
skill decides WHETHER and WHERE a recorded test is needed; the README defines
HOW the suite works.

## When a change requires a recorded test

The default criterion: a recording is required when the change alters bytes
sent to a provider, the interpretation of bytes received from one, or the
public output shape derived from them. A cassette-backed expectation must come
from the provider rather than from us. Apply the two repository-owned
non-cassette exceptions below separately.

Concretely, add or update a recorded test when:

1. **The wire path changes**: `nemoguardrails/llm/clients/` (request
   construction, SSE parsing, error mapping, usage extraction). Fakes here
   only encode our own assumptions about the provider dialect. Division of
   labor: `tests/llm/clients/` with `httpx.MockTransport` owns synthetic wire
   conditions (status-code mapping, malformed bodies); `tests/recorded/clients/`
   owns real traffic (happy path, real error shapes).
2. **The public API output contract changes over provider-driven flows**:
   `generate_async` / `stream_async` / `check` output shapes, new
   `GenerationResponse` fields, usage metadata, streaming chunk framing. If
   such outputs change and no snapshot under `tests/recorded/rails/public_api/`
   moved, either coverage has a hole or the change did not do what you think.
3. **A new provider integration, engine, or model type is added**: minimum one
   happy-path recorded test plus one provider-error test (the invalid-model
   404 pattern), so the integration is drift-detectable from day one.
4. **A model-calling library rail is added or its prompt/parsing changes**
   (content safety, topic control, jailbreak detection, self check): the
   block/allow decision depends on real model output, so
   `tests/recorded/rails/library/` needs the outcome triad (allow, block,
   error). Pure Python rails such as `regex` need a suite entry too, since
   `nemoguardrails/library/README.md` requires recorded e2e coverage for
   every new library rail; they get non-vcr suite tests with their own config
   directory (exemplar: `tests/recorded/rails/library/test_regex.py`), not
   zero coverage. For the full contribution checklist of a new rail, see the
   `add-library-rail` skill.
5. **A bug fix whose trigger is provider-shaped data**: the bug only
   reproduces with a real payload (an SSE framing edge, a missing usage field,
   a provider error body variant). The regression test must be recorded, or a
   fake cassette per the README ladder if the payload is unrecordable.

A cassette-backed test is NOT required when:

- The change is deterministic runtime logic, even when the diff touches
  `llmrails.py` (our own error envelopes, action dispatch, rail sequencing,
  fail-closed handling of local failures). Unit test in `tests/`.
- The change is a pure refactor. The recorded suite's job in a refactor is to
  be RUN, not extended: replay with `--block-network` before and after is the
  equivalence proof.
- The change is config or input validation, Colang runtime behavior, docs, or
  tooling, unless it changes a public-API contract intentionally pinned in the
  recorded suite or the mandatory library-rail entry.

File paths are a heuristic, not the rule: `llmrails.py` contains both
wire-adjacent contract code and deterministic logic in the same file. Always
apply the byte-level criterion above.

## Placement checklist

Before applying the provider-traffic checklist, preserve these existing suite
responsibilities:

- `tests/recorded/rails/public_api/` may pin deterministic validation and
  public method contracts used for engine/refactor equivalence.
- `tests/recorded/rails/library/` has one public entry per library rail by
  repository policy. A pure-Python rail uses a non-vcr test; IORails parity may
  reuse a sibling cassette or deliberately exercise a no-cassette case.

For every other scenario, answer these in order. A "no" that routes to
`tests/` is final.

1. **Does the scenario send at least one real provider request?** Main model,
   a rail's model, or embeddings must actually cross the wire during the test.
   If nothing crosses the wire, there is nothing to record. Unless one of the
   two suite responsibilities above applies, write a unit test in `tests/`
   using `TestChat` with `llm_completions` (see `tests/utils.py`), or
   `FakeLLMModel` from `nemoguardrails.testing.fake_model` directly.

   Red flags that mean the answer is "no":
   - The test passes `generator=` to `stream_async` AND no rail model or
     vendor call is made. With `config.rails.output.streaming.enabled`
     false, `stream_async` returns the generator directly and NO rail runs;
     when it is true only output rails run. `generator=` bypasses the MAIN
     model, so a
     rail whose own model or HTTP call still crosses the wire remains
     recorded-suite material; the library suite's own `stream_with_fake_main`
     helper is built on it.
   - The only rail action involved is a local Python action registered by the
     test config (a fake checker, a deliberately failing action).
   - The assertion fires before any model call (input validation,
     `pytest.raises` on config errors).
   - The effective test markers do not include `vcr`. That means no cassette;
     require one of the explicit suite responsibilities above.

2. **Is the interesting behavior in the provider response, or in our own
   deterministic logic?** Provider response shape (streaming chunk framing,
   error body format, usage metadata, tool-call structure, header handling)
   belongs here. Deterministic library logic belongs in `tests/`, with the
   separate public library entry required above. The question
   to ask: "if the provider changed its behavior tomorrow, could this test's
   expected output change?" If it cannot, the test does not need a recording.

3. **Does an equivalent test already exist?** Search both `tests/` and
   `tests/recorded/` for the behavior. Do not duplicate the same assertion
   unless the recorded version exercises provider-shaped behavior or the
   library/public-API suite contract at a distinct public layer.

4. **Is the scenario a failure that cannot occur against a live provider?**
   Follow the negative-path ladder in `tests/recorded/README.md`: prefer a
   recordable real error (for example an invalid model name yielding a real
   404), then pure runtime `pytest.raises`, and only as a last resort a
   `@pytest.mark.fake_cassette` with the required metadata. If the failure is
   injected by local test code rather than the provider, that is rule 1
   territory: normally a unit test.

## Colang dialects

LLMRails library recordings are dialect-single: do not duplicate a provider
recording across Colang 1 and Colang 2 when both dialects produce identical
wire bytes. Deterministic flow routing belongs in both-dialect unit tests.
IORails parity tests are engine-equivalence tests rather than another Colang
dialect; they may reuse the LLMRails cassette and may contain deterministic
no-cassette cases. One separate exception exists:

- In `tests/recorded/rails/public_api/`, the v2 runtime is a different
  generation pipeline (different prompts and LLM calls for dialog and
  generation surfaces), so v2 coverage there records genuinely different
  provider traffic and is legitimate recorded-suite material.

## Non-vcr and mixed-marker suite tests

The suite intentionally hosts non-vcr, pure-runtime tests
(for example `StreamingNotSupportedError` validation, `generate_async`
argument validation over `FakeLLMModel`, and the `check()` contracts over
local-rail configs) because `public_api` doubles as the public API surface
contract used for refactor equivalence proofs. This exception is a
convention, not an enforcement: nothing in `tests/recorded/conftest.py`
inspects markers, so a misplaced test collects and passes silently and no
tooling will tell you it is in the wrong suite. A non-vcr suite test carries
the module-level `pytestmark = [pytest.mark.recorded]` and no `vcr` mark,
adding `pytest.mark.asyncio` only when the module's tests are async
(exemplar: `tests/recorded/rails/library/test_regex.py` for the async form,
`tests/recorded/test_fake_cassettes.py` for the synchronous one).

That describes tests you add that exercise the library through this suite.
The cassette meta-tests directly under `tests/recorded/` are infrastructure
for the suite itself rather than replay tests, and they do not all follow
it: `test_cassette_provenance.py` carries no `pytestmark` at all. Nothing
rejects that either, so do not read those modules as a template for a new
suite test. Because nothing fails for you,
apply these criteria yourself. Add a non-vcr suite test only when all of
these hold:

- it pins a public API contract of the same surface as the neighboring
  recorded tests, or is input validation on that surface,
- it has no natural home in `tests/` (no equivalent unit test exists or could
  express the same public contract),
- it needs no new config directory of its own, EXCEPT for a rail under
  `tests/recorded/rails/library/`, where `nemoguardrails/library/README.md`
  mandates recorded coverage and a per-rail config directory is the usual
  shape. It is a convention, not a documented requirement, and the suite also
  holds shared multi-rail directories (`configs/full_stack`,
  `openai_input_stack`); reuse one before adding your own.

When in doubt, put it in `tests/`. Do not park a test here just because it
collects; nothing is checking, so the discipline is yours.

A module-level `vcr` mark is a default, not proof that every test opens a
cassette. A mixed-purpose parity module may deliberately route a deterministic
test to an empty cassette directory. Review cassette completeness against the
test's actual provider path and cassette routing, not the inherited marker
alone.

## Worked examples

Belongs in `tests/recorded/`:

- Streaming against a real OpenAI model, pinning chunk framing and final
  content (`test_openai_stream_async_public_contract`), with usage metadata
  pinned separately by
  `test_stream_async_matches_recorded_chat_completion_metadata`. The chunk
  boundaries come from the provider's SSE stream; only a recording can pin
  them honestly.
- A content-safety rail whose checker model is a real provider call, pinning
  what the rail blocks. The block decision depends on the real model output.
- An invalid model name producing the provider's real 404, pinning the
  status-bearing `LLMCallException` on LLMRails and IORails. The error body is
  provider-shaped and refreshable. IORails converts ordinary action errors
  without an upstream HTTP status to a blocking failed outcome instead;
  synthetic error-path tests belong in `tests/`.

Belongs in `tests/` instead:

- A rail action that raises `RuntimeError`, asserting the stream yields an
  internal-error envelope. The action is local, the envelope is constructed
  by our code, and `TestChat(llm_completions=...)` covers it with zero
  cassettes. (See `tests/test_streaming_internal_errors.py` for the canonical
  pattern.)
- Any assertion on our own JSON error format, rail ordering, or config
  validation.

## How, once placed in tests/recorded/

Follow `tests/recorded/README.md` exactly. The short version:

- Module-level `pytestmark = [pytest.mark.recorded, ...]`; add
  `@pytest.mark.vcr` per test when the module mixes vcr and non-vcr tests.
- Request credentials as fixture parameters (`openai_api_key`,
  `nvidia_api_key`); the fixture name is what gates recording.
- Config constants live in the suite-local `configs.py`; reuse an existing
  config directory before creating a new one.
- Record with credentials using the first pytest command in
  `tests/recorded/README.md`, fill an empty snapshot offline with
  `--inline-snapshot=create`, then verify replay with `--block-network`.
  `--inline-snapshot=fix` only updates an existing value. The current
  `make record-cassettes` target runs its record pass before snapshot creation,
  so a brand-new empty `snapshot()` stops that target at the expected teardown
  error; use the documented explicit two-pass workflow for a new test.
- Name negative tests `test_<surface>_<failure>_<behavior>` with `_raises`,
  `_fails_closed`, or `_invalid_*` suffixes.

## Review checklist for diffs touching tests/recorded/

- Test that actually sends or replays provider traffic with no committed
  cassette: incomplete, must be recorded. A module-level `vcr` mark can cover
  a deliberate deterministic no-cassette test, so inspect its routing.
- New non-vcr suite test: challenge it against the exception criteria above;
  it usually signals a test that belongs in `tests/`. Nothing enforces this,
  so read every test in the diff that has no `vcr` mark.
- New config directory for a single deterministic test: the test is in the
  wrong suite, unless it is the per-rail config for a new
  `nemoguardrails/library/` rail, which is the usual shape there.
- Same behavior asserted in both `tests/` and `tests/recorded/` in one change:
  require a provider-shaped distinction or one of the explicit public
  contract/library policy reasons; otherwise keep the unit test.
- Diff changes `nemoguardrails/llm/clients/` or a model-calling library rail
  with no change under `tests/recorded/`: ask where the recorded coverage is.
