Recorded Tests: When and How
tests/recorded/ is primarily a replay suite: it pins real provider wire
traffic in cassettes and pins the public API outputs produced from that traffic
in inline snapshots. Its value is refreshability: make rewrite-cassettes
re-records against live providers and detects drift. make record-cassettes runs record
mode once, so it only fills in cassettes that are missing and will not
refresh an existing one. A test that records no provider
traffic gets none of that value. Two repository policies still place selected
non-provider cases here: public-API equivalence/validation contracts and the
mandatory public entry for each library rail. Those tests do not fabricate a
cassette.
Read tests/recorded/README.md before writing any test in this suite. This
skill decides WHETHER and WHERE a recorded test is needed; the README defines
HOW the suite works.
When a change requires a recorded test
The default criterion: a recording is required when the change alters bytes sent to a provider, the interpretation of bytes received from one, or the public output shape derived from them. A cassette-backed expectation must come from the provider rather than from us. Apply the two repository-owned non-cassette exceptions below separately.
Concretely, add or update a recorded test when:
- The wire path changes:
nemoguardrails/llm/clients/(request construction, SSE parsing, error mapping, usage extraction). Fakes here only encode our own assumptions about the provider dialect. Division of labor:tests/llm/clients/withhttpx.MockTransportowns synthetic wire conditions (status-code mapping, malformed bodies);tests/recorded/clients/owns real traffic (happy path, real error shapes). - The public API output contract changes over provider-driven flows:
generate_async/stream_async/checkoutput shapes, newGenerationResponsefields, usage metadata, streaming chunk framing. If such outputs change and no snapshot undertests/recorded/rails/public_api/moved, either coverage has a hole or the change did not do what you think. - A new provider integration, engine, or model type is added: minimum one happy-path recorded test plus one provider-error test (the invalid-model 404 pattern), so the integration is drift-detectable from day one.
- A model-calling library rail is added or its prompt/parsing changes
(content safety, topic control, jailbreak detection, self check): the
block/allow decision depends on real model output, so
tests/recorded/rails/library/needs the outcome triad (allow, block, error). Pure Python rails such asregexneed a suite entry too, sincenemoguardrails/library/README.mdrequires recorded e2e coverage for every new library rail; they get non-vcr suite tests with their own config directory (exemplar:tests/recorded/rails/library/test_regex.py), not zero coverage. For the full contribution checklist of a new rail, see theadd-library-railskill. - A bug fix whose trigger is provider-shaped data: the bug only reproduces with a real payload (an SSE framing edge, a missing usage field, a provider error body variant). The regression test must be recorded, or a fake cassette per the README ladder if the payload is unrecordable.
A cassette-backed test is NOT required when:
- The change is deterministic runtime logic, even when the diff touches
llmrails.py(our own error envelopes, action dispatch, rail sequencing, fail-closed handling of local failures). Unit test intests/. - The change is a pure refactor. The recorded suite's job in a refactor is to
be RUN, not extended: replay with
--block-networkbefore and after is the equivalence proof. - The change is config or input validation, Colang runtime behavior, docs, or tooling, unless it changes a public-API contract intentionally pinned in the recorded suite or the mandatory library-rail entry.
File paths are a heuristic, not the rule: llmrails.py contains both
wire-adjacent contract code and deterministic logic in the same file. Always
apply the byte-level criterion above.
Placement checklist
Before applying the provider-traffic checklist, preserve these existing suite responsibilities:
tests/recorded/rails/public_api/may pin deterministic validation and public method contracts used for engine/refactor equivalence.tests/recorded/rails/library/has one public entry per library rail by repository policy. A pure-Python rail uses a non-vcr test; IORails parity may reuse a sibling cassette or deliberately exercise a no-cassette case.
For every other scenario, answer these in order. A "no" that routes to
tests/ is final.
Does the scenario send at least one real provider request? Main model, a rail's model, or embeddings must actually cross the wire during the test. If nothing crosses the wire, there is nothing to record. Unless one of the two suite responsibilities above applies, write a unit test in
tests/usingTestChatwithllm_completions(seetests/utils.py), orFakeLLMModelfromnemoguardrails.testing.fake_modeldirectly.Red flags that mean the answer is "no":
- The test passes
generator=tostream_asyncAND no rail model or vendor call is made. Withconfig.rails.output.streaming.enabledfalse,stream_asyncreturns the generator directly and NO rail runs; when it is true only output rails run.generator=bypasses the MAIN model, so a rail whose own model or HTTP call still crosses the wire remains recorded-suite material; the library suite's ownstream_with_fake_mainhelper is built on it. - The only rail action involved is a local Python action registered by the test config (a fake checker, a deliberately failing action).
- The assertion fires before any model call (input validation,
pytest.raiseson config errors). - The effective test markers do not include
vcr. That means no cassette; require one of the explicit suite responsibilities above.
- The test passes
Is the interesting behavior in the provider response, or in our own deterministic logic? Provider response shape (streaming chunk framing, error body format, usage metadata, tool-call structure, header handling) belongs here. Deterministic library logic belongs in
tests/, with the separate public library entry required above. The question to ask: "if the provider changed its behavior tomorrow, could this test's expected output change?" If it cannot, the test does not need a recording.Does an equivalent test already exist? Search both
tests/andtests/recorded/for the behavior. Do not duplicate the same assertion unless the recorded version exercises provider-shaped behavior or the library/public-API suite contract at a distinct public layer.Is the scenario a failure that cannot occur against a live provider? Follow the negative-path ladder in
tests/recorded/README.md: prefer a recordable real error (for example an invalid model name yielding a real 404), then pure runtimepytest.raises, and only as a last resort a@pytest.mark.fake_cassettewith the required metadata. If the failure is injected by local test code rather than the provider, that is rule 1 territory: normally a unit test.
Colang dialects
LLMRails library recordings are dialect-single: do not duplicate a provider recording across Colang 1 and Colang 2 when both dialects produce identical wire bytes. Deterministic flow routing belongs in both-dialect unit tests. IORails parity tests are engine-equivalence tests rather than another Colang dialect; they may reuse the LLMRails cassette and may contain deterministic no-cassette cases. One separate exception exists:
- In
tests/recorded/rails/public_api/, the v2 runtime is a different generation pipeline (different prompts and LLM calls for dialog and generation surfaces), so v2 coverage there records genuinely different provider traffic and is legitimate recorded-suite material.
Non-vcr and mixed-marker suite tests
The suite intentionally hosts non-vcr, pure-runtime tests
(for example StreamingNotSupportedError validation, generate_async
argument validation over FakeLLMModel, and the check() contracts over
local-rail configs) because public_api doubles as the public API surface
contract used for refactor equivalence proofs. This exception is a
convention, not an enforcement: nothing in tests/recorded/conftest.py
inspects markers, so a misplaced test collects and passes silently and no
tooling will tell you it is in the wrong suite. A non-vcr suite test carries
the module-level pytestmark = [pytest.mark.recorded] and no vcr mark,
adding pytest.mark.asyncio only when the module's tests are async
(exemplar: tests/recorded/rails/library/test_regex.py for the async form,
tests/recorded/test_fake_cassettes.py for the synchronous one).
That describes tests you add that exercise the library through this suite.
The cassette meta-tests directly under tests/recorded/ are infrastructure
for the suite itself rather than replay tests, and they do not all follow
it: test_cassette_provenance.py carries no pytestmark at all. Nothing
rejects that either, so do not read those modules as a template for a new
suite test. Because nothing fails for you,
apply these criteria yourself. Add a non-vcr suite test only when all of
these hold:
- it pins a public API contract of the same surface as the neighboring recorded tests, or is input validation on that surface,
- it has no natural home in
tests/(no equivalent unit test exists or could express the same public contract), - it needs no new config directory of its own, EXCEPT for a rail under
tests/recorded/rails/library/, wherenemoguardrails/library/README.mdmandates recorded coverage and a per-rail config directory is the usual shape. It is a convention, not a documented requirement, and the suite also holds shared multi-rail directories (configs/full_stack,openai_input_stack); reuse one before adding your own.
When in doubt, put it in tests/. Do not park a test here just because it
collects; nothing is checking, so the discipline is yours.
A module-level vcr mark is a default, not proof that every test opens a
cassette. A mixed-purpose parity module may deliberately route a deterministic
test to an empty cassette directory. Review cassette completeness against the
test's actual provider path and cassette routing, not the inherited marker
alone.
Worked examples
Belongs in tests/recorded/:
- Streaming against a real OpenAI model, pinning chunk framing and final
content (
test_openai_stream_async_public_contract), with usage metadata pinned separately bytest_stream_async_matches_recorded_chat_completion_metadata. The chunk boundaries come from the provider's SSE stream; only a recording can pin them honestly. - A content-safety rail whose checker model is a real provider call, pinning what the rail blocks. The block decision depends on the real model output.
- An invalid model name producing the provider's real 404, pinning the
status-bearing
LLMCallExceptionon LLMRails and IORails. The error body is provider-shaped and refreshable. IORails converts ordinary action errors without an upstream HTTP status to a blocking failed outcome instead; synthetic error-path tests belong intests/.
Belongs in tests/ instead:
- A rail action that raises
RuntimeError, asserting the stream yields an internal-error envelope. The action is local, the envelope is constructed by our code, andTestChat(llm_completions=...)covers it with zero cassettes. (Seetests/test_streaming_internal_errors.pyfor the canonical pattern.) - Any assertion on our own JSON error format, rail ordering, or config validation.
How, once placed in tests/recorded/
Follow tests/recorded/README.md exactly. The short version:
- Module-level
pytestmark = [pytest.mark.recorded, ...]; add@pytest.mark.vcrper test when the module mixes vcr and non-vcr tests. - Request credentials as fixture parameters (
openai_api_key,nvidia_api_key); the fixture name is what gates recording. - Config constants live in the suite-local
configs.py; reuse an existing config directory before creating a new one. - Record with credentials using the first pytest command in
tests/recorded/README.md, fill an empty snapshot offline with--inline-snapshot=create, then verify replay with--block-network.--inline-snapshot=fixonly updates an existing value. The currentmake record-cassettestarget runs its record pass before snapshot creation, so a brand-new emptysnapshot()stops that target at the expected teardown error; use the documented explicit two-pass workflow for a new test. - Name negative tests
test_<surface>_<failure>_<behavior>with_raises,_fails_closed, or_invalid_*suffixes.
Review checklist for diffs touching tests/recorded/
- Test that actually sends or replays provider traffic with no committed
cassette: incomplete, must be recorded. A module-level
vcrmark can cover a deliberate deterministic no-cassette test, so inspect its routing. - New non-vcr suite test: challenge it against the exception criteria above;
it usually signals a test that belongs in
tests/. Nothing enforces this, so read every test in the diff that has novcrmark. - New config directory for a single deterministic test: the test is in the
wrong suite, unless it is the per-rail config for a new
nemoguardrails/library/rail, which is the usual shape there. - Same behavior asserted in both
tests/andtests/recorded/in one change: require a provider-shaped distinction or one of the explicit public contract/library policy reasons; otherwise keep the unit test. - Diff changes
nemoguardrails/llm/clients/or a model-calling library rail with no change undertests/recorded/: ask where the recorded coverage is.