/drasi

Synthetic user testing bundle

Use this bundle when creating or changing tutorials, getting-started flows, generated project scaffolds, examples, or documentation that a new user must follow.

Principle

Test Drasi guidance as a naive, literal, unforgiving user:

• Naive: assume no prior Drasi knowledge beyond what the instructions state.
• Literal: execute commands exactly as written.
• Unforgiving: fail when expected output, status, screenshots, or side effects do not match.

This prevents polished but non-executable AI-generated documentation.

When required

Run or design synthetic-user tests for:

• New Drasi project scaffolds.
• Getting-started documentation.
• Dev Container or Codespaces flows.
• End-to-end examples with PostgreSQL, MySQL, Kubernetes, Drasi Server, MCP, or reactions.
• Any reusable customer accelerator or internal template.

Test environment rules

• Use the same environment the user is expected to use: Dev Container, Codespaces, local Docker, k3d, AKS, or Azure Container Apps.
• Pin tool versions where possible: Drasi CLI, Docker image, Rust toolchain, database images, Node/Python helpers, and Azure tooling.
• Do not require hidden local state, pre-existing cluster context, pre-authenticated cloud sessions, or manual setup that is not documented.
• Use isolated test resources and teardown steps.
• Redact tokens, connection strings, and environment-specific identifiers from logs.

Test script shape

Every example should have a reproducible validation script or checklist:

set -euo pipefail

# 1. Verify tools
command -v drasi
command -v docker

# 2. Start dependencies
# docker compose up / k3d setup / azd up / cargo build

# 3. Apply or start Drasi resources
# drasi apply -f ...
# drasi-server --config ...

# 4. Wait explicitly, never with blind sleeps
# drasi wait -f query.yaml

# 5. Trigger known changes
# insert/update/delete test data

# 6. Assert query and reaction outputs
# jq/grep/curl/cargo test/playwright assertions

# 7. Cleanup

Prefer explicit waits such as drasi wait where supported. Avoid vague instructions like “wait until it is ready” without a command and expected output.

Acceptance criteria

A synthetic-user test passes only when:

• A clean environment can run the instructions from start to finish.
• Every command has the required prerequisites and expected output.
• Screenshots or UI checks match the current product UI when UI is involved.
• A source change produces the expected query result.
• The reaction side effect is verified, not merely created.
• The cleanup path works.
• The test can run repeatedly without hidden state pollution.

Common failures to catch

• Missing drasi wait or readiness checks.
• CLI output drift after product updates.
• Dev Container or Docker-in-Docker assumptions that broke silently.
• Database readiness races.
• Incomplete seed data.
• Examples that create resources but never prove event flow.
• Screenshots or UI labels that no longer match the product.
• Shell commands that depend on an undeclared working directory.

Security guardrails for agent-run tests

• Treat the container or sandbox as the security boundary.
• Use least-privilege tokens and short-lived credentials.
• Require maintainer approval for tests that can access external services or secrets.
• Deny broad web/file/network access unless the scenario explicitly needs it.
• Never allow synthetic tests to run destructive commands against shared or production environments.

Evidence template

### Synthetic-user test evidence

- Environment: <Dev Container/Codespaces/local/k3d/ACA/AKS>
- Fresh start: <yes/no>
- Commands executed exactly as documented: <yes/no>
- Source change tested: <insert/update/delete>
- Query result verified: <summary>
- Reaction side effect verified: <summary>
- Cleanup verified: <summary>
- Failures fixed: <summary>