/sketch

Content Policy Guardrails

Purpose

Every commercial image-generation pipeline needs guardrails. Not every prompt should generate; not every output should ship. This reference defines the layered policy stack: pre-prompt filtering, post-generation NSFW classification, brand safety, and regional regulatory compliance.

Scope Boundary

• IN scope: pre-prompt term filtering, persona refusal patterns, post-generation NSFW classifier wiring, brand-safety check (deepfake / public figure / minor / trademark), regional regulatory compliance (EU AI Act, China deep-synthesis, US state laws).
• OUT of scope: image generation itself (generate), prompt construction (prompt / cinematic), provenance / disclosure (provenance), upscaling (upscale), legal contract review (delegate to clause).

Core Concepts

The Five-Layer Policy Stack

Each layer catches what others miss. Single-layer pipelines fail.

Layer 1: Pre-Prompt Term Filter

Implementation: regex + semantic-similarity check against deny-list embeddings. Fast, conservative.

Layer 2: Persona / System Refusal

System prompt invariants for the generation pipeline:

You generate images. You will refuse:
- Sexual content involving minors (always).
- Sexually explicit images of real public figures.
- Violence depicting identifiable real people.
- Instructions to mimic a specific brand or product without explicit license.
- Misleading impersonation (deepfake).

When refusing, return a structured refusal: {reason, category, suggested_alternative}.

This is provider-internal but should also be enforced at the wrapper level (since system prompts can leak / drift).

Layer 3: Provider-Side Moderation

Each provider has built-in safety:

Open-weight models hosted yourself = full responsibility for all 5 layers. Hosted-API models = layers 1-2 + 4-5 still your responsibility, layer 3 the provider's.

Layer 4: Post-Generation Classifier

Even with layers 1-3, generated content may violate policy. Classifier ensemble:

Threshold tuning is critical:

Default: 0.6 for first-pass; adjust per-customer / per-context.

Layer 5: Human Review

Required for:

• Commercial campaigns (every asset).
• Public-figure depictions (always).
• Editorial / journalism (every asset).
• Government / regulated industry (every asset).

SLA: <2 hours during business; queue overnight if non-urgent.

Brand Safety Audit

Document the brand-safety bar in advance; ad hoc reviews drift.

Regional Regulatory Compliance

For each region in your distribution:

1. Check current rules (links go stale).
2. Map rules to layers 1-5 in the stack.
3. Document compliance posture.
4. Engage local legal for regulated industries.

Provider Policy Synchronization

Every provider has a policy doc that updates frequently:

Track changes; pipeline policy must keep pace.

Refusal UX

When the pipeline refuses:

Avoid vague "Sorry, can't generate that" — frustrating for legitimate users.

Metrics and Audit

Track:

Monthly audit: sample 100 rejected and 100 approved; verify decisions.

Red-Team Testing

Before launch and monthly thereafter:

1. Generate adversarial prompts (jailbreak, edge-case, evasion).
2. Run through full stack.
3. Score each layer's catch rate.
4. Patch gaps.
5. Document red-team report.

Standard test categories:

• CSAM (always layer 1 reject).
• Public-figure sexual content.
• Public-figure violent content.
• Real-event misinformation.
• Trademark misuse.
• Hate symbols.
• Self-harm content.
• Copyrighted characters in unauthorized contexts.
• Political ad without disclosure.
• Medical misinformation imagery.

Open-Weight Model Considerations

Self-hosted FLUX / SDXL has zero built-in moderation. The provider-layer responsibility shifts to you:

• Train or fine-tune a safety classifier.
• Apply pre-generation NSFW filter (CompVis Safety Checker, NudeNet).
• Log all prompts.
• Disclose AI generation to users.
• Document training-data attestation.
• Apply C2PA + SynthID-equivalent (Stable Signature, etc.).

The cost difference between hosted-API and self-hosted is partly the cost of moderation infrastructure.

Vendor / Platform Differences

Match pipeline strictness to lowest-common distribution channel.

Anti-Patterns

Workflow

1. Map distribution channels — list all targets; identify lowest-common policy bar.
2. Configure layer 1 — deny-list (banned terms + slurs + illegal categories) per region.
3. Author layer 2 — system prompt invariants; refusal catalog.
4. Verify layer 3 — provider safety filters active.
5. Wire layer 4 — NSFW + violence + face-recognition + brand classifier.
6. Set thresholds — start balanced (0.6); adjust per-feedback.
7. Define layer 5 — human review queue; SLAs; reviewer training.
8. Document refusal UX — message, alternative, appeal.
9. Set up logging — prompt, refusal category, layer, user; retain 90+ days.
10. Run red-team — pre-launch + monthly; 50+ adversarial prompts.
11. Configure region map — EU / China / US / etc. compliance posture.
12. Track provider policy updates — quarterly cadence.
13. Audit metrics — monthly false-positive / false-negative review.

Output Template

content_policy:
  distribution_channels: [internal_marketing, adobe_stock, instagram, x]
  lowest_common_bar: adobe_stock
  layer_1_pre_prompt:
    deny_list_path: configs/deny-en.txt + configs/deny-ja.txt + configs/deny-zh.txt
    semantic_similarity_check: yes
    threshold: 0.85
    fast_path_pct_of_traffic: 95
  layer_2_persona_refusal:
    system_prompt_version: v3.2
    refusal_catalog_count: 47
    structured_refusal: yes
  layer_3_provider:
    primary: google_vertex_imagen
    safety_setting: high
    auto_filter: yes
  layer_4_post_classifier:
    nsfw: nudenet_v3
    violence: detoxify_image
    face_recognition: facenet_against_sanctioned_list + public_figures
    brand_logo: yolov8_brand_classifier
    minor_age_est: insightface_age_model
    threshold: 0.6
  layer_5_human_review:
    triggers: [commercial, public_figure, editorial, regulated_industry]
    sla_hours: 2
    reviewer_training_done: yes
  refusal_ux:
    message_template: specific_category_with_alternative
    appeal_path: support_email
    rate_limit_repeat_offenders: yes
  region_compliance:
    eu:
      ai_act_article_50: complies_via_provenance + disclosure
      ai_act_article_5: layer_1_blocks_subliminal_manipulation
    cn:
      deep_synthesis_labeling: yes
      content_review_log: yes
    us:
      state_political_ad_disclosure: per_state
  provider_policy_review_cadence: quarterly
  metrics:
    rejection_rate_layer_1_pct: 2.1
    rejection_rate_layer_4_pct: 0.7
    human_review_queue_avg_hours: 1.4
    false_positive_audit_pct: 4.2
    false_negative_red_team_pct: 0.4
  red_team:
    last_run: 2026_04_15
    adversarial_prompts_tested: 62
    catch_rate_pct: 98
  log:
    retention_days: 180
    storage: s3_with_object_lock
  audit_cadence: monthly

Anti-Patterns

• One-layer policy — every layer has gaps.
• No layer 4 post-classifier — layer 1-3 leaks ship.
• No human review on commercial — brand crisis waiting.
• Vague refusal UX — user frustration drives jailbreak attempts.
• Static deny-list never updated — slang slips through.
• No region map — EU / China violations.
• No red-team testing — first attack succeeds.
• "Strict for everyone" — kills legitimate creative work.
• "Lenient for everyone" — guarantees violations.
• No false-positive audit — strictness drift unnoticed.
• Skipping layer on "test" runs — tests leak.
• Self-hosted models without classifier — full risk transfer to operator.
• Provider policy changes ignored — falling out of compliance silently.
• Policy mapped to highest distribution bar without per-channel customization — friction for lower-bar channels.
• No appeal path — legitimate user complaints become PR issues.
• Hard ban without log — no pattern analysis; same case recurs.
• No reviewer training — inconsistent decisions.
• No metrics — quality drift unnoticed.

Deliverable Contract

A content-policy stack is complete when:

• All 5 layers configured.
• Lowest-common distribution channel identified.
• Provider-policy alignment confirmed.
• Region-specific compliance documented (EU / China / US / etc.).
• Refusal UX templated.
• Logging configured (90+ day retention).
• Red-team testing run pre-launch.
• Metrics tracked (rejection rate, FP / FN).
• Audit cadence set (monthly).
• Reviewer training documented for layer 5.
• Provider policy review cadence (quarterly).

References

• EU AI Act Regulation (EU) 2024/1689.
• China Provisions on Administration of Deep Synthesis Internet Information Services (2023).
• China Interim Measures for Generative AI Services (2023).
• US Executive Order 14110 (2023).
• California AB 2655, AB 2839 (deepfake political ads).
• UK Online Safety Act 2023.
• Adobe Stock AI policy.
• Getty Images AI policy statement.
• Meta synthetic-content labeling policy.
• TikTok AI-generated content policy.
• YouTube Community Guidelines, synthetic-content rules.
• NudeNet — github.com/notAI-tech/NudeNet.
• Stability Safety Checker.
• DeepMind SynthID.
• Anthropic Constitutional AI.
• OWASP LLM Top 10 (2024) — analogous risk catalog.
• Partnership on AI — synthetic-media guidelines.
• C2PA standard (provenance complement).
• Lakera, Image Generation Attacks (2024-2026).