Index of every page on this site: llms.txt
All skills
jeremylongshore avatar

/langchain-content-blocks

@820dfc8
Run it yourself

Works correctly with LangChain 1.0's typed content blocks on AIMessage.content — text, tool_use, image, thinking, document — across Claude, GPT-4o, and Gemini, including multi-modal composition and tool-call iteration. Use when composing multi-modal messages, iterating tool_use blocks, handling Claude's thinking content, or unifying image inputs across providers. Trigger with "langchain content blocks", "AIMessage.content", "tool_use block", "claude image input", "langchain multimodal", "thinking block replay", "claude citations".

Use this Skill: https://skilld.dev/gh/jeremylongshore/tons-of-skills-marketplace/langchain-content-blocks

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

referencesmultimodal-composition.md

≈2.1k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Multi-Modal Composition — Universal image Block + Documents

LangChain 1.0 unifies the image-block shape across providers (P64). This reference covers the universal shape, base64 vs URL tradeoffs, per-provider size limits, and the document block used by Claude's citations API.

The universal image block (1.0)

{
    "type": "image",
    "source_type": "base64",         # or "url"
    "data": "<base64 string or URL>",
    "mime_type": "image/png",        # required for base64
}

Compose messages with this shape on any provider — the adapter translates to the wire format Claude, OpenAI, or Gemini expects. Do not hand-roll {"type": "image_url", ...} (OpenAI pre-1.0) or {"type": "image", "source": {"type": "base64", ...}} (Anthropic pre-1.0).

Composing a text + image HumanMessage

import base64
from pathlib import Path
from langchain_core.messages import HumanMessage

def image_block_from_file(path: str) -> dict:
    mime = {"png": "image/png", "jpg": "image/jpeg",
            "jpeg": "image/jpeg", "gif": "image/gif",
            "webp": "image/webp"}[Path(path).suffix.lstrip(".").lower()]
    data = base64.standard_b64encode(Path(path).read_bytes()).decode("ascii")
    return {
        "type": "image",
        "source_type": "base64",
        "data": data,
        "mime_type": mime,
    }

msg = HumanMessage(content=[
    {"type": "text", "text": "Describe what is broken in this screenshot."},
    image_block_from_file("screenshot.png"),
])
response = claude.invoke([msg])

Two rules that real code breaks:

  1. content must be a list when including non-text blocks. Passing content="Describe..." and then trying to "attach" an image elsewhere silently drops the image. All blocks go in the list.
  2. Order matters for Claude. Put the image block before the instruction text for best results — Claude attends most to the trailing tokens. Empirically, putting the instruction last improves follow-through on multi-image prompts.

Base64 vs URL

source_type When to use Tradeoffs
"base64" Default. Private files, generated bytes, anything not publicly hosted. Every byte ships in the request. For a 4MB PNG you pay ~5.4MB (base64 is 33% larger) of network + prompt tokens.
"url" Public, stable URLs (CDN assets, public S3). Zero upload cost, but the provider fetches the URL server-side — authentication headers you set are NOT forwarded. Private URLs behind auth: use base64 instead.

Claude also accepts URLs via source_type="url" as of anthropic >= 0.40. Earlier Anthropic SDKs required base64 only.

Per-provider size limits

Exact limits, as of November 2024 / January 2026 docs — these change; always verify against provider docs before scaling:

Provider Per-image limit Per-request image count Per-request total
Anthropic (Claude 3.5/4.x) 5 MB per image up to 20 images 8000 tokens max for images + text
OpenAI (GPT-4o) 20 MB per image no hard count combined with text within context window (128k)
Google (Gemini 2.5) 20 MB per request total no hard count 20 MB total including all images

Exceeding these:

  • Anthropic: anthropic.BadRequestError: image exceeds 5 MB limit
  • OpenAI: HTTP 400 "image_url is too large"
  • Gemini: google.api_core.exceptions.InvalidArgument: Request payload size exceeds the limit

Pre-resize before sending. For UX screenshots, 1024x1024 at JPEG quality 85 is typically < 500KB and preserves enough detail for Claude/GPT-4o vision to read UI text.

MIME type rules

mime_type Claude GPT-4o Gemini 2.5
image/png yes yes yes
image/jpeg yes yes yes
image/gif yes (static frame only) yes yes
image/webp yes yes yes
image/bmp no yes no
image/heic, image/heif no no yes

If you have iPhone HEIC uploads and need Claude or OpenAI compatibility, convert to JPEG at ingest. pillow-heif + Pillow handles this in about four lines.

What LangChain's adapter actually does

When you pass the universal image block to ChatAnthropic.invoke(), the adapter in langchain-anthropic rewrites it to Anthropic's native wire format:

# Your code passes:
{"type": "image", "source_type": "base64",
 "data": "...", "mime_type": "image/png"}

# langchain-anthropic sends on the wire:
{"type": "image", "source":
 {"type": "base64", "media_type": "image/png", "data": "..."}}

On ChatOpenAI:

# On the wire:
{"type": "image_url",
 "image_url": {"url": "data:image/png;base64,..."}}

On ChatGoogleGenerativeAI:

# Translated to google.generativeai.protos.Part with inline_data
{"inline_data": {"mime_type": "image/png", "data": "..."}}

You should not see these wire shapes in your code. If you do, you are bypassing the adapter (usually by importing anthropic.Anthropic directly inside a chain). Keep the universal shape at the LangChain boundary.

document blocks (Claude citations API)

A parallel feature to images: document blocks let Claude attach citations to specific output text. Input shape:

from langchain_core.messages import HumanMessage

doc_block = {
    "type": "document",
    "source": {
        "type": "base64",
        "media_type": "application/pdf",
        "data": base64_pdf_bytes,
    },
    "title": "Q3 Earnings Report",
    "citations": {"enabled": True},
}

msg = HumanMessage(content=[
    doc_block,
    {"type": "text", "text": "What were the key revenue drivers?"},
])

Supported source types: PDF (application/pdf), plain text (text/plain), and custom content (content array of text blocks).

The output side. Citations do NOT come back as top-level citation blocks. They attach to text blocks as citations arrays:

for block in response.content:
    if block.get("type") != "text":
        continue
    print(block["text"])
    for c in block.get("citations", []):
        print(f"  source: {c['document_title']}")
        print(f"  quoted: {c['cited_text']!r}")
        print(f"  location: chars {c['start_char_index']}-{c['end_char_index']}")

msg.text() strips the citations metadata. To surface citations you must iterate content manually.

Provider adapter checklist

Before composing a multi-modal message:

  1. Is content a list[dict]? (Not a str with an attached image.)
  2. Are all image blocks in the universal 1.0 shape (source_type, data, mime_type)?
  3. Is each image under the provider's per-image limit? (5 MB Anthropic, 20 MB OpenAI, 20 MB total Gemini.)
  4. Is the mime_type supported on the target provider? (See table above.)
  5. If using document blocks: are you using ChatAnthropic Sonnet 4+? (No other provider supports them.)
  6. If round-tripping a response: did you preserve any thinking blocks? (See thinking-blocks.md.)

Errors you will hit

Error Cause Fix
anthropic.BadRequestError: image exceeds 5 MB limit Un-resized screenshot Pre-resize to < 5 MB; 1024x1024 JPEG 85 is typically < 500 KB
openai.BadRequestError: Invalid image data Missing data:image/png;base64, prefix when hand-rolling Use the universal block — LangChain adapts the prefix
google.api_core.exceptions.InvalidArgument: Request payload size exceeds the limit Total request > 20 MB on Gemini Chunk requests; Gemini's limit is per-request total
Content shows up as garbled text HumanMessage(content="...") (string) with an attempted image attachment elsewhere Put all blocks inside the content list
Image ignored by model Image block after the instruction, and instruction says "If you see an image..." Put the image block BEFORE the instruction; Claude attends most to trailing tokens
Citations missing from output Read via msg.text() which strips metadata Iterate msg.content and read block["citations"] on text blocks

References

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts2mo3 checks · Risk SAFE
  • Gen Agent Trust Hub2mo

    The skill is a professional developer resource providing safe and standard patterns for utilizing LangChain 1.0 content blocks across various AI providers. It correctly addresses common pitfalls like model-specific data formats and streaming complexities without introducing security risks.

  • Socket2mo

    No alerts

  • Snyk2mo

    Risk: LOW · No issues

Signed by skilld at 820dfc8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 2 months ago
What it can do
Reads files Edits files Runs commands
version
2.7.0
author
Jeremy Longshore <jeremy@intentsolutions.io>
compatibility
Designed for Claude Code
All 4 allowed tools
ReadWriteEditBash(python:*)
Other metadata
tags
[
  "saas",
  "langchain",
  "langgraph",
  "python",
  "langchain-1.0",
  "content-blocks",
  "multimodal",
  "tool-use"
]

README badge

README badge for jeremylongshore/tons-of-skills-marketplace/langchain-content-blocks