Multi-Modal Composition — Universal image Block + Documents
LangChain 1.0 unifies the image-block shape across providers (P64). This
reference covers the universal shape, base64 vs URL tradeoffs, per-provider
size limits, and the document block used by Claude's citations API.
The universal image block (1.0)
{
"type": "image",
"source_type": "base64", # or "url"
"data": "<base64 string or URL>",
"mime_type": "image/png", # required for base64
}Compose messages with this shape on any provider — the adapter
translates to the wire format Claude, OpenAI, or Gemini expects. Do not
hand-roll {"type": "image_url", ...} (OpenAI pre-1.0) or
{"type": "image", "source": {"type": "base64", ...}} (Anthropic
pre-1.0).
Composing a text + image HumanMessage
import base64
from pathlib import Path
from langchain_core.messages import HumanMessage
def image_block_from_file(path: str) -> dict:
mime = {"png": "image/png", "jpg": "image/jpeg",
"jpeg": "image/jpeg", "gif": "image/gif",
"webp": "image/webp"}[Path(path).suffix.lstrip(".").lower()]
data = base64.standard_b64encode(Path(path).read_bytes()).decode("ascii")
return {
"type": "image",
"source_type": "base64",
"data": data,
"mime_type": mime,
}
msg = HumanMessage(content=[
{"type": "text", "text": "Describe what is broken in this screenshot."},
image_block_from_file("screenshot.png"),
])
response = claude.invoke([msg])Two rules that real code breaks:
contentmust be alistwhen including non-text blocks. Passingcontent="Describe..."and then trying to "attach" an image elsewhere silently drops the image. All blocks go in the list.- Order matters for Claude. Put the image block before the instruction text for best results — Claude attends most to the trailing tokens. Empirically, putting the instruction last improves follow-through on multi-image prompts.
Base64 vs URL
| source_type | When to use | Tradeoffs |
|---|---|---|
"base64" |
Default. Private files, generated bytes, anything not publicly hosted. | Every byte ships in the request. For a 4MB PNG you pay ~5.4MB (base64 is 33% larger) of network + prompt tokens. |
"url" |
Public, stable URLs (CDN assets, public S3). | Zero upload cost, but the provider fetches the URL server-side — authentication headers you set are NOT forwarded. Private URLs behind auth: use base64 instead. |
Claude also accepts URLs via source_type="url" as of anthropic >= 0.40.
Earlier Anthropic SDKs required base64 only.
Per-provider size limits
Exact limits, as of November 2024 / January 2026 docs — these change; always verify against provider docs before scaling:
| Provider | Per-image limit | Per-request image count | Per-request total |
|---|---|---|---|
| Anthropic (Claude 3.5/4.x) | 5 MB per image | up to 20 images | 8000 tokens max for images + text |
| OpenAI (GPT-4o) | 20 MB per image | no hard count | combined with text within context window (128k) |
| Google (Gemini 2.5) | 20 MB per request total | no hard count | 20 MB total including all images |
Exceeding these:
- Anthropic:
anthropic.BadRequestError: image exceeds 5 MB limit - OpenAI: HTTP 400 "image_url is too large"
- Gemini:
google.api_core.exceptions.InvalidArgument: Request payload size exceeds the limit
Pre-resize before sending. For UX screenshots, 1024x1024 at JPEG quality 85 is typically < 500KB and preserves enough detail for Claude/GPT-4o vision to read UI text.
MIME type rules
| mime_type | Claude | GPT-4o | Gemini 2.5 |
|---|---|---|---|
image/png |
yes | yes | yes |
image/jpeg |
yes | yes | yes |
image/gif |
yes (static frame only) | yes | yes |
image/webp |
yes | yes | yes |
image/bmp |
no | yes | no |
image/heic, image/heif |
no | no | yes |
If you have iPhone HEIC uploads and need Claude or OpenAI compatibility,
convert to JPEG at ingest. pillow-heif + Pillow handles this in
about four lines.
What LangChain's adapter actually does
When you pass the universal image block to ChatAnthropic.invoke(),
the adapter in langchain-anthropic rewrites it to Anthropic's native
wire format:
# Your code passes:
{"type": "image", "source_type": "base64",
"data": "...", "mime_type": "image/png"}
# langchain-anthropic sends on the wire:
{"type": "image", "source":
{"type": "base64", "media_type": "image/png", "data": "..."}}On ChatOpenAI:
# On the wire:
{"type": "image_url",
"image_url": {"url": "data:image/png;base64,..."}}On ChatGoogleGenerativeAI:
# Translated to google.generativeai.protos.Part with inline_data
{"inline_data": {"mime_type": "image/png", "data": "..."}}You should not see these wire shapes in your code. If you do, you are
bypassing the adapter (usually by importing anthropic.Anthropic
directly inside a chain). Keep the universal shape at the LangChain
boundary.
document blocks (Claude citations API)
A parallel feature to images: document blocks let Claude attach
citations to specific output text. Input shape:
from langchain_core.messages import HumanMessage
doc_block = {
"type": "document",
"source": {
"type": "base64",
"media_type": "application/pdf",
"data": base64_pdf_bytes,
},
"title": "Q3 Earnings Report",
"citations": {"enabled": True},
}
msg = HumanMessage(content=[
doc_block,
{"type": "text", "text": "What were the key revenue drivers?"},
])Supported source types: PDF (application/pdf), plain text
(text/plain), and custom content (content array of text blocks).
The output side. Citations do NOT come back as top-level citation
blocks. They attach to text blocks as citations arrays:
for block in response.content:
if block.get("type") != "text":
continue
print(block["text"])
for c in block.get("citations", []):
print(f" source: {c['document_title']}")
print(f" quoted: {c['cited_text']!r}")
print(f" location: chars {c['start_char_index']}-{c['end_char_index']}")msg.text() strips the citations metadata. To surface citations you
must iterate content manually.
Provider adapter checklist
Before composing a multi-modal message:
- Is
contentalist[dict]? (Not astrwith an attached image.) - Are all image blocks in the universal 1.0 shape (
source_type,data,mime_type)? - Is each image under the provider's per-image limit? (5 MB Anthropic, 20 MB OpenAI, 20 MB total Gemini.)
- Is the
mime_typesupported on the target provider? (See table above.) - If using
documentblocks: are you usingChatAnthropicSonnet 4+? (No other provider supports them.) - If round-tripping a response: did you preserve any
thinkingblocks? (Seethinking-blocks.md.)
Errors you will hit
| Error | Cause | Fix |
|---|---|---|
anthropic.BadRequestError: image exceeds 5 MB limit |
Un-resized screenshot | Pre-resize to < 5 MB; 1024x1024 JPEG 85 is typically < 500 KB |
openai.BadRequestError: Invalid image data |
Missing data:image/png;base64, prefix when hand-rolling |
Use the universal block — LangChain adapts the prefix |
google.api_core.exceptions.InvalidArgument: Request payload size exceeds the limit |
Total request > 20 MB on Gemini | Chunk requests; Gemini's limit is per-request total |
| Content shows up as garbled text | HumanMessage(content="...") (string) with an attempted image attachment elsewhere |
Put all blocks inside the content list |
| Image ignored by model | Image block after the instruction, and instruction says "If you see an image..." | Put the image block BEFORE the instruction; Claude attends most to trailing tokens |
| Citations missing from output | Read via msg.text() which strips metadata |
Iterate msg.content and read block["citations"] on text blocks |
References
- LangChain multimodal concepts: https://python.langchain.com/docs/concepts/multimodality/
- Anthropic vision: https://platform.claude.com/docs/en/build-with-claude/vision
- Anthropic citations: https://platform.claude.com/docs/en/build-with-claude/citations
- OpenAI vision: https://platform.openai.com/docs/guides/vision
- Gemini multimodal: https://ai.google.dev/gemini-api/docs/vision