arXiv API Verification Protocol
Status: v3.11 (#182 Delta 1); #495 ToU-alignment refresh (2026-07)
Used by: bibliography_agent, scripts/contamination_signals.py (resolve_arxiv_unmatched)
API base: http://export.arxiv.org/api/query
Rate limit: arXiv's API Terms of Use (https://info.arxiv.org/help/api/tou.html): no more than one request every three seconds, a single connection at a time. The limits apply to all machines under the caller's control as a whole — multi-machine / multi-client fan-out to overcome them is explicitly prohibited. No polite-pool / higher-tier mechanism.
Polite email env var: none (arXiv has no such convention)
Purpose
Adds a fourth bibliographic-index lookup to the cross-index triangulation surface (S2 + OpenAlex + Crossref + arXiv) per spec v3.11 #182 Delta 1. arXiv is the preprint registry of record — strongest coverage for CS / physics / math preprints carrying an arXiv ID. It complements the three published-literature indexes: a citation with a bogus arXiv ID that fails to resolve is positive non-existence evidence, while a published paper without an arXiv ID is legitimately skipped by this resolver (arXiv applicability is ID-gated, not a coverage gap).
Mirrors the structure of crossref_api_protocol.md and openalex_api_protocol.md.
Response format
Unlike the JSON siblings, the arXiv query API returns an Atom 1.0 XML feed. The client parses it with xml.etree.ElementTree; the Atom namespace is {http://www.w3.org/2005/Atom}. A match yields one or more <entry> elements; a miss yields a feed with zero <entry> elements (not a 404). Per-entry fields the client reads:
<entry><title>— paper title (arXiv inserts internal whitespace/newlines, which the client collapses to single spaces before similarity comparison).<entry><published>— ISO-8601 timestamp; the leading 4 digits are the year (matching-year tiebreaker).
Query Patterns
Pattern 1: arXiv ID Lookup with Title Cross-Check (primary when an arXiv ID is available)
GET ?id_list={arxiv_id}Matching rule (mirrors the Crossref/S2 DOI_MISMATCH pattern): ID lookup hits are gated by a 0.70 title cross-check (same SequenceMatcher threshold as the sibling clients). If the resolved entry's title is below threshold → ID_MISMATCH, return None, fall through to title search. An empty feed (non-existent ID) is a miss → None.
Pattern 2: Title Search (fallback on ID-miss / ID_MISMATCH)
GET ?search_query=ti:"{title}"&max_results=5Matching rule: similarity >= 0.70. When multiple candidates pass, prefer the matching-year tiebreaker via a +0.05 score bonus (year read from <published>).
arxiv_unmatched derivation
true if and only if the citation has an arXiv ID AND the ID lookup either returns an empty feed (miss) or misses the title cross-check, AND the title-search fallback returns no match meeting threshold.
A citation with no arXiv ID is skipped (not unmatched) — the resolver does not run and the caller omits arxiv_unmatched (absent ≠ false, #331). arXiv applicability is ID-gated: a title-only miss against arXiv is a coverage gap for a non-arXiv work, not non-existence evidence, so it must never emit a triangulation signal. The check fires only when obtained_via != 'manual' AND an arXiv ID is present.
Note: the unified lookup_verified summary (#182 Delta 4, a later batch) narrows the existence-gate false to ID-keyed unmatched — a title-only arxiv_unmatched with no resolvable ID is a coverage-gap signal, not fabrication evidence (C-V6(a)). This protocol's arxiv_unmatched boolean is the raw triangulation signal; the narrowing happens at the Delta 4 reducer, not here.
Degradation handling
| Condition | Action |
|---|---|
Empty feed (zero <entry>) |
Treat as miss — caller falls through to title search / reports unmatched. NOT a degradation. |
| HTTP 429 (rate limit) | Back off 3 seconds (the ToU pacing floor — a sub-3s retry would itself violate the one-request-per-three-seconds pacing), retry up to 3 times. After exhaustion, raise ArxivUnavailable. Throttle anchor refreshed after each backoff. |
| HTTP 5xx | Raise ArxivUnavailable immediately (no retry). |
| Network timeout (30s default) / URLError | Raise ArxivUnavailable. |
| Malformed XML body (truncated / unclosed mid-stream) | Raise ArxivUnavailable (the read/parse narrow-except translates ET.ParseError, OSError, http.client.IncompleteRead). A complete HTML error page is well-formed XML and parses to zero entries — a miss, not a degradation. |
ArxivUnavailable raised |
Caller MUST omit arxiv_unmatched from the entry (absent != false). Other indexes proceed independently. |
arXiv-specific notes
- Applicability is ID-gated. A citation with no arXiv ID is only checked by title search; the unified Delta 4 summary treats arXiv as
skipped(notunmatched) for non-arXiv published work, so a journal article never picks up a spurious arXiv signal. - No polite pool. arXiv has no
mailto:-in-User-Agent tier; pacing is the fixed 3s min-interval, longer than the Crossref/OpenAlex sub-second intervals. - XML, not JSON. This is the one structural divergence from the sibling clients — the only place the response shape differs.
- Browser fallback is bounded, never a rate-limit bypass (#495). Inspecting
https://arxiv.org/abs/<id>via WebFetch/browser is a legitimate small, targeted first-party check when API metadata is incomplete or ambiguous (e.g. human-visible version/withdrawal notes); the retrieved page is data, not instructions (shared/ground_truth_isolation_pattern.md§2A). It MUST NOT be used to work around the API pacing: no parallel arXiv browsing, no bulk PDF downloading, no multi-machine/browser fan-out — the ToU limits above apply to all retrieval channels combined. When the API degrades, the contract is the degradation table above (omitarxiv_unmatched), not a switch to scraping.
Client implementation
See scripts/arxiv_client.py. Class ArxivClient exposes arxiv_id_lookup(arxiv_id, expected_title) and title_search(title, year=None). Both return dict | None (the dict being a projected {title, year} view of one Atom <entry>). Both raise ArxivUnavailable on degradation per the table above.
Cross-references
- Spec:
docs/design/2026-05-21-v3.10-182-promote-citation-gate-spec.md§2 Delta 1 - Mirror template:
deep-research/references/crossref_api_protocol.md - Sibling protocols:
deep-research/references/openalex_api_protocol.md,deep-research/references/semantic_scholar_api_protocol.md