/hindsight-docs

Recall: How Hindsight Retrieves Memories

When you call recall(), Hindsight uses multiple search strategies in parallel to find the most relevant memories, regardless of how you phrase your query.

Figure: Multi-Strategy Retrieval (TEMPR). An animated diagram on the docs site; its narration, step by step:

• recall()
  1. recall() gets a query. Nothing is decided yet about which kind of search fits it best, so every arm that applies runs.
  2. Each arm searches its own index: meaning (vectors), exact words (BM25), the entity graph, and time. “March 2026” becomes a date range; a query with no date skips the time arm.
  3. All four point into the same memories. Each arm returns its own ranked list, and facts and observations compete in every one. The mark shows how many arms found each.
  4. RRF fusion merges the lists by rank, not raw score: a memory found near the top by several arms beats one found by a single arm.
  5. The top candidates (up to 300) go to a cross-encoder, which reads the query and each memory together and scores how well they match.
  6. Small multiplicative boosts nudge the score: recent memories, memories inside the asked time range, and observations backed by more evidence.
  7. Results are packed best-first until max_tokens is used up. Only the memory text counts toward the budget.
  8. The agent gets a short, ranked list it can put straight into its prompt.

The Challenge of Memory Recall

Different queries need different search approaches:

• "Alice works at Google" → needs exact name matching
• "Where does Alice work?" → needs semantic understanding
• "What did Alice do last spring?" → needs temporal reasoning
• "Why did Alice leave?" → needs causal relationship tracing

No single search method handles all these well. Hindsight runs four complementary strategies in parallel and fuses their results.

Four Search Strategies

Semantic Search

What it does: Understands the meaning behind words, not just the words themselves.

Best for:

• Conceptual matches: "Alice's job" → "Alice works as a software engineer"
• Paraphrasing: "Bob's expertise" → "Bob specializes in machine learning"
• Synonyms: "meeting" matches "conference", "discussion", "gathering"

Why it matters: You can ask questions naturally without matching exact keywords.

Keyword Search

What it does: Finds exact terms and names, even when they're spelled uniquely.

Best for:

• Proper nouns: "Google", "Alice Chen", "MIT"
• Technical terms: "PostgreSQL", "HNSW", "TensorFlow"
• Unique identifiers: URLs, product names, specific phrases

Why it matters: Ensures you never miss results that mention specific names or terms, even if they're semantically distant from your query.

Backends: Hindsight ships five pluggable BM25 backends, selected via HINDSIGHT_API_TEXT_SEARCH_EXTENSION:

If you need true BM25 ranking on a horizontally scaled Postgres (Citus) cluster, pg_search is the only option. See the pg_search docker-compose example.

Graph Traversal

What it does: Follows connections between entities to find indirectly related information.

Best for:

• Indirect relationships: "What does Alice do?" → Alice → Google → Google's products
• Entity exploration: "Bob's colleagues" → Bob → co-workers → shared projects
• Multi-hop reasoning: "Alice's team's achievements"

Why it matters: Retrieves facts that aren't semantically or lexically similar but are structurally connected through the knowledge graph.

Example: Even if Alice and her manager are never mentioned together, graph traversal can find the manager through shared projects or team relationships.

Temporal Search

What it does: Understands time expressions and filters by when events occurred.

Best for:

• Historical queries: "What did Alice do in 2023?"
• Time ranges: "What happened last spring?"
• Relative time: "What did Bob work on last year?"
• Before/after: "What happened before Alice joined Google?"

How it works: When the query contains a time reference, Hindsight parses it into a date window, then retrieves the memories whose time overlaps that window. Within the window it selects candidates by semantic relevance to the query — not by recency — so the most relevant in-window memory is never dropped just because other memories happen to be more recent. It then spreads the selection across the window's range: the window is divided into time-buckets and the strongest match from each populated bucket is taken first, so a "what happened in 2023?" query surfaces memories from across the whole year rather than clustering on whichever stretch is densest. Time is also used as a scoring signal — memories closer to the center of the window get a small boost (see Temporal proximity signal).

Why it matters: Enables precise historical queries that stay relevant and representative of the whole period. It also stays fast and meaningful on memory banks whose timestamps are densely clustered (for example when a large batch is ingested with one date): because selection is relevance-first, a time window that happens to match most of the bank still returns the best matches rather than an arbitrary slice.

Result Fusion

After the four strategies run, results are fused together:

• Memories appearing in multiple strategies rank higher (consensus)
• Rank matters more than score (robust across different scoring systems)
• Final results are re-ranked using a neural model that considers query-memory interaction

Why fusion matters: A fact that's both semantically similar AND mentions the right entity will rank higher than one that's only semantically similar.

Why Multiple Strategies?

Consider the query: "What did Alice say about Python last spring?"

• Semantic finds facts about Alice's views on programming
• Keyword ensures "Python" is actually mentioned
• Graph connects Alice → programming languages → related entities
• Temporal filters to "last spring" timeframe

The fusion of all four gives you exactly what you're looking for, even though no single strategy would suffice.

Token Budget Management

Hindsight is built for AI agents, not humans. Traditional search systems return "top-k" results, but agents don't think in terms of result counts—they think in tokens. An agent's context window is measured in tokens, and that's exactly how Hindsight measures results.

How it works:

• Top-ranked memories selected first
• Stops when token budget is exhausted
• You specify context budget, Hindsight fills it with the most relevant memories

Parameters you control:

• max_tokens: How much memory content to return (default: 4096 tokens)
• budget: Search depth level (low, mid, high)
• types: Filter by world, experience, observation, or all
• tags: Filter memories by visibility tags
• tags_match: How to match tags (see Recall API for all options)

Expanding Context: Chunks

Memories are distilled facts—concise but sometimes missing nuance. When your agent needs deeper context, you can optionally retrieve the source material:

Chunks return the raw text that generated each memory—useful when the distilled fact loses important nuance:

Memory: "Alice prefers Python over JavaScript"
Chunk:  "Alice mentioned she prefers Python over JavaScript, mainly because
         of its data science ecosystem, though she admits JS is better for
         frontend work and she's been learning TypeScript lately."

Use include_chunks=True with max_chunk_tokens to control the token budget for chunks. This is useful when generating responses that need verbatim quotes or when context matters (e.g., "What exactly did Alice say about the project?").

Tuning Recall: Quality vs Latency

Different use cases require different trade-offs between recall quality and response speed. Two parameters control this:

Budget: Search Depth

Controls how thoroughly Hindsight explores the memory bank—affecting graph traversal depth, candidate pool size, and cross-encoder re-ranking:

Example: "What did Alice's manager's team work on?" benefits from high budget to traverse multiple hops (Alice → manager → team → projects) and evaluate more candidates.

Max Tokens: Context Window Size

Controls how much memory content to return:

Example: "Summarize everything about Alice" benefits from higher max_tokens to include more facts.

Two Independent Dimensions

Budget and max_tokens control different aspects of recall:

They're independent. Common combinations:

Recommended Configurations

Scoring & Ranking Deep Dive

This section explains exactly how Hindsight turns raw retrieval results into a final ranked list. The pipeline has three stages: RRF fusion, cross-encoder reranking, and combined scoring.

Stage 1: Reciprocal Rank Fusion (RRF)

After all strategies run in parallel, their results are merged using Reciprocal Rank Fusion. RRF combines ranked lists by rewarding items that appear highly ranked across multiple strategies, without relying on raw scores (which aren't comparable across different retrieval methods).

Formula:

score(d) = Σ  1 / (k + rank_i(d))
           i

Where:

• k = 60 (smoothing constant — prevents top-ranked items from dominating)
• rank_i(d) = position of document d in strategy i (1-indexed)
• The sum runs over all strategies where d appears

Within RRF, all four strategies are weighted equally — fusion uses rank position, not the source, so no strategy gets an implicit multiplier. You can, however, deliberately bias a source with HINDSIGHT_API_RECALL_STRATEGY_BOOSTS: that boost is applied at a separate stage — before the reranking pre-filter cap, and again after a cross-encoder rerank — not inside the RRF fusion above. The pre-cap step runs only when the merged pool exceeds the cap, and it does not rewrite RRF scores. A passthrough reranker (mode rrf, an rrf provider, or a failover chain that has degraded to rrf) skips the post-rerank nudge: within the cap the setting then does nothing, and above the cap it only changes who is kept.

Why RRF over raw score merging? Each retrieval strategy produces scores on a different scale (cosine similarity, BM25 tf-idf, graph activation). These scores aren't comparable — a BM25 score of 12.5 and a cosine similarity of 0.85 don't mean the same thing. RRF sidesteps this by using only rank positions, making it robust across any scoring system without requiring calibration.

Example: A memory ranked #1 in semantic and #5 in BM25:

RRF score = 1/(60+1) + 1/(60+5) = 0.0164 + 0.0154 = 0.0318

A memory ranked #1 in semantic only:

RRF score = 1/(60+1) = 0.0164

The first memory ranks higher because it has consensus across strategies.

Stage 2: Cross-Encoder Reranking

RRF gives a good initial ranking, but it's based on positions, not on deep query-document understanding. The cross-encoder evaluates each candidate against the query as a pair, producing a relevance score.

Pre-filtering: Before reranking, candidates are trimmed to the top 300 (by RRF score) to limit computational cost. This is configurable via HINDSIGHT_API_RERANKER_MAX_CANDIDATES. If HINDSIGHT_API_RECALL_STRATEGY_BOOSTS is set, the boost is applied before this cut, so candidates from a favoured source are more likely to survive it. The boost promotes the favoured arm in rank space (its rank is divided by the level's divisor before the RRF contribution is computed) rather than scaling its score, so it reaches deeper into that arm without evicting the top-ranked hits of the others — including on banks whose merged pool is many times the cap. When trace: true is requested, the rerank_prefilter phase reports how many candidates were kept and dropped, the cap in force, the active boosts, and the per-arm composition of the survivors.

Why rerank after RRF? RRF is position-based — it knows a memory ranked well across strategies, but it never actually reads the query and the memory together. The cross-encoder does: it takes the query and each candidate as a pair and produces a relevance score based on their full interaction. This catches nuances that position-based fusion misses, like a memory that ranked #1 in keyword search because it matched a common term but is actually irrelevant to the query's intent.

Score normalization: Cross-encoders output raw logits (which can be negative). Scores that already fall within [0, 1] — as returned by calibrated external API rerankers (e.g. Cohere, SiliconFlow, ZeroEntropy, Alibaba, Jina) — are passed through unchanged to preserve their absolute confidence. Raw logits outside [0, 1] are normalized to [0, 1] using the sigmoid function:

CE_normalized = 1 / (1 + e^(-raw_logit))

Batch processing: Candidates are scored in batches — 32 pairs for the local reranker, 128 pairs for TEI.

:::tip No cross-encoder? When running without a cross-encoder (e.g., slim image with no external reranker), the system falls back to RRF-derived scores: candidates are assigned synthetic scores spread across [0.1, 1.0] based on their RRF rank, so the combined scoring boosts below still work meaningfully. HINDSIGHT_API_RECALL_STRATEGY_BOOSTS does not add its post-rerank nudge on this path. If the merged pool is within the reranker cap, the setting changes nothing; if the pool is larger, it only changes which candidates are kept, and their order still follows these RRF-derived scores plus recency, temporal and proof. :::

Stage 3: Combined Scoring (Boosts)

The normalized cross-encoder score is adjusted by three multiplicative boosts that incorporate signals the cross-encoder can't see: recency, temporal proximity, and evidence strength.

Why multiplicative instead of additive? Additive boosts (e.g., CE + 0.1 × recency) would give the same absolute bonus to every candidate regardless of relevance. A barely-relevant memory could leapfrog a highly-relevant one just by being recent. Multiplicative boosts keep adjustments proportional to the base relevance score — a +10% nudge on a high-relevance memory is a bigger absolute change than +10% on a low-relevance one. This ensures secondary signals never overpower the primary relevance judgment.

Formula:

final_score = CE_normalized × recency_boost × temporal_boost × proof_count_boost

Each boost is centered at 1.0 (neutral) and controlled by an alpha that caps how much it can swing:

boost = 1 + α × (signal - 0.5)

Recency signal

Linear decay over 365 days from the memory's occurrence date:

recency = clamp(1.0 - days_ago / 365, 0.1, 1.0)

A memory from the query timestamp has recency 1.0 (+10% boost). A memory from 6 months before the query timestamp has recency ~0.5 (neutral). A memory more than a year before the query timestamp has recency 0.1 (-8% penalty). If no query_timestamp is provided, the server's current time is used. Memories without dates get 0.5 (neutral — no boost or penalty).

Temporal proximity signal

Only active when the query contains a time reference (e.g., "last spring", "in 2023"). Measures how close a memory's date is to the center of the queried time window:

temporal_proximity = 1.0 - min(days_from_center / (window_days / 2), 1.0)

A memory at the center of the window gets 1.0 (+10% boost). A memory at the edge gets 0.0 (-10% penalty). For non-temporal queries, all memories get 0.5 (neutral).

Proof count signal

For observation-type memories, rewards those backed by more evidence using a logarithmic curve:

proof_norm = clamp(0.5 + ln(proof_count) / 10, 0.0, 1.0)

Maximum combined range

With all boosts at their extremes:

• Best case: ×1.10 × 1.10 × 1.05 ≈ +27%
• Worst case: ×0.90 × 0.90 × 0.95 ≈ -23%

The boosts are intentionally conservative — they nudge the ranking without overriding cross-encoder relevance.

Stage 4: Token Truncation

After scoring, results are sorted by final_score and selected top-down until the max_tokens budget is exhausted. Only the memory text counts toward the budget — metadata is free. A result too long for what is left of the budget is skipped and the packing continues with the next one, so a single long fact does not cost you the shorter ones ranked behind it. If nothing fits at all, the top-ranked result is still returned whole, so a query that matched something is never answered with an empty list.

How Budget Maps to Pipeline Parameters

The budget parameter (low/mid/high) controls search depth — how many candidates each strategy considers. Each level maps to a recall budget number that flows through every pipeline stage:

This recall budget flows through the pipeline as follows:

Reranking pre-filter (300 candidates) is independent of budget — it's a separate knob (HINDSIGHT_API_RERANKER_MAX_CANDIDATES).

:::info Adaptive budgeting An alternative budget mode scales the recall budget with max_tokens instead of using fixed values:

recall_budget = clamp(max_tokens × ratio, min, max)

The result is clamped to a floor of 20 (HINDSIGHT_API_RECALL_BUDGET_MIN) and a ceiling of 2000 (HINDSIGHT_API_RECALL_BUDGET_MAX).

Enable with HINDSIGHT_API_RECALL_BUDGET_FUNCTION=adaptive. :::

Graph Scoring Detail

The graph traversal (link expansion) combines three independent signals additively for each candidate:

graph_score = entity_score + semantic_score + causal_score   ∈ [0, 3]

The additive combination rewards convergent evidence — a memory connected to the query through multiple signal types ranks higher than one connected through a single strong signal.

Why tanh for entity scores? Raw shared-entity count is unbounded — a high-fanout entity like "user" could produce counts of 50+, drowning out the other two signals. tanh(count × 0.5) saturates naturally: the first few shared entities matter a lot (1→0.46, 2→0.76, 3→0.91), but additional ones contribute diminishing returns, keeping the entity signal in [0, 1] alongside semantic and causal scores.

Why additive instead of multiplicative here? Unlike the combined scoring boosts, graph signals are independent evidence channels, not adjustments to a base score. A memory might be connected only through causal links (no shared entities, no semantic similarity) — multiplicative combination would zero it out. Additive scoring lets each signal contribute independently, and the outer RRF fusion handles ranking across strategies.

Entity signal example: A memory sharing 1 entity with the query scores tanh(0.5) ≈ 0.46. Two shared entities score tanh(1.0) ≈ 0.76. Three or more saturate near 0.91+.

Next Steps

• Retain — How memories are stored with rich context
• Reflect — How disposition influences reasoning
• Recall API — Code examples, parameters, and tag filtering