Retrieval Ranking

Corroboration-aware retrieval ranking boosts observations that have been confirmed by multiple independent agents. When two or more agents reach the same conclusion independently, that observation is more likely to be accurate and useful - the ranking layer captures this signal with a logarithmic boost designed to run before feedback weighting.

The boost formula

boostedScore = baseScore + log₂(1 + corroborationScore) × 0.1

Observations not in any cluster receive no boost (+0.0). The boost is deliberately small relative to relevance scores - it is a secondary signal that tips the ordering between candidates of similar relevance, not a dominant reranker.

How clusters are built

The aggregation job (aggregation-job.ts) runs a near-duplicate clustering pass using SimHash (simhash.ts) followed by a corroboration pass (corroboration.ts). The clustering algorithm:

1. Computes a 64-bit SimHash for each observation's content.
2. Groups observations whose SimHash Hamming distance is ≤ 3 bits into candidate clusters.
3. For each cluster, selects a canonical member (the highest-weight observation) and labels the rest as corroborating.
4. Computes corroborationScore = count(corroborating members).

The aggregation job also runs a contradiction pass (contradictions.ts). The Drift Alerts analytics view runs the same detectContradictions pass on demand - it does not depend on the aggregation job.

rankWithCorroboration

Clusters are computed in-memory by the aggregation job and passed directly - there is no separate cluster table. The caller runs clusterNearDuplicates on the observation set, then passes the resulting array to rankWithCorroboration:

import { rankWithCorroboration } from '@/lib/memory/retrieval-ranking'
import { clusterNearDuplicates } from '@/lib/memory/corroboration'

// 1. Retrieve observations for the scope
const scored: ScoredObservation[] = await store.retrieve(scope, { limit: 10_000 })
const observations = scored.map(({ score: _s, ...obs }) => obs)

// 2. Build clusters in-memory (SimHash, Hamming distance ≤ 3)
const clusters: ObservationCluster[] = clusterNearDuplicates(observations, /* threshold= */ 3)

// 3. Apply corroboration boosts
const reranked: ScoredObservation[] = rankWithCorroboration(scored, clusters)
// reranked is sorted descending by boosted score
// the original `scored` array is NOT mutated

In the intended integration, the aggregation job (aggregation-job.ts) calls clusterNearDuplicates and returns clusters as part of its AggregationResult - those clusters can be cached in-process for the duration of the aggregation interval before being passed to rankWithCorroboration. Wiring this into buildSessionMemoryBlock is the open piece: nothing calls rankWithCorroboration in the deployed platform yet.

ObservationCluster shape

interface ObservationCluster {
  canonical: ScoredObservation    // the highest-weight member
  corroborating: ScoredObservation[]
  corroborationScore: number      // count of corroborating members
}

Ordering pipeline summary

The live context-injection ordering today is:

pgvector cosine score
        ↓
  × feedbackWeight (retrieval-feedback.ts)
        ↓
  token budget trim (formatObservationsBlock)

The intended pipeline inserts the corroboration boost between the raw store scores and feedback weighting:

pgvector cosine score
        ↓
  + corroboration boost (retrieval-ranking.ts)   ← not wired yet
        ↓
  × feedbackWeight (retrieval-feedback.ts)
        ↓
  token budget trim (formatObservationsBlock)

This layered design keeps each ranking concern in a separate module and makes each step independently testable.

SimHash near-duplicate detection

The SimHash implementation (simhash.ts) produces 64-bit fingerprints for string content:

• Tokenises on whitespace, strips punctuation.
• Hashes each token to a 64-bit integer.
• Combines using the bit-sign voting algorithm (SimHash).
• Two observations with Hamming distance ≤ 3 are treated as near-duplicates.

The 3-bit threshold was chosen empirically to catch paraphrases and minor rewording while not merging substantively different observations.

Background aggregation job

The aggregation job (aggregation-job.ts) runs near-duplicate clustering and contradiction detection across all active observations for an org/project scope. It operates entirely in-memory - clusters are computed from a full observation scan and returned in the AggregationResult; there is no dedicated cluster table. The caller is responsible for caching or forwarding the cluster results.

The module ships a scheduleAggregation interval wrapper (default 60 seconds, first pass immediate) for running it periodically - like the ranking function, nothing schedules it in the deployed platform yet. Each pass:

1. Retrieves all observations for the scope (up to 10,000 rows) with no text filter.
2. Calls clusterNearDuplicates(observations, threshold=3) to produce ObservationCluster[].
3. Calls detectContradictions(observations, threshold, oppositionPairs) to identify conflicting claims.
4. Returns { clusters, contradictions, corroboratedCount, elapsedMs }.

The Drift Alerts view surfaces the same contradiction signal, computed on demand by its own API route.

Related pages

• Feedback Retrieval - feedback weighting applied after corroboration boost
• Context Injection - the end-to-end injection pipeline
• Drift Alerts - contradiction detection from the same aggregation job
• Memory Diagnostics - misleading observation detection and downweighting