How It Works

Pelorus keeps the normal RAG path of retrieving source chunks, then adds a demand-shaped layer above it. As users and agents ask real questions, repeated needs can become source-grounded Extracts that are easier to inspect, reuse, and improve.

What an Extract contains

An Extract is a self-contained, source-grounded data packet built to accommodate an associated query, or a cluster of related queries for a Cluster Extract. It is additive to chunks: shaped around the user need rather than the original document boundary, and it never replaces the corpus. The cited chunks are always there — they are the grounding, and every other part of the Extract is read against them. This is the Extract as served: the five fields below are the specification, and anything an implementation keeps behind them (the extractor's working notes, how the cites were established, whether a summary is stale) is not part of it.

Three of the fields are different forms of the same grounded material, and a consumer picks the one it wants: the chunks themselves, a Summary that is equal to them, or Content that is targeted to the query. Because all three share one cite list, a consumer holding chunk hits from its own retrieval can dedupe by id against whichever form it took. The remaining two fields carry different kinds of trust:

• Summary, Content and Context must be true to the corpus. They may be denser than the chunks, Content may leave things out, and Context may draw on the wider document, but every statement in them must be an accurate representation of the corpus — validated and trustworthy.
• Curation notes do not need to be grounded in the corpus. Curated by a person or by AI, outside Pelorus's grounding guarantee, they can carry supplementary information, links, statements or guidance — anything that addresses the live stream of incoming questions, including questions the corpus does not answer.

A gap Extract is the same shape with no cited chunks, no Summary and no Content: a verified statement that the corpus does not answer the question, served alone and labeled, with Context and Curation notes still able to say where to go instead. It is covered under Gaps and partial coverage.

Who validates what

Keeping the cited chunks, Summary, Content and Context grounded and true to the corpus is Pelorus's responsibility, including validating its own summaries. Curation notes are the curator's responsibility and are taken as valid information. An evaluation harness such as RAGauge trusts whatever a RAG connector returns as valid for the corpus: it does not check that summaries, context or curated content are grounded. It measures whether what came back addresses the question, and whether a generated answer is grounded in what the model was given.

That makes evaluation a feedback loop. A regenerated Extract is new content with new grades, so you can compare Extract versions, or ways of generating summaries, the way you would compare two settings of a retrieval pipeline. Content an evaluation reports as disagreeing with a test's expected answer is a signal to investigate, not a verdict that it is ungrounded.

What gets matched, what gets returned

Pelorus decouples the thing being matched from the data returned. An incoming query is matched against a stored query or a cluster of related queries, because query-to-query similarity is usually a cleaner signal than query-to-chunk similarity. The matched target then returns its associated Extract: a custom data packet shaped to answer that query or cluster while accurately representing the corpus.

Matching is not a single winner-take-all path. If a query matches overlapping clusters, Pelorus can return both Cluster Extracts. If a specific Query Extract or document chunks are close enough under the retrieval settings, those can surface too. Cluster coverage is meant to give unseen queries useful context, while still allowing more specific Extracts and chunks to handle nuance.

Query Extracts

A Query Extract couples one observed query with one Extract. The query is the matching surface; the Extract is the information served when an incoming query matches that stored query closely. This makes repeated queries concrete objects you can inspect and curate.

Cluster Extracts

A Cluster Extract couples a cluster of related queries with one Extract. Cluster matching is handled separately from query-vector matching: a cluster can use a centroid with a proximity threshold, or a trained classification model for harder shapes. The matched cluster returns its associated Extract.

Each cluster carries its own threshold. When a cluster is created or its membership edited, that threshold is set to the similarity of its loosest member, so every member query sits inside the region its own cluster serves by; a curator can then tune it. Overlapping clusters serve together, each reporting its own centroid similarity, and a Query Extract that also matches serves alongside them.

What happens when a query runs

A runtime query returns from what Pelorus already knows. It can match one or more Cluster Extracts, match a close Query Extract, include closely related document chunks, fall back to chunks when no Extract fits, or return a miss. Normal query handling does not build a new Extract before responding.

The adaptive part happens after the response. When a useful chunk fallback exposes unmet demand, Pelorus can queue asynchronous Extract creation. The next similar query can then reuse a curated Extract instead of relying only on document chunks.

Two rules keep that loop honest. Demand comes only from a query that matched nothing curated. A query served by a Query Extract or Cluster Extract records a hit and nothing more, even when a document chunk happens to score higher than the chunks the Extract cites: query-to-chunk similarity never overrules a query-to-query match. And Pelorus never re-extracts on its own. Regenerating an Extract, or replacing it with a new one, is a curator's decision taken in the console, so a good Extract cannot be quietly rewritten by the system that served it. Admin and test queries are kept out of all of this: unless you mark one as live traffic, it records no hits and no demand on any tier.

Gaps and partial coverage

Not every observed need can be answered accurately from the corpus. Pelorus can preserve that outcome as missing or partial coverage, so the same gap is visible instead of repeatedly disappearing into retrieval misses. As more demand arrives, you can tune thresholds, refine Extracts, split or merge clusters, or decide the corpus itself needs new source material.

Questions your corpus cannot answer

A plain vector store has no way to say “nothing here answers that.” Top-K is a structural promise: ask it something your documents never covered and it still returns K passages, ranked, with scores attached. The absence of an answer looks exactly like the presence of a weak one, and the difference only surfaces downstream, in whatever your LLM decides to do with material that does not answer the question. Pelorus treats that case as an outcome worth keeping.

• The gap is recorded once. When a query arrives that nothing in the corpus answers, Pelorus records the query together with that outcome. The next time the same question is asked it matches the recorded outcome instead of raising fresh extraction demand — so a question your corpus cannot answer stops costing you an extraction every time somebody asks it.
• It stays visible as a gap. Not serving is not deleting. The record keeps existing and keeps matching, which is what puts the gap on the Query Map and into the supply-and-demand analytics rather than letting it disappear into a stream of near-miss retrievals.
• It is never served as a ranked result. A recorded absence can never be sorted, hold a rank against content, or spend a slot in the result budget alongside other candidates. That gate matters more than it sounds: a recorded “nothing answers this” matches its own query almost perfectly, so without it the record would take the top slot and push the real source passages out.
• A verified gap is served as a disclosure, alone. When every curated candidate is withheld and one of them is a verified gap — the corpus was searched and confirmed to have no answer — the response serves that absence as a single labeled item, and nothing else. No chunks ride alongside it: a disclosure of absence sitting next to five plausible-looking answerless chunks would undo the point of disclosing it. The label travels through both runtimes (is_gap on the served item, pelorus_gap in analytics row metadata) so a consumer can say “I don't know” because it was told, grounded in a served fact about the response, rather than because it declined to trust passages it was handed.
• Curated content can travel with a gap. The SME-curated section of a gap Extract can still help the person asking — where to go instead, what to do next — without pretending the corpus answers. An evaluation such as RAGauge scores the corpus outcome (no corpus passage claimed is correct) and notes the curated content beside it, never counting it as a corpus answer.

Designed: not yet built

Freshness for a recorded gap. A recorded “nothing answers this” is a claim about the corpus, and unlike an ordinary Extract it cites no source passages — so nothing ties it to the material it is a claim about. Add a document and the claim can quietly become wrong. Re-validating these records when the corpus changes is designed and not built; it belongs with cite re-resolution on a re-chunk, because the claim needs re-checking rather than discarding. Its other job — keeping the same question from re-queueing extraction work forever — stays correct either way. Until the re-validation piece exists, a recorded gap is refreshed the same way any other Extract is: a curator regenerates it from the console after the corpus changes. Pelorus does not re-extract a served record on its own, gap or otherwise.

Summary

• Cluster matching can serve broad areas of related demand by matching a query against a cluster boundary and returning the associated Cluster Extract.
• Query matching handles more specific needs by comparing the incoming query to vectorized queries attached to Query Extracts.
• Chunk fallback remains available when no Extract is strong enough, so retrieval can still return source passages from the underlying corpus.
• Thresholds are configurable. Cluster Extracts, Query Extracts, and chunks surface based on match settings instead of fixed proximity labels.
• Overlap is expected. A query can genuinely fit more than one cluster, and a narrower query or chunk match can still surface when it provides better coverage.
• Traffic becomes visible. Repeated user needs become durable queries that can be inspected, clustered, curated, and used to create better Extracts.
• A question the corpus cannot answer is an outcome, not a miss. The gap is recorded once, stays visible in analytics, and is never ranked against content — a verified gap is served alone as a labeled disclosure, and an unverified one falls back to document chunks.
• Served hits raise no demand, and nothing is re-extracted automatically. Demand is recorded only when a query matches nothing curated; regenerating an Extract is a curator's action.
• Coverage evolves as more traffic arrives. You can grow, split, overlap, and refine queries and clusters as demand changes or drifts in real time.