Operating loop

Core Workflow

The early release focuses on a compact operating loop: set up a dataset, route real queries, process the query/extract queue, then use the resulting insights to create and curate clusters.

1. Setup dataset

Start by identifying the corpus you want Pelorus to observe and represent. A dataset points at that source material and is attached to a Dataset Group, which controls shared settings such as embedding space and storage.

During setup, configure the extraction template (the Extract Config and prompt). The template controls what each Extract should capture, how the fields are used, and how the resulting information packet stays faithful to the corpus while addressing a query.

Setup also covers the core server settings — models, retrieval, and processing. In normal Admin usage, these persisted settings become the source of truth after startup.

Models

Configure embedding models and extract LLMs. Keep public config files pointed at environment variable names instead of raw keys.

Retrieval

Tune extract thresholds, chunk thresholds, max results, and cluster matching behavior after checking real query results.

Processing

Use manual batches while validating prompts. Move to realtime only after demand quality and extraction behavior are acceptable.

llm_configs:
  - name: gemini-flash
    provider: google
    model: gemini-2.0-flash
    api_key_env: GEMINI_API_KEY
    key_source: env

retrieval_config:
  query_extract_threshold: 0.98
  chunks_threshold: 0.45
  extraction_processing_mode: manual_batch

2. Route queries

Enable the dataset when it is ready to serve. Application traffic can then be routed through a connector, such as the Python client, so Pelorus can answer requests and observe the query patterns emerging from real usage.

Routing traffic is what turns the system from static retrieval into an adaptive context layer. Queries can match chunks, Query Extracts, and Cluster Extracts while also creating the signals used to improve future coverage.

3. Process the extract queue (Extract Processing)

As queries reveal repeated needs, Pelorus can queue work to create or update query/extract pairs. Processing the queue turns observed demand into Extracts that are tailored to those queries and true to the corpus.

Queue processing may be manual while you validate the strategy and prompts, or more automated as confidence grows. The important control point is that new Extracts are created from the corpus with an explicit query in mind.

4. View insights and curate clusters

Use the Query Map and related insight views to see where traffic is concentrating, where gaps are appearing, and where similar queries form useful neighborhoods.

From there, create and curate clusters. Cluster Extracts let you serve broader areas of demand, while manual curation lets you refine cluster boundaries, handle overlap, and keep coverage aligned with how user traffic changes over time.

You can also curate Extracts directly. For example, a subject-matter expert can add trusted context, references, or clarifications that should be available across related queries without editing the source corpus and reingesting documents. This is often most useful on a cluster, where the curated information can serve a neighborhood of related queries.