Ideas: where do the documents discuss something?

Some questions are about an idea, not a thing: where do we discuss pricing risk?, which documents talk about data retention? No field holds the answer and no entity is the answer — the answer is the parts of documents that discuss it. Search answers these with the same guarantees it gives a question about things: every match, the sentence behind each, the near misses, and whether that is everything.

Every field below is in the generated verb reference. The ordinary side of a search is search.

How it works: label once, group per question

1. At load, every sentence is labelled. When a document has been read, a model reads each of its parts — the sections, clauses and paragraphs the segmenter cuts (a short sentence standing alone is a part too; a heading opens the part after it) — and gives every sentence that states something one to three short topics: a phrase of a few words naming what the sentence is about, never the sentence restated, each with one broader term (the general category it falls under). Several parts go to the model per call, and several documents are labelled at once, in the background. A topic is kept only if it is short (at most five words), uses at least one word of its part, and does not restate its sentence; its receipt is that sentence's span. Coverage is checked per sentence. A sentence left without a topic (none given, or every one refused) is asked once more, alone, with why; one still without a topic is recorded on its part — and counted by every idea question over that document (sentences_not_labelled, below).
2. At question time, labels — not passages — are sorted. The distinct labels of your documents in scope are sorted by the model into means it, possibly and no for the idea. Each label is shown with one example: a sentence it was given for, taken from a document you may read (the first such sentence in document order), so it is judged as your documents use it — "saving changes", given for Click Save to keep your changes., is a button, not data retention. A label carried by more than two parts is judged twice, in two different calls, the second time shown a different sentence when your documents hold one (when every part says the same sentence, it is shown again). It means it only when both judgments say so; when they disagree it is possibly — never no. One verdict files every part carrying its label, so this is where a wrong verdict would cost most. The verdicts are stored under the idea's words (case and spacing aside), each with how many judgments it rests on, and are reused by the next question of that idea, which sorts only the labels that are new since — and judges again a label judged once that has since come to be carried by more than two parts.
3. The answer is a lookup. Matches are the parts filed under a means it label; the parts under a possibly label are listed apart.

The labels are facts like any other: the relation part_label (slots label, broader), one subject per part. limma_facts enumerates them (relation_query: {relation: "part_label", where: [["label", "eq", "price hike"]]}) and limma_region redeems every sentence. A part with a sentence left without a topic carries a withheld coverage gap (reason sentence_not_labelled), so limma_facts never reads that part's labels as all it says.

The model is your deployment's own (LIMMA_PROVIDER; on-prem, your open-weight endpoint). The two steps can be routed to different models with LIMMA_MODEL_ROUTES roles part_labeller and idea_grouper.

Sending one

POST /v1/public/search
{"workspace_id": "ws_1", "idea": "where do we discuss pricing risk?", "detail": "compact"}

An idea may stand alone (every document you can read is in scope), or beside constraints / brief / entity_ids — then only the documents about the entities those name are in scope. It may not stand beside criteria (a criterion is answered over entities, an idea over parts: send two searches).

Sorting labels new to an idea costs model calls and stores verdicts, so it needs a key with write permission. A read-only key sends idea_budget: {"max_calls": 0} and is served the verdicts already stored. idea_budget.max_calls (default 20, 25 labels a call) and max_seconds bound what one call may sort; what is left is counted and sorted by the next call. A label judged twice takes a place in two calls: the first judgments of every label are made first, the second judgments after them; a first judgment the budget left without its second is kept, and the label is counted as not sorted (labels_not_grouped) until a later call makes the second.

What comes back

The compact reply (the MCP tool's default):

{"detail": "compact", "idea": "where do we discuss pricing risk?",
 "matched": {"total": 2, "showing": [1, 2], "next_offset": null},
 "complete": true,
 "matches": [
   {"source_id": "9c1…", "document": "memo.txt", "location": [0, 66],
    "quote": "We reserve the right to raise our fees once a year.",
    "quote_at": [6, 57], "labels": ["raise our fees (pricing)"]}],
 "possibly": {"total": 1, "showing": [1, 1], "next_offset": null, "matches": [
   {"source_id": "9c1…", "document": "memo.txt", "location": [68, 96],
    "quote": "The price list is attached.", "quote_at": [68, 95],
    "labels": ["price list (pricing)"]}]},
 "counts": {"documents": 2, "parts": 5, "labelled": 5},
 "more": {"receipt": "…", "labels": "…", "full": "…"}}

• matches — each part: its document, where it is (location, character offsets), the sentence behind it (quote; quote_at is its exact span — limma_region returns it byte for byte) and the labels it is filed under.
• possibly — the parts that may discuss the idea. Read them; never drop them. Both lists are paged by the same offset, and every count is whole.
• detail: "full" returns the idea block: each part's id, every label with its verdict and sentence span, and the grouping's own counts (labels, reused, judged_now, judged_twice — of those, the labels judged for agreement — and disagreed — those whose two judgments did not agree — not_grouped, verdicts), and spent when the call sorted labels.

What "complete" means here

complete: true says: every part of every document in scope was labelled, every sentence in them got a topic, and every label in scope was sorted for this idea — so every part filed under a label that means it is in matches. When that is not so, complete is false and incomplete_because counts what is missing, in parts unless the reason names documents or sentences:

reason	what it counts
not_labelled:model_unavailable	parts no model could label (none configured, or the spend switch on)
not_labelled:being_labelled	parts queued or being labelled now — ask again
not_labelled:failed	parts whose labelling call failed; labelled again later
sentences_not_labelled	sentences of labelled parts the model gave no usable topic, even when asked again — they can be filed under no idea, so their part's absence from matches proves nothing
documents_being_read	documents whose ingest has not produced text yet
documents_unreadable	documents with no text to read
documents_not_labelled	documents nothing has labelled yet — loaded before labelling existed, or registered by connect with their read deferred; a write key's question hands them to the labeller, a bounded number at a time
labels_not_grouped	parts whose labels are not all sorted for this idea (budget, time, or a read-only call)
entity_scope_incomplete	the entities your conditions matched are not a complete set

The same state is on each document: the ingest reply carries labelling — {state, parts, labelled, not_labelled: {cause: parts}, sentences_not_labelled} — so a document with no model to label it says so the moment it is loaded.

Per-user access. A document the user may not open is never counted, named or quoted, and neither its labels nor any sentence of it is ever shown to the grouping model on that user's call: the example under a label is read under the same restriction as every other read, and how many parts carry a label — which decides whether it is judged twice — is counted over the user's own documents only.

What a stored verdict can and cannot reveal. The verdicts of an idea are shared by every user of the workspace: a label sorted on one user's call is not sorted again on another's. A stored verdict holds no text — the idea's normalised words, the label's words (which the second user's own documents hold, or it is never looked up for them), one of means it / possibly / no, and the number of judgments it rests on. It may have been judged on a sentence in a document the second user cannot open, so it carries at most one of three words about how such a sentence read for that idea. It never carries that sentence, which document holds it, whether one exists, or how many parts carry the label anywhere but in the user's own documents.

The one silent risk, and how it is measured

A sentence given no topic is not silent: it is counted (sentences_not_labelled). A sentence can be labelled and still be missed: if the labeller gave it a topic that does not fit the idea, its labels are sorted no and nothing in the reply can tell. complete is true over the labelled sentences; it cannot vouch for a topic that does not fit.

That risk is measured, not assumed. benchmarks/scale/idea_worlds.py generates contracts, policies, meeting notes and support articles with ideas planted in many phrasings (and near misses that share an idea's words, not its meaning), with gold, and reports per idea the planted sentences found against those expected, the parts matched that hold none, and how many landed in possibly — beside the trivial optima (match every part; match the idea's own words) — and, for each sentence missed, the topics it was given with their verdicts, or why it has none. Its scripted model checks the harness itself: when one phrasing is left without a topic, the harness names exactly those sentences and the reply says it is not complete, counting them; when one phrasing is given a topic that does not fit, the harness names exactly those sentences while the reply still says complete — the risk above. Every run is scored twice: with the gold as planted (a sentence counts for the one idea it was planted for), and under a multi-idea reading, where a planted sentence that discusses two ideas — The contract terminates automatically if the invoice remains unpaid. — counts for both. benchmarks/scale/ idea_live_run.py runs the same world with your open-weight model; it prints what it would call and spend and refuses until --spend-ok is given, and reports what was spent from the cost ledger (--reuse-labels re-sorts an earlier run's labels, so only the grouping spends).