Sync and access — keeping a workspace equal to a source system, per user¶

This page is for the engineers of a platform that embeds Limma and pushes its customers' data in through its own connectors (SharePoint, Drive, S3, a data room, a warehouse), and then calls Limma for many end users with one key. Two things make that correct:

The sync contract — every file carries the source system's own id for it, so a changed file replaces the old one (whose facts stop answering), an unchanged one is a no-op, a removed one is deleted, and a reconcile listing tells the connector what Limma holds.
Per-user access — every document can carry the list of principals who may see it, and a request made with a platform key says which end user it is for. That user's replies are computed over their documents only — counts and completeness included.

Both work over REST and over MCP (MCP over HTTP), with two exceptions named below: a connect source's access list and the workspace's access_required policy are set over REST only.

1. The sync contract¶

Sending a file¶

Send each file with up to three extra fields:

field
`external_id`	the source system's stable id for the file (an item id, a file id, an object key). Compared exactly; at most 1,024 characters, no control characters.
`external_version`	optional: the source system's version of the file (an etag, a modified time). Needs `external_id`.
`access`	optional: who may see the document — see Access lists.

They are accepted on both document doors, which share one implementation:

POST /v1/public/ingest (text, JSON body) — the fields above, as JSON. MCP: limma_ingest(..., external_id=, external_version=, access=).
POST /v1/workspaces/{workspace_id}/documents (PDFs, Office files, images; multipart) — the same three as form fields, access written as a JSON array (access=["user:alice@x.com","group:finance"]). A request carrying external_id carries one file (422 otherwise): send each file of the source system in its own request.

What happens¶

There is never more than one live version of an external_id in a workspace. Each reply carries a sync block (on the upload door, on the file's entry in documents):

`sync.outcome`	when	what Limma did
`created`	the workspace held no document under this id	registered it
`unchanged`	`unchanged_because: same_external_version` — the version you sent is the live one's	nothing was read, whatever bytes you sent (`bytes_differ: true` says they differed: the version is your statement that nothing changed)
`unchanged`	`unchanged_because: same_bytes` — the bytes are the live version's	nothing new is registered. A document whose extraction failed is retried, as any re-send of the same bytes is
`replaced`	the bytes changed	the new version is now the live one, and the old version was erased

A replacement erases the old version through the same provable delete as a document delete: every generation of it, every table the erasure census places, its bytes. Its facts stop answering at once. sync.replaced says what was erased and what is left:

"replaced": {"source_id": "…", "external_version": "etag-1", "erased": true,
             "generations_erased": ["…"], "claims_removed": 12,
             "relations_erased": {"…": 0}, "residue": {}, "complete": true}

residue is read back from the database after the erasure — every table holding the erased document, by name, with the rows still there. {} is the proof.

The old version is erased when the new one is registered, not when the new one has finished extracting: for a few seconds (or, with a queued worker, until it has run) the id answers nothing new rather than two versions at once. A new version whose extraction fails is reported processing_status: failed; send it again to retry.

Two edges:

Identical bytes under two ids (one file copied into two folders) are one document. Replacing or deleting one id keeps the document for the other (erased: false, kept_because); only that id's record and its access grant go.
A legal hold on the live version (or on a generation it superseded) refuses a new version — 409 replaced_version_on_hold, before anything is ingested — because the old version could not be erased and the id would have two live versions.

A field that cannot be read (an empty external_id, a version without an id, a malformed access) is 422 sync_field_invalid, before anything is written.

Deleting a file¶

POST /v1/public/delete_document
{"workspace_id": "…", "external_id": "drive:01ABC"}

Erases the live version through the provable delete and reads back what is left: {external_id, source_id, external_version, deleted, erased, generations_erased, claims_removed, relations_erased, jobs_forgotten, finished_interrupted_erasure, residue, complete}. 404 external_id_not_found for an id the workspace does not hold; 409 legal_hold for a held document (nothing changed); 500 erasure_incomplete names the tables still holding rows — the same call finishes it: the id's record is the last row the delete removes, so a delete a database failure interrupted is found again by the same external_id, and the call that finishes it answers complete: true with finished_interrupted_erasure: true. The document's queued (or finished, or failed) ingest job goes with it, and with the job the copy of the bytes it held (jobs_forgotten). So does its trail: every event of the workspace's log and of the audit trail that named the document keeps its shell (type, time, counts) and loses its content — the file name, a claim's text, a value, the question asked about it (relations_erased.events_redacted, audit_events_redacted) — and residue reads the trail too. MCP: limma_delete_document (admin tier).

A file deleted or replaced while it is still being read is erased the same way, and stays erased: the erasure fences the document before it deletes anything, so the reading can write nothing of it afterwards (its job ends skipped: source_erased). The fence waits for a write of the document already in flight to commit — at most LIMMA_ERASURE_FENCE_WAIT_S (30 s); past that the answer is 503 erasure_busy with Retry-After, and nothing was erased. A request that writes about a document whose erasure has begun is answered 409 document_erased. The fence keeps one row per erased document — its id and when, no content — which the workspace's own delete removes.

The reconcile listing¶

GET /v1/public/reconcile?workspace_id=…&limit=100&after=

Every external id the workspace holds, in id order, one page at a time:

{"documents": [{"external_id": "drive:01ABC", "source_id": "…",
                "external_version": "etag-2", "content_hash": "…",
                "recorded_at": "…", "processing_status": "complete",
                "access": ["group:finance"]}],
 "total": 1240, "next_after": "drive:01ABC", "access_required": false}

Pass next_after as after for the next page; it is null on the last. A connector's nightly pass is a diff against its source system:

for each file in the source system:
    if Limma lists its id at the same version  -> skip
    else                                        -> send it (ingest / upload door)
for each id Limma lists that the source no longer has -> delete_document

MCP: limma_reconcile (admin tier).

Safe retries¶

idempotency_key on the ingest verb is an in-process memo: a retry that reaches the same API process returns the first response. external_id is the durable dedupe: across processes, replicas and restarts, a re-send of the same file is the no-op above.

Who may call these¶

Ingest, the upload door, delete_document and reconcile are writes: a read-only (viewer) key is refused 403, as is a key confined to documents. They are made by the platform as itself: asserting principals on them is refused (below).

2. Per-user access¶

Access lists¶

access is a list of principal strings — whatever names your platform gives users and groups, e.g. user:alice@x.com, group:finance. Each is compared exactly: it must not be empty, begin or end with a space, or hold a comma or a control character (422 otherwise, never trimmed). At most 1,000 per list.

sent	meaning
omitted	the document's list is left as it is. A new version keeps the old version's list (`sync.access_carried_over: true`) — leaving the field out never makes a document visible to everyone
`["user:a", "group:b"]`	these principals may see it — replacing the list this id set before
`[]`	a list that names nobody: no asserted principal sees it

Sent with an unchanged version, the list replaces the old one without reading the document again — a permission change in the source system changes no bytes. sync.access is the document's list now (null: no list).

The list is recorded on the document before its extraction is queued, so no process reads a document — and no principal sees its facts — before its list holds. (An API built without an ingest queue cannot promise that, and refuses an ingest that carries external_id or access; every image Limma ships runs one.)

Records loaded by connect: an access list on the connect request applies to every record of that source — its facts, its earlier generations, and the text documents a document-store pull registers from it. Omitted on a later pull, the previous pull's list carries over. Per-record access is not offered: records that need different lists are different sources (one export, or one table, per audience). The list is set over REST (POST /v1/public/connect); the MCP limma_connect tool does not take it — it rides the default menu, whose every schema each client sends with every request — and a pull over MCP keeps the list the source already has.

An export your connector uploads (POST /v1/public/upload_export, then connect with document_source.export_id — for a deployment whose API host your connector cannot write to: the operator's page) is the same source as a file on the host: its records' access list is the one the connect sends, and a re-sync is tonight's export uploaded and connected again — the line of generations is the collection, never the file or its id. The upload is held for its workspace alone, read only by a connect of that workspace, and deleted by the pull that loads it; an id of another workspace answers 404 as one never minted. Asserting principals on the upload is refused 400, as on every write the platform makes as itself.

The platform key¶

Only a platform key may say which end user a request is for. The operator makes a key a platform key by naming its key id in EPISTEM_PLATFORM_KEYS (comma-separated; key_ + 16 hex digits — the id the audit trail records as caller_key_id, never the key itself). The key keeps its role (use service); being a platform key adds one permission, assert_principals.

A request then asserts the end user's principals in one header:

X-Limma-Principals: user:alice@x.com,group:finance

the request	answer
a platform key, on a read verb	served over that user's documents only
any other key sending the header — or none, in development mode	403 `principals_not_permitted` — never served as if the header were absent
the header on a call the platform makes as itself — a write (ingest, upload, connect, `delete_document`, the access policy, `apply_pack`, …) or the connector's `reconcile` — or on any route that does not narrow to documents	400 `principals_not_applied`
a header that is no list (empty, `a,,b`, a trailing comma), or sent twice	422 `principals_invalid`

The verbs that honour it are the read verbs: search, facts, region, state, ask, plan, precheck, derive, verify, attest, diff, generations, corrections — and every MCP tool over them. Over POST /mcp send the header on the HTTP request; every tool call of that request carries it to its verb, and is refused there exactly as over REST. Over stdio (python -m epistem.mcp.server), set LIMMA_PRINCIPALS.

What a user sees¶

With principals asserted, the documents a request may read are

those whose access list names at least one of the principals, and
those with no list at all — unless the workspace sets access_required —

intersected with the key's own document restriction, if it has one. That set is pushed into every query of the verb (the same restriction a key confined to documents uses); nothing outside it is read.

`access_required`¶

PUT /api/workspaces/{workspace_id}/access-policy
{"access_required": true}

Every document with no list becomes invisible to every asserted principal (a request asserting none — the platform as itself — still reads it). GET on the same path reads it, and the reconcile listing reports it. It is an operator's setting, made when a workspace is provisioned: a write (viewer keys are refused), on the workspace routes and not on /v1/public/* or MCP — an agent holding a write key cannot change who may see the workspace's documents. A body that is not {"access_required": <bool>} is 422.

What counts mean¶

For an end user a document they may not see does not exist, and every reply is computed that way:

Search: matches, counts, close matches and every "others" count are over the user's documents. The completeness statement (coverage.completeness_licensed) is licensed over that set — a restriction from principals is not reported as a narrowing (narrowed_by never says acl for it), because saying so would tell the user that hidden documents exist.
Facts: coverage.sources_in_workspace and sources_answerable count the user's documents; an enumeration's licence is the same as above. A ranked read records its demands as scoped to the user's documents (demands[].scope: source). Each demand lists one page of them: subject_source_ids holds at most source_ids_limit ids (default 20, at most 1000), subject_sources_total counts all of the user's documents, subject_sources_truncated says ids were left out, and subject_sources_next_after, sent back as subject_sources_after on POST /v1/public/facts, returns the next page (the request fields are on the REST verb; the MCP tool does not take them yet). A hidden document is in no page and in no count.
Ask: the answer machine retrieves from the user's documents only, and its warrant's coverage.sources_in_workspace / sources_answerable count them.
sources_unreadable (facts, state, precheck, derive, verify, ask) is empty: the hidden documents are not named.
State: its statements, registrants, grid accounts and pages_not_read are the user's documents'. Its two workspace-wide merge blocks (entity_duplicate_proposals, row_merges — pairs of names and rows across documents) are never shown to an asserted principal, whether or not anything is hidden from them, so their presence says nothing.
Region of a hidden document answers exactly as a document that does not exist (404).

The same request without the header reads everything the platform's key reaches, as before.

The audit trail¶

Every verb-call event records the principals the call asserted (principals, as sent), refused calls included — a non-platform key trying the header is exactly what an auditor looks for. They are erased with the workspace, like the question a call asked.

Limits¶

Per-record access on connect sources is not offered (above).
The available_coordinates menu on a facts reply lists the workspace's clause-type names; it is not narrowed to the user's documents.
A user may see any number of documents. Their documents are never sent to the database as a list: each read carries the rule itself — the principals, the workspace's access_required, the key's own document list — and the database applies it (held at 70,000 visible documents, on SQLite and on Postgres). On Postgres, with the planner statistics autovacuum keeps, a read costs what it reads; on SQLite each read also pays a pass over the user's documents (about 75 ms at 70,000). A workspace where nothing is hidden from the user carries no restriction at all.
A list of document ids on a facts reply is a page, not the whole set: demands[].subject_source_ids, and coverage.sources_unreadable for a key confined to documents. Each comes with its total, a truncated flag and the cursor for its next page, so the reply stays the same size however many documents the user may see (held at 70,000 visible documents: about 5 KB for a question that names no address). The next page is asked of POST /v1/public/facts; the MCP tool does not take the cursors yet, so over MCP an agent reads the count and the first page.