Skip to content

Corrections, generations and erasure

A relation row can be wrong, can change at the source, and can have to disappear. Those are three different events with three different verbs, and none of them silently rewrites what an earlier answer cited:

event verb what happens to the row
the store holds a wrong row POST /v1/public/correct (limma_correct) retired from every address, kept on record with who, when and why
the source changed POST /v1/public/connect with replaces (limma_connect) the old generation retires; limma_diff names each changed slot
the data must be gone DELETE /v1/workspaces/{workspace_id}/documents/{source_id} that generation, every generation it replaced, and every row only they stated are erased

The examples each build a fresh copy of roster with new_roster(client), because they change it. Every name is invented.

Correcting a row

limma_correct(row_id, reason, actor, workspace_id=...) retires one relation row. row_id is the row_id of a relations.rows[] entry; reason and actor are required and recorded on the row and in exactly one audit event:

from docs.examples.roster import new_roster
ws = new_roster(client).workspace_id
rows = client.post("/v1/public/facts", json={
    "workspace_id": ws, "question": "roles", "mode": "enumerate",
    "relation_query": {"relation": "held_role", "entity_id": "halvard_quist",
                       "where": [["title", "eq", "Director, Sales"]]}}).json()["relations"]["rows"]
row_id = rows[0]["row_id"]
r = client.post("/v1/public/correct", json={
    "workspace_id": ws, "row_id": row_id,
    "reason": "title misrecorded in the CRM", "actor": "ops@example.org"})
body = r.json()
c = body["correction"]
print(r.status_code, "| retired:", body["retired"], "| audit event:", body["audit_event_id"] is not None)
print(c["entity_id"], c["relation"], c["slots"]["title"], c["period_from"], "->", c["period_to"])
print("retired_by:", c["retired_by"], "| reason:", c["retired_reason"], "| at:", bool(c["retired_at"]))

again = client.post("/v1/public/correct", json={
    "workspace_id": ws, "row_id": row_id, "reason": "a second opinion", "actor": "someone"})
print(again.status_code, again.json()["detail"]["type"],
      "| the first stands:", again.json()["detail"]["correction"]["retired_by"])
blank = client.post("/v1/public/correct", json={
    "workspace_id": ws, "row_id": row_id, "reason": "  ", "actor": "ops@example.org"})
unknown = client.post("/v1/public/correct", json={
    "workspace_id": ws, "row_id": "rel_doesnotexist", "reason": "r", "actor": "a"})
print("blank reason:", blank.status_code, "| unknown row:", unknown.status_code)

200 | retired: True | audit event: True
halvard_quist held_role Director, Sales 2019-01 -> None
retired_by: ops@example.org | reason: title misrecorded in the CRM | at: True
409 already_corrected | the first stands: ops@example.org
blank reason: 422 | unknown row: 404

  • A second correction of the same row is a 409 carrying the first correction under detail.correction; it is never overwritten.
  • A blank reason or actor is a 422. A row this workspace does not hold is a 404 — and so is a row no document your key may read states, so a scoped key cannot probe for rows.
  • RETIRED, NEVER DELETED: the row stays in the store so it can be listed and audited. Its documents' statements and its entity's coverage stay too — the source still says what it said, and was still read; the correction records that the fact is wrong. A re-load of the same content does not bring it back.

The audit event is event_type: correction, action: correct_relation_row, target: <row id>, actor_id: <actor>, with the reason, relation and entity in its metadata; audit_event_id on the response is its id. How an operator reads it is in operator: connect.

What a correction does to a count

A corrected row is at no address — no limma_facts query, no search, no count finds it. But its TRUE value is unrecorded while a document in scope still states it, so a count that simply dropped it would license these are all of them over a set with a known hole. It does not:

from docs.examples.roster import new_roster
ws = new_roster(client).workspace_id
row_id = client.post("/v1/public/facts", json={
    "workspace_id": ws, "question": "roles", "mode": "enumerate",
    "relation_query": {"relation": "held_role", "entity_id": "halvard_quist",
                       "where": [["title", "eq", "Director, Sales"]]}}).json()["relations"]["rows"][0]["row_id"]
client.post("/v1/public/correct", json={"workspace_id": ws, "row_id": row_id,
            "reason": "title misrecorded in the CRM", "actor": "ops@example.org"})

body = client.post("/v1/public/facts", json={
    "workspace_id": ws, "question": "roles", "mode": "enumerate",
    "relation_query": {"relation": "held_role", "entity_id": "halvard_quist"}}).json()
print("halvard's roles:", [x["slots"]["title"] for x in body["relations"]["rows"]],
      "| count:", body["relations"]["count"])
print("licensed:", body["coverage"]["completeness_licensed"], body["coverage"]["narrowed_by"])

search = client.post("/v1/public/search", json={"workspace_id": ws, "constraints": [
    {"relation": "held_role", "slot": "company", "op": "eq", "value": "Lumen Ridge"}]}).json()
print("search matched:", search["entities"]["matched"],
      "| undetermined:", search["entities"]["undetermined"])
print("search coverage:", search["coverage"])

listing = client.get("/v1/public/corrections", params={"workspace_id": ws}).json()
print("corrections:", listing["count"],
      [(e["entity_id"], e["slots"]["company"], e["retired_by"]) for e in listing["corrections"]])

halvard's roles: ['Vice President, Sales'] | count: 1
licensed: False ['corrected_rows']
search matched: [] | undetermined: ['halvard_quist']
search coverage: {'completeness_licensed': False, 'withheld_because': 'narrowed', 'narrowed_by': ['corrected_rows_unanswerable:held_role']}
corrections: 1 [('halvard_quist', 'Lumen Ridge', 'ops@example.org')]

  • limma_facts delivers the rows that remain and withholds the licence with narrowed_by: corrected_rows — over the query's own entity, or the whole relation workspace-wide, whatever the where: the missing row's true value could satisfy any of it.
  • limma_search puts an unmatched entity holding a corrected row in undetermined, never none, and names the relation (corrected_rows_unanswerable:<relation>). Halvard's Lumen Ridge role was the one being corrected; "no recorded row says he worked at Lumen Ridge" is exactly the claim that cannot be made.
  • An entity whose EVERY row of a relation was corrected is not a licensed none: its absence verdict is not_recorded with the row ids in corrected_rows.
  • GET /v1/public/corrections (limma_corrections) lists every correction in your scope, oldest first. It is the one place a corrected row is still named.

The hold lifts for one reason only: a re-sync whose export no longer states the corrected row, below.

Generations: a re-sync with replaces

A connected table is a source; pulling it again is a new generation of that source when you say so with replaces=<the source id you hold>. The new generation is written first and the old one retired after. Without replaces, an unchanged table deduplicates to the source you already hold (deduplicated: true), and a changed one becomes a second, unrelated source with both sets of rows live — so a routine re-sync always names what it replaces.

from docs.examples.roster import new_roster, update_export, connect, ROLES_MAPPING
r = new_roster(client)
ws = r.workspace_id
row_id = client.post("/v1/public/facts", json={
    "workspace_id": ws, "question": "roles", "mode": "enumerate",
    "relation_query": {"relation": "held_role", "entity_id": "halvard_quist",
                       "where": [["title", "eq", "Director, Sales"]]}}).json()["relations"]["rows"][0]["row_id"]
client.post("/v1/public/correct", json={"workspace_id": ws, "row_id": row_id,
            "reason": "title misrecorded in the CRM", "actor": "ops@example.org"})

# The warehouse fixes the title; the nightly re-sync names the generation it replaces.
update_export(r.export, "UPDATE roles SET title = ? WHERE row_id = ?", ("Head of Sales", 2))
pulled = connect(client, ws, r.export, "roles", ROLES_MAPPING, replaces=r.roles_source).json()
new_source = pulled["source_id"]
print("rows_written:", pulled["relations"]["rows_written"],
      "| replaced the old generation:", pulled["relations"]["replaced"] == r.roles_source)

chain = client.get("/v1/public/generations",
                   params={"workspace_id": ws, "source_id": r.roles_source}).json()["generations"]
print("generations:", [(g["is_current"], g["superseded_by"] == new_source) for g in chain])

diff = client.get("/v1/public/diff", params={
    "workspace_id": ws, "from_source_id": r.roles_source, "to_source_id": new_source}).json()
print("linked:", diff["linked"], "| counts:", diff["counts"])
for change in diff["changes"]:
    print(change["kind"], change["coordinate"]["entity_id"], change["coordinate"]["db_table"],
          change["coordinate"]["row_key"], change["changed_slots"])

body = client.post("/v1/public/facts", json={
    "workspace_id": ws, "question": "roles", "mode": "enumerate",
    "relation_query": {"relation": "held_role", "entity_id": "halvard_quist"}}).json()
print("halvard now:", sorted(x["slots"]["title"] for x in body["relations"]["rows"]),
      "| licensed:", body["coverage"]["completeness_licensed"])
print("corrections still on record:",
      client.get("/v1/public/corrections", params={"workspace_id": ws}).json()["count"])

rows_written: 6 | replaced the old generation: True
generations: [(False, True), (True, False)]
linked: True | counts: {'added': 0, 'removed': 0, 'changed': 1, 'unchanged': 5}
changed halvard_quist roles {"row_id":2} {'title': {'before': 'Director, Sales', 'after': 'Head of Sales'}}
halvard now: ['Head of Sales', 'Vice President, Sales'] | licensed: True
corrections still on record: 1

  • limma_generations returns the chain oldest first; is_current: false on the generation your receipt names means an answer you gave from it needs limma_diff before it is re-used (supersession).
  • On a relation changed, coordinate is {entity_id, relation, db_table, row_key} for a connected row — rows pair across generations on the export's key — and changed_slots names exactly the slots (and period_from / period_to) that differ, before and after. before_row_id / after_row_id name the two rows. A document-stated row has no export key and reads as removed + added.
  • The re-sync no longer states the corrected row (row 2 now says "Head of Sales"), so the correction's hold lifted and the licence is back; the correction itself stays listed. Re-syncing an export that STILL states the corrected row keeps it retired and keeps the licence withheld.

Erasure

DELETE /v1/workspaces/{workspace_id}/documents/{source_id} hard-deletes a document — that generation and every generation it replaced (a delete of the current generation erases the whole chain behind it; a delete of a retired generation never erases the generations that replaced it, so the current one and its facts stand) — and cascades through the relation rows only it stated, their statements, their coverage rows, the slot index, the parse cache, and the corrections of rows only it stated. A legal hold on ANY generation refuses the whole delete before anything is removed. A row another document also states survives, with that document's receipt.

The reply names what the delete took: generations_erased (every generation, deepest first) and relations_erased (relation rows, statements, slot entries, coverage rows, coverage gaps — what a loader could not read — cleared parse caches and the attest records that pinned an erased generation). A relation export holds no claims, so claims_removed: 0 alone would read as a delete that did nothing. It also carries what it read back: residue (rows still holding an erased generation, per table) and complete — a 200 always has residue: {}; rows left are a 500 erasure_incomplete naming them. A reading of the document still running cannot add to it: the delete fences the document before it deletes anything. Erasure is still verified by enumeration, not by the reply: ask for the erased entity at every address and confirm it is at none — including the not-recorded listings. Asked by name, an erased entity reads unbound_because: entity_unknown with not_recorded: [], exactly as an id the workspace never held.

from docs.examples.roster import new_roster, connect, ROLES_MAPPING
r = new_roster(client)
ws = r.workspace_id
second = connect(client, ws, r.export, "roles", ROLES_MAPPING, replaces=r.roles_source).json()
row_id = client.post("/v1/public/facts", json={
    "workspace_id": ws, "question": "roles", "mode": "enumerate",
    "relation_query": {"relation": "held_role", "entity_id": "yusuf_alderane"}}).json()["relations"]["rows"][0]["row_id"]
client.post("/v1/public/correct", json={"workspace_id": ws, "row_id": row_id,
            "reason": "wrong start month", "actor": "ops@example.org"})

deleted = client.delete(f"/v1/workspaces/{ws}/documents/{second['source_id']}")
print("delete:", deleted.status_code, sorted(deleted.json()))

def enumerate_(query):
    return client.post("/v1/public/facts", json={
        "workspace_id": ws, "question": "erasure check", "mode": "enumerate",
        "relation_query": query}).json()["relations"]

for relation in ("held_role", "screening", "engagement"):
    one = enumerate_({"relation": relation, "entity_id": "yusuf_alderane"})
    every = enumerate_({"relation": relation})
    listed = {x["entity_id"] for x in every["rows"]} | set(every["not_recorded"])
    print(relation, "| yusuf bound:", one["bound"], "rows:", one["rows"],
          "| anywhere in the workspace listing:", "yusuf_alderane" in listed)
for generation in (r.roles_source, second["source_id"]):
    print("generations of an erased source:", client.get("/v1/public/generations", params={
        "workspace_id": ws, "source_id": generation}).status_code)
print("corrections:", client.get("/v1/public/corrections", params={"workspace_id": ws}).json()["count"])
survivor = enumerate_({"relation": "screening", "entity_id": "halvard_quist"})
print("halvard's screening, stated by the other document:", survivor["count"])

delete: 200 ['claims_removed', 'complete', 'contradictions_removed', 'document_id', 'filename', 'finished_interrupted_erasure', 'generations_erased', 'jobs_forgotten', 'orphan_axioms_removed', 'relations_erased', 'residue']
held_role | yusuf bound: False rows: [] | anywhere in the workspace listing: False
screening | yusuf bound: False rows: [] | anywhere in the workspace listing: False
engagement | yusuf bound: False rows: [] | anywhere in the workspace listing: False
generations of an erased source: 404
generations of an erased source: 404
corrections: 0
halvard's screening, stated by the other document: 1

Every relation, per entity and workspace-wide, answers with Yusuf at no address: not a row, not none, not even not recorded — an erased entity is gone, not unknown. Both generations of the roles export are gone (404), the correction of a row only it stated went with it, and Halvard's screening, which the other document states, survived.

The delete's reply carries the cascade's claim counts (claims_removed, contradictions_removed, orphan_axioms_removed), the generations it erased, the relation counts, and what it read back afterwards (residue, complete). Enumeration, as above, is the check that does not depend on the reply. It proves the entity is at no ADDRESS; it does not prove the store holds no trace of it. The audit trail is not erased with the document: your own verb calls' recorded requests (a relation_query naming the entity) and the correction's audit event still name it. An operator's store-level scan, and what a document delete leaves, are in operator: data handling.