Data handling for a workspace holding personal data¶
What this page covers is built today: where the demo window persists what it saw, how to turn that off, the one-shot purge of records that were once committed, loading a pseudonymised export, and erasing a document with its proof by enumeration. Deployment profiles, keys, the worker and sizing are the rest of this guide and are not written yet (operator).
Where the demo window writes: LIMMA_DATA_DIR¶
The demo window (demo/ui_server.py) persists three things under one root, resolved once
when the server starts (demo/paths.py):
| directory | what lands there |
|---|---|
<root>/runs/ |
every run record — the question, the facts the agent received, the answer; what replay reads |
<root>/cache/ |
the RAG baseline's embedding cache (the document's chunk TEXT and its vectors) and the old demo's recordings |
<root>/state/ |
the old demo's working copies of a corpus store |
The root is $LIMMA_DATA_DIR when it is set (~ expanded, relative paths made absolute)
and demo/ in the checkout otherwise. All three are git-ignored — but a record of a real
customer's document has no business anywhere in a checkout. Set LIMMA_DATA_DIR outside
the repository for anything that is not the synthetic corpus:
import os, tempfile, warnings
from demo import paths
scratch = tempfile.TemporaryDirectory(prefix="limma_data_")
outside = scratch.name
previous = os.environ.get(paths.ENV_VAR)
try:
os.environ[paths.ENV_VAR] = outside
print("runs under the root:", paths.runs_dir() == os.path.join(outside, "runs"))
print("cache under the root:", paths.cache_dir() == os.path.join(outside, "cache"))
# an existing directory of the checkout that is not the default, so the resolver's
# makedirs creates nothing
os.environ[paths.ENV_VAR] = os.path.join(paths.REPO_ROOT, "docs")
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
paths.data_dir()
print("inside the checkout warns:", [w.category.__name__ for w in caught])
a_file = os.path.join(outside, "a-file")
open(a_file, "w").close()
os.environ[paths.ENV_VAR] = a_file
try:
paths.data_dir()
except NotADirectoryError:
print("a file refuses: NotADirectoryError")
finally:
if previous is None:
os.environ.pop(paths.ENV_VAR, None)
else:
os.environ[paths.ENV_VAR] = previous
scratch.cleanup()
runs under the root: True
cache under the root: True
inside the checkout warns: ['RuntimeWarning']
a file refuses: NotADirectoryError
A root inside the checkout that is not the default is warned about, because nothing
ignores that directory and its records are one git add . from history; run the server
with python -W error to make the warning a refusal. A root that names a file refuses to
start rather than fall back into the repo.
Persist nothing: --no-record¶
For a workspace holding a real document, start the window with --no-record:
LIMMA_DATA_DIR=/srv/limma-data .venv/bin/python demo/ui_server.py \
--workspace <workspace id> --port 8766 --no-record
With it the run's events still stream to the page — the room sees the agent and the RAG
baseline — and nothing is written: no run record, no RAG cache on disk (the baseline
indexes in memory), and the replay list is empty (GET /api/runs returns [], a saved
run is a 404). The server says which mode it is in on start: recording OFF
(--no-record): no run records, no RAG cache, replay lists nothing, or recording runs to
<root>/runs. tests/test_demo_records_leave_the_repo.py pins both, including that with
--no-record not even the root directory is created.
Records that were committed: the purge script¶
The window's run records and the RAG cache were tracked in git before the data root
existed. scripts/purge_demo_history.sh removes demo/runs/ and demo/cache/ from EVERY
commit on every ref. It rewrites history, so it is run once, before any real upload:
scripts/purge_demo_history.sh # the plan: what it will do; changes nothing; exit 1
scripts/purge_demo_history.sh --report-only # the history report alone (run it before AND after)
scripts/purge_demo_history.sh --yes # do it
With --yes it:
- refuses if tracked files have uncommitted changes, a stash exists, or another worktree
is attached (each keeps the old history reachable, and
git filter-repo --forcewould skip its own checks for all three), or ifgit filter-repois not installed (exit 2); - copies the working tree's
demo/runsanddemo/cacheto$LIMMA_DATA_DIR/purge-backup-<timestamp>/(or~/.limma/...); - runs
git filter-repo --path demo/runs --path demo/cache --invert-paths --force, which also removes theoriginremote so the old history cannot be pushed by accident; - verifies no file under either path remains in any commit (exit 4 if one does; exit 5 if
the check could not run — never reported as clean) and counts every synthetic expert
name, in every byte spelling a record can hold, over the packed history outside
demo/expert_network/.
Then re-add the remote, git push --force --all origin and git push --force --tags
origin; every other clone must re-clone.
A pseudonymised load¶
A structured export reaches the store through limma_connect with a relation_mapping,
and the connector reads only the columns the mapping names — its SELECT lists them
explicitly (RelationMapping.columns() in epistem/ingestion/sql_connector.py). A
pseudonymised load is therefore two decisions made in the export, not in Limma: the
entity column holds the customer's own opaque id, and names and emails are either stripped
before the export or simply left out of the mapping. The store holds no display name for a
relation entity loaded this way and never derives one (limma_search answers name:
null). (A document_source pull is pseudonymised the same way — leave name_path out of
its mapping; when the mapping names it, each person's display name is stored under the
generation and on the documents registered for them, and limma_search then names the
person by it in every reply that lists them: the next section.)
import sqlite3
from docs.examples.roster import connect, export_dir, new_workspace
ws = new_workspace(client)
client.post("/v1/public/apply_pack", json={"workspace_id": ws,
"bundle_id": "bundle_expert_network"})
export = export_dir() / "pseudonymised.db"
with sqlite3.connect(export) as db:
db.execute("CREATE TABLE roles (row_id INTEGER PRIMARY KEY, expert_id TEXT, "
"full_name TEXT, email TEXT, company TEXT, seniority TEXT)")
db.execute("INSERT INTO roles VALUES (1, 'EXP-0042', 'Mira Tollund', "
"'mira@example.org', 'Kestrel Pumps AG', 'VP')")
pulled = connect(client, ws, export, "roles", {
"relation": "held_role", "entity_column": "expert_id", "key_columns": ["row_id"],
"slot_columns": {"company": "company", "seniority": "seniority"}}).json()
row = client.post("/v1/public/facts", json={
"workspace_id": ws, "question": "roles", "mode": "enumerate",
"relation_query": {"relation": "held_role"}}).json()["relations"]["rows"][0]
print("entity:", row["entity_id"], "| slots:", row["slots"])
print("columns receipted:", sorted(r["db_column"] for r in row["slot_receipts"].values()))
print("name anywhere in the row:", "Mira" in str(row) or "mira@" in str(row))
entity: EXP-0042 | slots: {'company': 'Kestrel Pumps AG', 'seniority': 'VP'}
columns receipted: ['company', 'seniority']
name anywhere in the row: False
What this licenses: the unmapped columns were not read by this pull. It does not license anything about the documents a workspace also ingests — a profile that prints a name carries it into its parse cache and its clauses — nor about what a customer's export already contains.
What a document-store pull stores, and erasing one person from it¶
A document_source pull (operating the connect path)
writes three kinds of thing, and they are not erased by the same act:
| what | where it lives | what erases it |
|---|---|---|
the rows, their statements, coverage and coverage gaps; the pull's own record — the mapping, the counts, and the first 200 quarantined values, each cut to 200 characters; and — when the mapping names name_path — the display name each record states for its subject, one row per subject |
the GENERATION: one source per pull of the collection | deleting the generation's document — every generation of the line goes with it |
each registered bio or call text: the text, and beside it the subject's id, the document's month and — when the mapping names name_path — the subject's display name |
a document OF ITS OWN, one per record and field | deleting THAT document; deleting the generation does not touch it |
| every field the mapping does not name | nowhere — it is read past | nothing to erase |
Three consequences, each easy to get wrong:
- Deleting the generation leaves the bios. The registered texts are documents of their own, so that a re-sync that changes one title does not re-register forty thousand unchanged profiles; the price is that they are erased one by one.
- A person removed upstream is retired, not erased. The next pull retires their
rows — they leave every count — and their bio when every document of that export could
be identified (a line nobody could read, or a document with no usable key, keeps it:
kept_unconfirmed, said inread_errors, until a pull that identifies every document). A bio the export still holds for them but states empty, and a call it still holds but no longer attributes to them, are retired by that pull whatever else it could read. Retired, the text is still in the store, andlimma_regionstill returns it to a caller who holds its source id, until its document is deleted. - "The store holds no display name" is true of a
relation_mappingpull only. A document pull withname_pathstores each person's name twice: on every document registered for them (it is what a reader of the document is told the document is about), and once per person under the generation, for every record that states one — a person with rows and no registered document is named from there. A later pull replaces the generation's names with what it read (a name the export stops stating is removed); a retired generation's names are retired with it and stay in the store until the generation is deleted.
To erase one person from a connected collection: delete each document registered for them, re-sync an export that no longer holds their record (their rows retire with the old generation), and — to remove the retired rows too — delete the line's generation document and connect the export again. Then verify by enumeration, as in Erasure below.
import json, os
from sqlalchemy import create_engine, text
from docs.examples.roster import export_dir, new_workspace
ws = new_workspace(client)
client.post("/v1/public/apply_pack", json={"workspace_id": ws, "bundle_id": "bundle_expert_network"})
export = export_dir() / "experts.jsonl"
people = [{"_id": key, "name": name, "country": "Norway", "updated_at": "2024-03-01T09:00:00Z",
"hourly_rate": 310, "bio": f"{name} has run a cold store for nine years. They began in Bergen.",
"experiences": [{"company_name": "Kestrel Pumps AG", "career_title": "Director",
"date_from": "2016-03-01", "date_till": "2019-01-01"}]}
for key, name in (("EXP-0042", "Mira Tollund"), ("EXP-0043", "Odd Varga"))]
mapping = {"key_path": "_id", "entity_path": "_id", "name_path": "name", "as_of_path": "updated_at",
"relations": [{"relation": "held_role", "iterate": "experiences[]",
"slot_paths": {"company": "company_name", "title": "career_title"},
"period_from_path": "date_from", "period_to_path": "date_till",
"period_end": "exclusive", "missing_end": "open"}],
"documents": [{"path": "bio", "kind": "profile"}]}
store = create_engine(os.environ["DATABASE_URL"])
def pull(records):
export.write_text("".join(json.dumps(r) + "\n" for r in records), encoding="utf-8")
return client.post("/v1/public/connect", json={
"workspace_id": ws, "document_mapping": mapping, "allow_shrink": True,
"document_source": {"path": str(export), "collection": "experts"}}).json()
def held(needle):
"""Which sources of the workspace hold `needle` in their text or their own record."""
with store.connect() as conn:
return sorted((row[0], "current" if row[1] else "retired") for row in conn.execute(text(
"SELECT file_type, COALESCE(is_current, 1) FROM sources WHERE workspace_id = :w "
"AND (extracted_text LIKE :p OR metadata_json LIKE :p)"), {"w": ws, "p": f"%{needle}%"}))
def bio_of(key):
with store.connect() as conn:
return conn.execute(text("SELECT source_id FROM sources WHERE workspace_id = :w "
"AND entity_name = :e AND file_type = 'txt'"),
{"w": ws, "e": key}).scalar()
os.environ["EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS"] = str(export.parent)
try:
first = pull(people)
mira = bio_of("EXP-0042")
print("after the pull, Mira's name is held by:", held("Mira Tollund"))
print("the unmapped rate is held by:", held("310"))
pull(people[1:]) # upstream removed her record
print("removed upstream:", held("Mira Tollund"), "| region still answers:",
client.get("/v1/public/region", params={"workspace_id": ws, "source_id": mira,
"start": 0, "end": 12}).status_code)
client.delete(f"/v1/workspaces/{ws}/documents/{first['source_id']}")
print("the generation deleted:", held("Mira Tollund"), "| Odd's bio:", held("Odd Varga"))
client.delete(f"/v1/workspaces/{ws}/documents/{mira}")
print("her document deleted:", held("Mira Tollund"), "| region:",
client.get("/v1/public/region", params={"workspace_id": ws, "source_id": mira,
"start": 0, "end": 12}).status_code)
finally:
os.environ.pop("EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS", None)
after the pull, Mira's name is held by: [('txt', 'current')]
the unmapped rate is held by: []
removed upstream: [('txt', 'retired')] | region still answers: 200
the generation deleted: [('txt', 'retired')] | Odd's bio: [('txt', 'current')]
her document deleted: [] | region: 404
Mira's name is in ONE source — her registered bio, in its text and on its own record
(subject_name) — and never on the generation or in a row (rows carry her id). The rate
the mapping does not name is nowhere. When her record leaves the export her bio is
retired and still readable by its id; deleting the generation changes nothing for it; and
only the delete of her own document removes it.
Erasure¶
A document — every generation of it — is erased with
DELETE /v1/workspaces/{workspace_id}/documents/{source_id}. The cascade removes its
claims, its relation rows and their statements, coverage rows and slot index entries
(unless another document also states a row — that row survives with the other document's
receipt), the record of what its loader could not read (relation_coverage_gaps, which
can quote a quarantined value), its source row and parse cache, the stored answers and answer records that cited
it, and the corrections of rows only it stated. A legal hold on ANY generation refuses the whole delete before
anything is removed. The integrator's view, with the API-level proof, is on
corrections, generations and erasure.
The route's reply carries claims_removed, contradictions_removed,
orphan_axioms_removed, generations_erased (every generation the delete took) and
relations_erased (relation rows, statements, slot entries, cleared parse caches, the
attest records that pinned an erased generation, and one count per table that holds rows
by document: relation_coverage, relation_coverage_gaps, entity_names,
entity_identities, unplaced_values — the words of the document no vocabulary could
place — criterion_reading_claims (a claim a search's reader holds while it reads the
document) and e1_statements where that table exists). Verify by enumeration,
not by the reply. At the API: ask for the entity at every relation, per entity and workspace-wide,
and confirm it is in no row and no not_recorded list. At the store, an operator with read
access to the database searches EVERY column of EVERY table for the id — not only the
entity columns: an id also sits inside JSON text (a recorded request, an event's metadata),
and a scan of the entity columns alone reports clean over a store that still holds it. The
example first reads Yusuf's roles the way any agent would and corrects one of them, so the
audit trail holds a request and a correction naming him:
import os
from sqlalchemy import create_engine
from docs.examples.roster import new_roster
from epistem.workspace.erasure import rows_naming
r = new_roster(client)
store = create_engine(os.environ["DATABASE_URL"])
def columns_naming(needle, workspace_id):
"""Every column of every table: how many rows of this workspace hold `needle` anywhere
in the column's text (`rows_naming` escapes `_` and `%`, so an id matches only itself)."""
with store.connect() as conn:
return rows_naming(conn, needle, workspace_id=workspace_id)
row_id = client.post("/v1/public/facts", json={
"workspace_id": r.workspace_id, "question": "roles", "mode": "enumerate",
"relation_query": {"relation": "held_role", "entity_id": "yusuf_alderane"}}
).json()["relations"]["rows"][0]["row_id"]
client.post("/v1/public/correct", json={"workspace_id": r.workspace_id, "row_id": row_id,
"reason": "wrong start month", "actor": "ops@example.org"})
print("before:", columns_naming("yusuf_alderane", r.workspace_id))
deleted = client.delete(f"/v1/workspaces/{r.workspace_id}/documents/{r.roles_source}")
print("delete:", deleted.status_code)
print("after:", columns_naming("yusuf_alderane", r.workspace_id))
print("his row id, after:", columns_naming(row_id, r.workspace_id))
print("the screening export's entity is still in relations:",
"relations.entity_id" in columns_naming("odile_fenwright", r.workspace_id))
before: {'audit_events.event_json': 1, 'audit_events.request_json': 1, 'relation_coverage.entity_id': 1, 'relation_provenance.slot_receipts_json': 1, 'relations.entity_id': 1, 'relations.slot_receipts_json': 1}
delete: 200
after: {'audit_events.event_json': 1, 'audit_events.request_json': 1}
his row id, after: {'audit_events.event_json': 3}
the screening export's entity is still in relations: True
yusuf_alderane appears only in the roles export, so after its delete no relation,
coverage or source table names him. Before it, his id also sat inside the slot receipts:
a structured receipt records the cells that bind its row to its fact (the subject and
period columns), so a re-keyed or re-assigned row can never verify — which is exactly why
a scan of the entity columns alone would under-report. Two audit rows still name him: the facts call's
recorded request, and the correction event's metadata. His row id survives in three
(the ids the facts call returned, the correct call's record, the correction's
target). odile_fenwright is also stated by the screenings export, which was not
deleted. Run the scan over the store the deployment actually uses (DATABASE_URL), with
the ids the erasure request named — it reads every row, so on a large store run it against
a copy; drop the workspace filter to check every workspace an entity id might also appear
in.
What a document delete leaves¶
A DOCUMENT delete reaches the document's trail. Every event of the workspace's log
(events) and every audit event (audit_events) that names the document — the event log
records the document each event is about, and an id inside an event's text counts too —
keeps its SHELL and loses its content, by the one rule the workspace delete and the S3
archive use: its id, type, time, action, result and key are kept, and so are its counts
(claims read, rows removed, milliseconds); its file name, a claim's text, a disagreement's
values, the question a call asked and the spans it returned are erased, and the shell no
longer names the document. A verb call so redacted cannot be replayed
(replay_unavailable_reason: document_erased). The delete's own records — the
source_deleted event and the delete_document audit event — name the document's id and
nothing else, not its file name. The reply's relations_erased counts what was redacted
(events_redacted, audit_events_redacted), and residue reads the trail too: a delete
never answers complete while an event still names the document. Events an earlier
release wrote, before each named its document, are decided inside the delete that needs
them: tied to their document by the claim, the id or the file name they name, or — when
that document is already gone — redacted.
What the document delete leaves of an ENTITY it named: a correction's audit event names
the relation row it retired and its metadata.entity_id, not the document, and is kept (the
audit trail is described on operating the connect path);
a call that never named the document — its question, its request — is kept as well. The
running API's in-memory buffer of recent audit events keeps them whole until a restart,
and the demo window's run records (above) hold what an agent was told. So a request to
erase every stored trace of ONE entity inside a workspace that lives on is not met by the
document delete alone: the scan above is how an operator finds what remains, and deleting
the workspace (below) is what erases the trail's content.
The delete also keeps ONE row of its own per erased generation: source_erasures (the
generation's id, its workspace and when — no content). It is written BEFORE anything is
deleted and it is what makes the erasure final while the document is still being read:
the database refuses every later write that names an erased generation, in every table
that holds a document's rows (the ingest worker's, the part labeller's, a criterion
reader's), so a reading that outlives the delete cannot write the document back; that
reading ends skipped: source_erased. /health names any such table left without that
guard (deployment.unfenced_tables, [] on a healthy store). The workspace delete removes
these rows with the rest of the workspace.
What else goes with a document. Its own disagreements — two of its values for one
metric and period, recorded with no claim named — are recorded with the document and
erased with it, as is every finding one of its claims is party to; so are the definition
edges between its clauses. A contradiction row written before rows carried their document
is decided once, at boot: attributed to its document where that can be determined, and
removed where it cannot (its document is gone, or several documents state its values) —
each affected workspace's events log gets one contradiction_sources row with the
counts, and the boot logs a WARNING when any row was removed. With the ingest queue, the
document's job goes too: a queued, finished or failed job holds the document's bytes and
its result, and is deleted with them (jobs_forgotten in the reply; null when the queue
could not be asked — the delete is then not complete, and the same call asks again). A job
still running is fenced, and takes its own record with it when it ends.
An interrupted delete is finished by the same call. When the database fails part way,
the document stays fenced and the record of its external id is the LAST row the delete
removes: the same POST /v1/public/delete_document (or DELETE …/documents/{id} with its
id) finds it again, finishes it, and answers complete: true with
finished_interrupted_erasure: true.
Deleting a workspace¶
DELETE /api/workspaces/{workspace_id} with an admin key deletes the workspace and
proves it: the tables are read from the database's live schema, not from a list, so a
table added in a later release cannot be left out. The reply:
{"status": "deleted", "workspace_id": "…",
"erasure": {
"complete": true,
"rows_deleted": {"relations": 4, "relation_provenance": 4, "sources": 1, …},
"tables_checked": 66,
"audit_rows_redacted": 12,
"source_blobs_deleted": 1,
"jobs_forgotten": 0,
"jobs_remaining": [],
"residue": {},
"process_state_evicted": {"connected_source_adapters": 1, …},
"warnings": []}}
rows_deleted— rows removed, per table, for every table that held any.residue— rows STILL carrying the workspace, per table, read back after the commit.{}is the proof;tables_checkedis how many tables that check read.jobs_forgotten/jobs_remaining— a queued, failed or finished ingest job carries the document's bytes; every job of the workspace that is not running is deleted from the queue, and any id still there is named.source_blobs_deleted— objects removed from the source backend, found by the workspace's prefix (so bytes whose row was never written go too).audit_rows_redacted— see below.
The answer is 200 only when residue is {} and jobs_remaining is []. Otherwise:
| status | detail.error |
what happened | what to do |
|---|---|---|---|
| 409 | legal_hold |
an active hold; nothing changed | release the hold |
| 409 | workspace_busy |
an ingest is running (running names the job); its queued jobs were cancelled (cancelled), nothing was deleted |
retry when the job has ended |
| 500 | unclassified_table |
the database holds a table the delete cannot place (tables); nothing changed |
drop a table that is not Limma's, or install a release that classifies it — /health names it from boot under deployment.unclassified_tables |
| 500 | erasure_incomplete |
the delete ran and something is left (erasure.residue, erasure.jobs_remaining) |
send the same DELETE again |
| 404 | no such workspace, and nothing of it left |
The delete is final for late writers too. It keeps ONE row of its own:
workspace_erasures — the workspace's opaque id and when it was deleted; no name,
organization or content — written in the delete's own transaction (a delete that fails
leaves none) and kept for the life of the database. From then on the database refuses every
write naming the workspace in every table the delete takes, so a background reader still
holding one of its documents (the part labeller) writes nothing back, and the id is never
made again. The audit trail alone is not refused (below).
A DELETE of a workspace that no longer exists does something in one case only: rows or
jobs of it are left (an audit record of a call made after the delete, or a job) and the
caller is an admin of the organization that owned it. For every other caller an erased
id answers 404, exactly as an id that never existed. A verb call made on an erased id
afterwards is answered 404 and is recorded in the trail like any call, with what its
caller sent: that is content filed under the erased id, the residue check counts it
(audit_events), and the same DELETE by the owner's admin redacts it.
A search's criteria are read in the background, after the search has returned. A judgment that comes back from the model after its workspace — or its document — was deleted is not written: the reader looks for the document again, under the lock the delete holds, before it writes. The same holds for the claim it takes on each document before reading it (so two API processes never read one document twice): both deletes clear the claims they find, and no claim is written for a document that is gone.
The audit trail is kept and its content erased. For every event filed under the
workspace the row stays — its id, its type, when, which organization, which key, which
verb, the status — and what it said is removed: the recorded request (request_json),
the actor a caller typed, a domain event's target and metadata (it reads
{"erased": true, "erased_at": …}), and from a verb call's record the question or
brief, the fact ids and spans returned and the criteria read. A redacted call lists with
replay_unavailable_reason: "workspace_erased", and replaying it answers 409
replay_unavailable. The deletion's own event is written after the redaction and
carries the owning organization and the counts above, never an id.
What a workspace delete does NOT reach: your export files in ./imports (Limma reads
them and never deletes them), backups taken earlier, Postgres dead tuples until vacuum,
container logs, the running API's in-memory copy of recent audit events until a restart,
the S3 audit sink when one is configured, another API replica's caches until its
restart, and your agents' own transcripts.
Where personal data is held¶
Every table that can hold a person's data, and every place outside the database, with
how each delete reaches it. This table is generated from the erasure census
(epistem/workspace/erasure.py) — the same census the delete runs from — and a test
fails when the two differ.
| table | what it holds | workspace delete | document delete |
|---|---|---|---|
answer_records |
stored answers: the question text and the value delivered | every row of the workspace is deleted | not per-document |
ask_answers |
stored /ask answers: the question and the answer text | every row of the workspace is deleted | not per-document |
ask_index_source_docs |
the /ask index over document text (nothing on the live path writes it) | every row of the workspace is deleted | not written by the product; removed with the workspace |
audit_events |
the audit trail: who called which verb, when, with what status. A workspace delete KEEPS each row and erases its content (the recorded request, the question, the ids returned, the actor typed); a document delete does the same to every event naming the document | rows kept, content erased | each event naming the document keeps its skeleton; its content is erased (the delete's own record names the document's id and nothing else) |
claims |
numeric and prose statements extracted from a document | every row of the workspace is deleted | the document's rows are deleted |
comments |
comments on a claim | every row of the workspace is deleted | not per-document |
contradictions |
pairs of claims that disagree, and a document's own disagreements: the metric, the entity, the period and both values | every row of the workspace is deleted | the document's rows are deleted |
document_access |
who may see each document: the principals its access list names (a user's address, a group) | every row of the workspace is deleted | the document's rows are deleted |
e1_statements |
prose statements of the E1 extractor, with the clause body | every row of the workspace is deleted | the document's rows are deleted |
entity_aliases |
confirmed spellings of one entity's name, with the evidence | every row of the workspace is deleted | not per-document |
entity_duplicate_proposals |
names that look like one entity and were not merged | every row of the workspace is deleted | not per-document |
entity_identities |
the names and pseudonymised identifier keys each load read for a person | every row of the workspace is deleted | the document's rows are deleted |
entity_link_decisions |
an operator's merge and un-merge decisions, with the reason and actor typed | every row of the workspace is deleted | not per-document |
entity_link_pairs |
pairs of records judged for being one person, with the evidence | every row of the workspace is deleted | not per-document |
entity_links |
which records are one person | every row of the workspace is deleted | not per-document |
entity_names |
the display name each load read for a person | every row of the workspace is deleted | the document's rows are deleted |
events |
the workspace's event log: what happened to each document and when — its file name, a claim's text, a disagreement's values, a reading's progress | every row of the workspace is deleted | each event about the document keeps its shell (type, time, its own ids, counts); its content and the document's id are erased |
external_documents |
the source system's id and version for each synced document (a connector's file id, which can be a path naming a person) | every row of the workspace is deleted | deleted last, once the rest of the document is gone (an interrupted delete is finished by the same external id) |
extraction_checkpoints |
an interrupted extraction's partial result | deleted through the row it belongs to | deleted with the document's source row |
idea_groupings |
for each idea a question asked, the verdict on each part label (the idea's words and the label's words); a document delete removes the verdicts on labels no remaining document holds | every row of the workspace is deleted | not per-document |
ingestion_impacts |
what each upload changed (counts and the file name) | every row of the workspace is deleted | the document's rows are deleted |
label_changes |
one row per change of the workspace's names (added, retired, re-spelled, an alias accepted or undone, a proposal rejected): the version, the name, who decided; a document delete erases the names it retires from every row, keeping the version and the kind | every row of the workspace is deleted | not per-document |
label_decisions |
an operator's decisions on naming proposals (the name, the name it was proposed as, the decision, the reason and actor typed) and the old spelling of each re-spelled name; a document delete removes the decisions about the names it retires | every row of the workspace is deleted | not per-document |
label_names |
every name a current fact of the workspace carries, with the version that added it; a document delete removes the names it retires | every row of the workspace is deleted | not per-document |
label_proposals |
for each name a document's facts carry, the names a similarity tier proposed for it (the label as printed, the tiers and their scores, a count) | every row of the workspace is deleted | the document's rows are deleted |
proofs |
formula recalculations over the workspace's figures: the formula, its inputs' values and the result | every row of the workspace is deleted | every row of the workspace is deleted (each is computed from all its documents); the next reading computes them again |
pull_record_digests |
for each record a pull through a mapping loaded, the record's key as the export states it and a SHA-256 of what was loaded from it (no value) | every row of the workspace is deleted | the document's rows are deleted |
relation_coverage |
which entities each document was read for, per relation | every row of the workspace is deleted | the document's rows are deleted |
relation_coverage_gaps |
what a loader could not read for an entity; can quote the value it set aside | every row of the workspace is deleted | the document's rows are deleted |
relation_provenance |
which document states each fact, with its receipt | every row of the workspace is deleted | deleted, unless another document also states the fact |
relation_schemas |
the relation schemas in force, and for a criterion a search minted or a schema drafted from the customer's records, its definition (the criterion's text) | every row of the workspace is deleted | not per-document |
relation_slot_index |
the searchable values of each fact's slots | every row of the workspace is deleted | not per-document |
relations |
one row per fact about an entity (a role held, a place, a screening), with its slots and receipt | every row of the workspace is deleted | deleted, unless another document also states the fact |
semantic_claims |
prose statements with their verbatim text and embedding | every row of the workspace is deleted | the document's rows are deleted |
sources |
one row per document generation: file name, subject, the parsed text and document map | every row of the workspace is deleted | the document's rows are deleted |
statement_conflicts |
pairs of a document's statements judged to state the same property with different values: each statement's id, its value quoted as printed with its position, and the model that judged it | every row of the workspace is deleted | the document's rows are deleted |
unplaced_values |
words of a document the vocabulary could not place, quoted with their position | every row of the workspace is deleted | the document's rows are deleted |
upload_links |
one row per upload link an agent asked for, until it expires: the SHA-256 of its token and of the key that minted it, and the ingest fields the upload is read under (a file name, the subject, an access list naming people) | every row of the workspace is deleted | not per-document |
value_mentions |
every date, amount, percentage and duration a document prints, quoted as printed with its position and its normalised value | every row of the workspace is deleted | the document's rows are deleted |
| outside the database | what it holds | how a workspace delete reaches it |
|---|---|---|
| source bytes | each document's original bytes, under <workspace_id>/<source_id> in the source backend (memory, a directory, or a bucket) |
SourceBackend.delete_workspace: everything under the workspace's prefix, whether or not a row names it — BEFORE the database commit, so a failure rolls the delete back |
| ingest jobs | a queued, failed or finished ingest job carries the document's bytes (Redis keeps a failed one for 24 hours) | forget_workspace_jobs: every job of the workspace that is not running is deleted from every registry, before the delete and again after it; a RUNNING job refuses the delete (409) |
| process memory | the calibration, preference, scale-resolver and answer-corpus caches; a connected source's open engine; a criterion's background reading | evicted by WorkspaceManager.delete_workspace in the process that served the delete; another replica's caches and the in-memory audit download buffer end with a restart |
| export files | the customer's own export under ./imports |
never: Limma reads those files and does not own them — the operator removes them |
| uploaded exports | an export file uploaded for a connect (POST /v1/public/upload_export), under <workspace_id>/<export_id> in the source backend, until a pull reads it (a pull that writes its generation deletes it) or the backend's own rule ends it; a connect's temporary copy is removed when that connect ends |
SourceBackend.delete_workspace, with the source bytes: everything under the workspace's prefix — counted in source_blobs_deleted |