Operating the connect path¶
POST /v1/public/connect (limma_connect) loads a table as addressed facts — numeric
facts with a mapping, relation rows with a relation_mapping — or an export file with
a document_mapping: a document-store export (below), or a
CSV / TSV file or an Excel sheet, whose rows are records
(CSV and Excel). This page is
what an operator configures and watches: which databases and files it may open, what a
re-sync does to a source's generations, what a correction records and how to read its
audit event. The
integrator's side of the same verbs is on
corrections, generations and erasure.
Which databases it may open¶
A connection string in a request is a request to open a socket on the caller's behalf, so the verb opens NONE until an operator says where it may:
| variable | gates | unset (the default) |
|---|---|---|
EPISTEM_CONNECT_ALLOWED_HOSTS |
network drivers — PostgreSQL (postgresql://), MySQL and MariaDB (mysql+pymysql://, mariadb+pymysql://; the image carries PyMySQL): the URL's host must be listed, by name |
every network URL refused |
EPISTEM_CONNECT_ALLOWED_SQLITE_DIRS |
sqlite:/// files: the resolved path must be under a listed directory |
every sqlite URL refused |
EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS |
document export files (document_source.path): the resolved path must be under a listed directory |
every export file refused |
An export file the caller UPLOADS needs none of the three: see A file the API host cannot reach.
All three are comma-separated and read on every call. Refused under ANY listing: this deployment's own store (by resolved path, so a symlink does not help) and any SQLite file that holds a workspace store's tables (a copy or a backup); a relative sqlite path; a network URL carrying query parameters (several override the host). Every refusal is a 422 naming what to change, and nothing is opened:
import os, sqlite3
from docs.examples.roster import ROLES_MAPPING, export_dir, new_workspace
ws = new_workspace(client)
client.post("/v1/public/apply_pack", json={"workspace_id": ws, "bundle_id": "bundle_expert_network"})
exports = export_dir()
store_copy = exports / "backup_of_a_store.db"
with sqlite3.connect(store_copy) as db:
for table in ("workspaces", "sources", "claims"):
db.execute(f"CREATE TABLE {table} (id TEXT)")
def attempt(url):
r = client.post("/v1/public/connect", json={
"workspace_id": ws, "database_url": url, "table": "roles",
"relation_mapping": ROLES_MAPPING})
print(r.status_code, "|", r.json()["detail"])
print("-- nothing configured")
attempt(f"sqlite:///{exports / 'export.db'}")
attempt("postgresql://warehouse.internal:5432/crm")
print("-- the export directory listed")
os.environ["EPISTEM_CONNECT_ALLOWED_SQLITE_DIRS"] = str(exports)
try:
attempt("sqlite:///relative/export.db")
attempt(f"sqlite:///{store_copy}")
finally:
os.environ.pop("EPISTEM_CONNECT_ALLOWED_SQLITE_DIRS", None)
-- nothing configured
422 | file-backed sources are not enabled on this deployment. A sqlite URL names a path on the API host rather than a database of yours; an operator enables specific directories with EPISTEM_CONNECT_ALLOWED_SQLITE_DIRS.
422 | this deployment has no connectable databases configured, so /connect is disabled. An operator enables it by setting EPISTEM_CONNECT_ALLOWED_HOSTS to the hosts this API may reach.
-- the export directory listed
422 | a relative sqlite path is refused — it would be opened against the API process's working directory, which the caller cannot know. Give an absolute path (four slashes: sqlite:////abs/path.db).
422 | that path is a workspace store (this deployment's own, a copy or a backup) and can never be connected as a source.
Use a read-only database account: the connector issues SELECT and nothing else, but
the account is yours to scope. The documentation's examples set
EPISTEM_CONNECT_ALLOWED_SQLITE_DIRS to their export's directory for the duration of each
call (docs/examples/roster.py); a deployment sets it once in the
API server's environment:
EPISTEM_CONNECT_ALLOWED_SQLITE_DIRS=/srv/exports \
.venv/bin/uvicorn epistem.api.app:create_app --factory --port 8000
A file the API host cannot reach: upload it, then connect its id¶
A document_source.path names a file ON THE API HOST. A caller of a deployment it does
not run cannot put a file there, and a deployment of several API processes behind a load
balancer shares no disk between them, so a file placed on one process's disk is absent
for the process the connect reaches. Such a caller uploads the export instead, and the
connect names what the upload answered:
POST /v1/public/upload_export— multipart: the form fieldworkspace_idand the file asfile. It answersexport_id(minted by the server:exp_, 32 random hex digits, the file's suffix),bytes,sha256,filename,format,expires_atandretention. The SDK'supload_export(workspace_id, "experts.jsonl")sends the same request. There is no MCP tool for it: a file's bytes do not belong in a tool call's arguments, so the integrator uploads and the agent connects the id.POST /v1/public/connectwithdocument_source: {"export_id": "<it>", "collection": "experts"}in place ofpath—mode: proposefirst, then the pull, exactly as for a path: the same reader reads it (its format is its name's, orformat), from a private temporary copy (a directory only the API's user may open) that is removed when the connect ends, whatever it answered.- A pull that writes its generation DELETES the export (
export.deleted: trueon the reply): a re-sync is tonight's file uploaded and connected again, and an unchanged one writes nothing. A proposal, and a refused pull, keep it for the next call.
WHERE THE BYTES ARE HELD: the deployment's source backend, LIMMA_SOURCE_BACKEND — the
one every document's bytes already go through — under <workspace_id>/<export_id>. With
several API processes it must be one they all reach: s3 (a bucket,
LIMMA_SOURCE_S3_BUCKET, whose lifecycle rule is reported as expires_at) or
filesystem (LIMMA_SOURCE_DIR on a disk every process mounts). The default, memory,
holds an upload in the one process that took it until it restarts — the reply's
retention says so — and a connect that reaches another process answers 404. Deleting
the workspace removes every upload under its prefix
(data handling).
| variable | what it sets | unset (the default) |
|---|---|---|
EPISTEM_CONNECT_MAX_EXPORT_BYTES |
the largest file the upload takes; over it, 413 naming the variable. Read at start: a value that is no whole number above zero refuses the start | 536870912 (512 MiB) |
| refusal | when |
|---|---|
422 export_name_refused |
the file's name states no format the reader takes: it must end in .csv, .tsv / .tab, .xlsx / .xlsm, .json / .jsonl / .ndjson (an upload is never read as JSON by default, as a path of any other name is) |
| 413 | the file is over EPISTEM_CONNECT_MAX_EXPORT_BYTES; nothing is stored |
| 403 | a read-only key, a key confined to documents, a key of another workspace or organisation — the same callers connect refuses |
| 422 | a document_source naming both path and export_id, or neither; export_id_invalid for a string that is no id this deployment mints |
404 export_not_found |
this workspace holds no export under that id: never uploaded HERE, already read by a pull, or ended by the backend's rule. An id of another workspace is this same 404, so its existence is never told |
503 export_not_stored / export_unreadable |
the source backend failed the write or the read; nothing was stored or written |
No route reads the bytes back, and nothing re-reads them after the pull: a row read from
an upload re-reads row_unreadable (could not check) — the customer's system is
re-checked by uploading its export again.
import json
from docs.examples.roster import new_workspace
ws = new_workspace(client)
client.post("/v1/public/apply_pack", json={"workspace_id": ws, "bundle_id": "bundle_expert_network"})
records = [
{"_id": "e1", "name": "Hanne Solberg", "country": "Norway",
"updated_at": "2024-02-10T09:00:00Z",
"experiences": [{"company_name": "Kestrel Grid", "career_title": "VP Operations",
"date_from": "2017-04-01", "date_till": "2022-01-01"}]},
{"_id": "e2", "name": "Tomas Ruiz", "country": "Chile",
"updated_at": "2024-02-10T09:00:00Z", "experiences": []},
]
export = "".join(json.dumps(r) + "\n" for r in records).encode()
stored = client.post("/v1/public/upload_export", data={"workspace_id": ws},
files={"file": ("experts.jsonl", export)}).json()
print(stored["format"], stored["bytes"] == len(export), stored["export_id"][:4])
source = {"export_id": stored["export_id"], "collection": "experts"}
mapping = {"key_path": "_id", "entity_path": "_id", "name_path": "name",
"as_of_path": "updated_at",
"relations": [{"relation": "held_role", "iterate": "experiences[]",
"slot_paths": {"company": "company_name", "title": "career_title"},
"period_from_path": "date_from", "period_to_path": "date_till",
"period_end": "exclusive", "missing_end": "open"},
{"relation": "based_in", "slot_paths": {"place": "country"}}]}
pulled = client.post("/v1/public/connect", json={
"workspace_id": ws, "document_source": source, "document_mapping": mapping}).json()
print(pulled["row_count"], pulled["export"]["deleted"])
found = client.post("/v1/public/search", json={"workspace_id": ws, "constraints": [
{"relation": "based_in", "slot": "place", "op": "eq", "value": "Norway"}]}).json()
print(found["entities"]["matched"])
again = client.post("/v1/public/connect", json={
"workspace_id": ws, "document_source": source, "document_mapping": mapping})
print(again.status_code, again.json()["detail"]["type"])
What a pull writes¶
A relation_mapping pull is validated WHOLE before anything is written — a value that
fails its slot's type, a duplicate or NULL key, a period_to before its period_from —
and a refusal names the row key and the column. Every row lands declared, confidence
1.0, with a {db_table, row_key, db_column} receipt per slot that is re-read from the live
table when the row is delivered. Every entity pulled gets a coverage row for the relation;
a row whose every mapped slot and period is NULL writes coverage and no row (an absence
row). coverage_columns ({"screening": "screened"}) declares another relation looked at
per entity: a true value writes its coverage row, false or NULL writes none, anything
else rejects the pull.
The receipts are redeemed in the process that registered them: a pull registers a
resolver for its source id, and a source with no registered resolver — after a restart,
until the table is connected again — delivers its slot receipts with byte_verified:
false and reason: row_unreadable (never row_changed: an unreadable row is not
evidence that it moved).
import sqlite3
from docs.examples.roster import ROLES_MAPPING, connect, export_dir, new_workspace
ws = new_workspace(client)
client.post("/v1/public/apply_pack", json={"workspace_id": ws, "bundle_id": "bundle_expert_network"})
export = export_dir() / "export.db"
with sqlite3.connect(export) as db:
db.execute("CREATE TABLE roles (row_id INTEGER PRIMARY KEY, person TEXT, company TEXT, "
"title TEXT, seniority TEXT, function TEXT, geography TEXT, period_from TEXT, "
"period_to TEXT, screened TEXT)")
db.executemany("INSERT INTO roles VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", [
(1, "piet_arnaud", "Sable Point", "Director, Sales", "Director", "Sales",
"Germany", "2018-01", "2021-01", "yes"),
(2, "rhea_sollis", "Corvid Analytics", "Director, Sales", "Director", "Sales",
"Austria", "2020-03", "2019-06", None)])
r = connect(client, ws, export, "roles", ROLES_MAPPING)
detail = r.json()["detail"]
print(r.status_code, detail["type"], "| row:", detail["row_key"], "| column:", detail["column"])
print(detail["detail"])
422 relation_export_rejected | row: {"row_id":2} | column: period_from+period_to
row {"row_id":2}, column 'period_from+period_to': rhea_sollis/held_role: period_to 2019-06 precedes period_from 2020-03
Generations and replaces¶
A connected table is a source, and each pull of it is a generation. Three cases:
- The table did not change — the same rows, datasource, table and mapping: the pull
deduplicates to the source you hold (
deduplicated: true) and writes nothing. A nightly sync of an unchanged table does not multiply facts. - The table changed and the pull names
replaces=<the current source id>— the new generation is written first and the old one retired after. Rows the new pull restates stay current (a row's identity is its content, not the source); rows it no longer states retire with the old generation andrelations.rows_retired_from_replacedcounts them.limma_generationsshows the chain,limma_diffnames each changed slot of each changed row (rows pair across generations on the export's key). - The table changed and the pull does NOT name
replaces— the routine re-sync. The pull is the next generation of the SAME export (same datasource, table and mapping), so it supersedes that export's current generation by itself, exactly as ifreplaceshad named it:supersededlists the generations it retired. (Before §10 1.5's fix this left both generations live — the old row counted beside the new one.)
replaces must name a CURRENT source of THIS workspace; anything else is a 422, and a
corrupted re-pull leaves the current generation untouched. Changing the mapping over the
same table is a different export (its identity includes the mapping): a routine re-sync
never retires it, and replaces is how you do.
from docs.examples.roster import new_roster, connect, update_export, ROLES_MAPPING
r = new_roster(client)
same = connect(client, r.workspace_id, r.export, "roles", ROLES_MAPPING).json()
print("unchanged, no replaces: deduplicated:", same["deduplicated"],
"| same source:", same["source_id"] == r.roles_source)
update_export(r.export, "DELETE FROM roles WHERE row_id = ?", (5,))
resync = connect(client, r.workspace_id, r.export, "roles", ROLES_MAPPING,
replaces=r.roles_source).json()
print("re-sync: rows_written:", resync["relations"]["rows_written"],
"| rows_retired_from_replaced:", resync["relations"]["rows_retired_from_replaced"])
chain = client.get("/v1/public/generations", params={
"workspace_id": r.workspace_id, "source_id": resync["source_id"]}).json()["generations"]
print("chain, oldest first:", [g["is_current"] for g in chain])
stale = connect(client, r.workspace_id, r.export, "roles", ROLES_MAPPING,
replaces=r.roles_source)
print("replaces a retired generation:", stale.status_code)
unchanged, no replaces: deduplicated: True | same source: True
re-sync: rows_written: 5 | rows_retired_from_replaced: 1
chain, oldest first: [False, True]
replaces a retired generation: 422
A document-store export¶
document_source loads a mongoexport FILE on the API host — one document per line, or
a JSON array, Extended JSON v2 in either mode — under a stated document_mapping. (A CSV
or an Excel export is read through the same door, one row a record: everything this
section says holds for it too — what differs is below.) The
file must lie under a directory listed in EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS; a relative
path, one that resolves outside those directories (through .. or a symlink), a
directory, a pipe, and a workspace store or a copy of one are refused, and the refusal
says nothing about which files the host holds. Keep the file where it was connected from:
a row's receipts are re-read from it when the row is delivered, and — as for a table —
only by the process that ran the connect (row_unreadable after a restart, until the
export is connected again, which writes nothing when it has not changed). A file that is
not on the API host is UPLOADED and named by export_id instead of path
(above): everything
below holds for it, and its receipts read row_unreadable from the start.
What the mapping states, and what it may not leave open:
relations[]— one entry per relation:iterate: "experiences[]"makes each element of that array a row, andslot_pathsare then relative to the element.period_end(exclusive|inclusive) andmissing_end(unknown|open_if_latest|open) are REQUIRED wherever a period is mapped and are never defaulted: either default is wrong for somebody's data.open_if_latestcompares an element with the other elements of its array, so it is for an iterated relation only: a collection of one role per document statesopenorunknown(two documents are never siblings, soopen_if_latestthere is refused rather than read asopen).mode: "propose"reads a sample and returns a mapping with those questions leftnull, each with the counts that help answer it; it writes nothing and is never applied for you.documents[]— prose fields (abio, each call'stext) registered as documents about the record's subject: stored with their text, readable throughlimma_regionand by a criterion search, and read by NO model unless the entry saysread: "queue"with amax_queued. An unchanged text is never registered twice, and a document countsregisteredonly once its text has been read back as stored. What a pull does with a document registered earlier is decided by what the pull knows of it (retired_by_reasoncounts each case):- the pull met its record tonight and the record names no usable subject any more —
a call still exported with its expert removed — the document is retired
(
subject_removed): it is no longer read for the person it was registered under; - the pull met its record and the text is gone, blank or no longer a string — it is
retired (
text_removed). For a text mapped through an array (calls[].text) that is also a call removed from the record, the array gone, and the array no longer an array: the pull read the record by its key, and it holds no such text; - the pull did not meet its RECORD at all — it is retired (
removed_upstream), but only when every document of the export could be identified. When a line could not be read, or a document has no usable key, the missing record may be that one, so nothing of it is retired on that ground (kept_unconfirmedcounts what was kept, andread_errorssays how many documents could not be identified). Only a record the pull did not read is held back so: one keyless record elsewhere in the export hides nothing about a record it did read; read: "retire"is your instruction, not an inference, and retires every document of its path whatever the pull could read (retired_by_mapping).
- the pull met its record tonight and the record names no usable subject any more —
a call still exported with its expert removed — the document is retired
(
A pull whose mapping names no documents entry for a path never reads that path, so
it never retires what an earlier pull registered from it — and says so in
read_errors on every pull ("… are not maintained by this pull"), so a job that lost
its documents block cannot keep serving old texts unnoticed. The documents register
after the rows are written, so a failure there never fails the pull: it is said in
read_errors, and sending the pull again registers the rest.
* The subject need not be the key. In a collection of calls each document has its own
_id and names its expert in another field: entity_path: "expert_id" files every
call under that expert, as_of_path on the entry dates it by the call rather than by
the record, and turns states how the transcript writes who is speaking (a pattern
with a label group, and the labels that are not the subject) — never inferred. A
proposal asks which field the subject is — "entity_path": null, the key and the
other field as candidates — wherever an id-named field (its name ends in id, uid,
uuid, guid, key or ref: expert_id, personId, expertRef) that at least
nine in ten sampled records hold REPEATS across them (counts: how many hold it, how
many values it takes); where none does, the subject is proposed as the key. The rule
reads the last word of the name: a subject kept under a name with no such word
(expert, speaker, expertid) is not asked about — state entity_path yourself.
Which field it is stays your statement, and the data cannot check it: the first pull
of a collection under a subject that is not its key says in read_errors how many
documents it read and how many distinct subjects they name. A collection of people
that carries an account manager's id, answered with that id, reads as that many
account managers — the sentence is where you see it.
* The key is the export's own _id wherever every
sampled record holds it and no two share it. Where some sampled record does not —
one record without an _id — the key is ASKED ("key_path": null), never replaced
by another field: the export's key is a candidate when it is unique among the records
that hold it (counts: how many hold it), and a pull keyed on it skips and counts
the records without one (key_missing). A display name (name, full_name) is a
candidate and never the proposal's own choice: a name unique in a sample says nothing
of the collection, two people of one name refuse the pull, and a renamed person is a
new one. Only where no sampled record holds the stated key and exactly one other
field is unique in all of them — and is no display name — is that field proposed.
* What an array that holds no row says is YOUR statement, per entry, and each has its
own key: empty_means: "none" — an EMPTY array says "there are none";
absent_means: "none" — so does a record with NO such array (no key, or a null: a
store that writes no key where it would write an empty array). Know what the second
one costs: a record that LOSES the key on some night — a projection that dropped it —
then reads as having none, and the pull cannot tell that from a list that was never
written; on the night somebody who HELD A ROW in the generation before reads "none"
for want of the array, read_errors says so — how many people, with the counts of
both nights (it follows who moved, not two totals: new people arriving the same night
hide nothing). The sentence is for somebody who holds NO row of the relation tonight:
a person who keeps a row from another array, or from another of their documents, and
loses the rest is not named — where that would matter, leave the entry at
not_recorded, which withholds. Left at
not_recorded, the default of both, the array has said nothing. A relation may be
read from more than one PLACE (the current role in one array, the earlier ones in
another; an array and the record's own fields), and then EACH PLACE ANSWERS FOR
ITSELF: a person is none for the relation only where every place gave them a row
or reads "none" by your statement. Somebody with a current role and an empty, or
missing, earlier-roles array left at not_recorded keeps the row they have,
matches on it, and is undetermined — never none — for anything else about the
relation; an answer that leaves them unmatched is not licensed complete
(withheld_rows_unanswerable), relations.by_relation.<relation>.entities_place_silent
counts them, and read_errors says which place said nothing, for how many, and how
it found them. The record itself is one place, however many of its fields are read:
it has spoken for a person when one of them held a value, and where it holds none of
them it has said nothing — no convention makes a missing field "none". Two entries
over ONE array (below) are two readings of one place: it has spoken when either
stated a row, and reads "none" only when both do. A subject read from SEVERAL
documents (a collection of roles or calls) is judged across them, and there a
statement outranks a silence: somebody whose one document states her roles and whose
other holds no such array is complete for the relation by the first.
* EMPTY means the array HOLDS NOTHING: [], or only elements that hold nothing ([{}],
nulls, blanks). An element that holds a value and gives A RELATION nothing — no slot,
no period, at any entry of that relation that reads its array: a role written under
other keys, a reference, a note — was NOT READ for that relation, and that is never
"empty". This is judged RELATION BY RELATION: where two relations are read from one
array (the employer of a role, and the country it was held in), an element only the
second reads is still unread for the first. It is quarantined
(element_unread: its key and path, never its value); its person keeps the rows that
were read, is undetermined — never none — for what that element would have
stated, and no answer that leaves them unmatched is licensed complete.
relations.by_relation.<relation>.elements_unread counts such elements, and
read_errors says how many, in which array, for how many people. The pull cannot
tell a role under an older schema's keys from a note that states no role, and does
not guess: if they are rows, add an entry that reads the same array at the keys they
carry. If they are NOT rows — blank subdocuments that carry only an id, stamps,
notes; or a relation only SOME elements state (a language few roles carry) — that
is your statement too, per entry: unread_means: "nothing" (the default is
withheld) counts them as empty elements. An element that gives the entry its
PERIOD and no slot value — a role with dates and no language, for a relation read
with the roles' dates — is a fault of its own WHATEVER the entry states
(tenure_of_nothing): a period is evidence of a row, and a role under a renamed
employer key whose date stands under the mapped one must never read as "no row". So a
relation only some elements state is read WITHOUT the elements' dates — remove the
entry's period paths, and its period_end / missing_end with them (its rows are
then undated), and state nothing; a pull refused over such elements says so. A proposal asks — "unread_means": null, with how many
sampled elements give the entry no slot value and how many of those carry its
period — wherever one is. Know what it
costs: stated nothing, a role written under other keys reads as no role. The pull
cannot tell you which it was; it counts such elements
(relations.by_relation.<relation>.elements_read_as_nothing) and says so in
read_errors on the first pull of a collection and on every pull that read more of
them than the generation that answers was written with — the night their number
grows, also when that pull changed no mapped value and wrote nothing (it is then
said again each night until a generation is written). Where
several entries of a relation read the array, all must state it. Likewise a path to
the array that runs into something that is not an object
("profile": "n/a" under profile.experiences[]) is quarantined
(path_not_an_object) — something is there; it is not "no such array". Both count
toward max_quarantine_rate (below): an entry whose paths the elements do not carry
refuses the pull instead of loading nothing.
What a pull does with a value it cannot read. It is QUARANTINED — named in
quarantined[] with what it cost — and never guessed. A role the loader could not read
leaves its person undetermined for every question that role could have answered,
never none, and no answer over them is licensed complete; a person whose record says
nothing for a relation is known and not_recorded. A bare year is not a month: the row
is kept undated, and a question asked at a month reads it undetermined. Past a
threshold (at least 3 faults and more than max_quarantine_rate, 5% by default, of any
one population) the whole pull is refused: that many faults is a wrong mapping.
A string the store cannot hold is one more value it cannot read — by ONE rule on SQLite
and Postgres, so the deployment's database never decides what loads. Three kinds: a
string holding the character U+0000 (a document store keeps one; Postgres text cannot);
one that is not valid Unicode (a lone surrogate — what a JavaScript writer emits for a
string cut inside a character); and, where the store indexes it, one longer than 2,000
bytes of UTF-8. At a slot it costs that slot (value_unstorable, value_too_long): the
row keeps its other slots, and a question about the dropped one reads the person
undetermined. In a key or a subject id it costs that document (key_unreadable,
entity_unreadable, the reason in detail). In a bio or a call's text the document is
not registered (text_unstorable) and the one registered earlier for that record stays
— while tonight's record still names the same subject: a call the export now gives to
somebody else is retired from the person it was registered under (subject_removed),
whether or not its new text could be stored.
A slot value past the bound is prose — map the field under documents[]. A SQL table
pulled as relation rows (relation_mapping) is held to the same rule with its own
consequence: the pull is refused, naming the row and the column. And a question that
names a value holding U+0000 or broken Unicode (a relation_query, a search constraint)
is a 422 on both: no row can hold that value, so nothing can be asked of it.
What the reply reports. relations.by_relation counts, per relation, the elements read
(once per entry that reads them), the rows written, the elements that gave an entry
nothing and hold nothing — or were read by another entry of the same array —
(empty_elements), the elements that hold something and that NO entry of their array
read (elements_unread: each is in quarantined[]), who is covered, withheld or
silent, for how many subjects with rows another place the relation is read from said
nothing (entities_place_silent), and how many rows were kept
undated and why; documents counts, per mapped text path, what was registered,
unchanged, retired or deferred; names counts the display names the pull read at
name_path — {written, removed, unchanged}, one per subject, for every record that
names its subject whether or not it holds a row or a text (a subject two records name
differently has none). quarantined[] lists the first 500 faults, sorted by
key and then path, and quarantined_total, quarantined_by_reason and
quarantined_by_consequence count them all. A quarantined value is quoted, cut to 200
characters (a list or an object shows only its type, and a document's text is never
quoted): it is your data — in the reply, on the generation's own record and, where the
row is kept, in the row's stored receipt — and all of it is erased with the generation.
detail is the reader's own words about the value and never shows more of it than
value does. Fields the mapping does not name are read past and stored nowhere.
What a proposal shows. mode: "propose" returns field names, types and counts, and no
VALUE of the export unless you ask. With sample_values: true it also returns up to ten
values per field, and ONLY values this workspace's own vocabulary resolves (a country, a
seniority level — a spelling a pack already knows) on a field that vocabulary
recognises, or a boolean word. Every other field says why it shows none
(samples_withheld: not_in_a_vocabulary, not_text, proposed_as_name_path, …).
Repetition is not the rule: five interviewers named on three hundred calls are five
names, however often each repeats.
A field NAME can be somebody's data too — a map keyed by e-mail address, by date, by a
reviewer's name. So a key of the export is listed as a field only on evidence that it is
one: it reads as an identifier (ASCII letters, digits and underscores, not starting with
a digit, at most 64 characters), and at least two sampled documents carry it at that
path. Inside an object or an array the documents in which it holds a key decide the
rest (an empty object says nothing, as an absent one). The object is keyed by data, and
none of its keys is listed, when more than half of them
carry a key that does not read so — or, for an object of more than twenty keys most of
which a single document carries, a key no other document carries. Otherwise the keys
more than half of them carry are its shared keys, and are listed. A key FEWER of them
carry is listed only in a record with a few optional fields: the object has shared
keys, every key of it reads as a field name, and its other keys — all of them — are no
more than the shared ones. That is the end date of a role, which only a closed interval
has, and a field written in two spellings over the years; it is not thirty reviewers
keying ratings, each on a few records. An array of records is judged by the same rule,
by its records: one stray key in one role does not make the roles a map — their fields
are listed and the stray key is counted. Whatever is not listed is counted under the
one segment * — contacts.*, or contacts.*[] where it holds an array; unmapped[]
says why and how many (names_not_listed, the object's count) — and nothing is
proposed from a path through it. So an address, a link, a number, a date or a name
written in two words never comes back as a path; neither does a real field called
First Name, one only a single sampled document carries (a one-document export lists
nothing), nor a key few records carry in an object nobody shares a key of, beside a key
that does not read as a field name, or among more such keys than shared ones: state
such a path yourself in the mapping.
A mapping reads ONE level of array and names no key that is data, and the proposal
reads no further than a mapping can. So two shapes are containers whose contents it
never sees: an array of objects INSIDE an array's elements (groups[].positions[]), and
objects under keys that are not listed (a map of records keyed by id, roles: {"r17":
{…}}; a top-level key written Work History). Rows held there can be read by NO
mapping: reshape the export so they stand in a first-level array of objects ($unwind
the outer array, $objectToArray the map) and propose again. evidence.containers_not_read
lists each such container two or more sampled documents hold (what one sampled record
alone holds is not listed), by a path that names no key. An object nested deeper than
the eight levels read for field names is judged by the same evidence: keys the records
share are fields, and only objects under keys no two records share are a map.
VALUES one level below what a path reads are not read either: a list of values inside
an array's elements (groups[].company_history: ["A", "B"]), or values under keys that
are data (company_history: {"2015": "A"}). At the record's first level a list of that
name is offered as a source of rows; one level down no mapping can read it. So where
such a container's OWN KEY is evidence for the PARTY a relation's rows name (an
employer) — by the rule a first-level field of that name is held to — that relation
carries the null of the first case below, unresolved names the container, and
evidence.values_not_read lists it with what its name is evidence for. A container
whose name is evidence only for ANOTHER slot (the industries a role is tagged with)
asks nothing — at the first level such a list is not offered as rows either (below) —
and one whose name is evidence of nothing asks nothing, there as at the first level:
unmapped[] says of each that its contents were not read, and why nothing is asked.
A PROPOSAL IS A DRAFT, AND A DRAFT IS NEVER LOADED AS IT CAME. A proposal is a guess
from a sample, and a guess can always be shown a file it guesses wrong: an end date in
a field called left_on, a parent_company beside the employer, a list of tags that
shares one value with the employers. So every relation entry a proposal writes carries
"confirmed": null, and a pull refuses the mapping while any entry does
(relations[<n>].confirmed is null). For each relation unresolved holds one question
at relations[<relation>].confirmed; its not_read lists every field of the entry's
own place that no key of the relation's entries reads and that could be a row's date
or its party — how many of its sampled values read as dates or as timelines, how many
are objects or lists, and which of the relation's slots its name holds. A date left
unread is a start or an end nobody reads; a party left unread is a row nobody finds.
Answer the entry's other nulls, check the paths it reads against your data, read
not_read, correct the entry, then set "confirmed": true — or remove the key: the
entry is then your own statement. The mark is not part of what a mapping says: a
confirmed draft is kept, and compared on the next pull, as the mapping you stated. An
entry that is only a question about another place (nothing but "iterate": null)
carries no mark; it is stated or removed, as below.
What the proposal could not decide — or knows it did not read — is a null in the
mapping, never an omission you could send back without noticing. Seven cases — and
their limits: a proposal reads a SAMPLE (sampled of scanned; the counts in a
question are the sample's), so a shape only unsampled records hold is not seen at all,
and a field whose name is evidence for no slot is offered to no relation (state it
yourself). A pull says what it could not read; it does not look for what a mapping
never names.
- A relation is read from an array some of whose keys were not listed.
evidence.relations.<relation>.keys_not_listedsays how many. Where the proposal found a start date there and NO end, the entry carries"period_to_path": nullandunresolvedasks: the end may be one of those keys (a pull that read a start and no end would count every role as still held). State the end's path andperiod_end, or remove the key if the rows carry no end date. Themissing_endquestion then counts only the elements — how many have no end is not known until the end's path is. The samenullstands where two paths tie for the end (the end key in two spellings) — and there either answer has a cost the question states with its counts: one entry reads one end path, so the roles written under the other spelling have NO end under the mapping, andmissing_end: openwould read the ended ones as still held — as wouldopen_if_latesta person's latest role, where its end stands at the other path. Choosemissing_end: unknown, or write the end under one key in the export. - A relation's period stands at a path NOT EVERY sampled value of which reads, or at
more than one path. THE PROPOSAL CHOOSES A PERIOD PATH ONLY WHERE IT IS THE ONE
CANDIDATE FOR ITS KEY AND EVERY SAMPLED VALUE IT HOLDS READS — by the pull's own date
reader: a word for nothing recorded (
n/a,-,null) and an EMPTY object or list state nothing, here as at the pull, which reads each as a MISSING date; a bare number from 1000 to 9999 is a year; an object or a list that holds something is a value that is no date. Everything else is anullthe pull refuses, asked inunresolvedunder the key it stands at, with every candidate and its counts (counts.<path>: of the values that state something, how many read and how many do not —objects_or_listsamong them). The same counts stand inevidence.relations.<relation>.period, andunmapped[]names each candidate that was not chosen. The cases. ONE value that is no date among many:"period_to_path": null— state the path, withperiod_end, and the pull sets each such element aside (date_unreadable) with its person, counted towardmax_quarantine_rate(the question works the rate out for the sample); or remove the key if the rows carry no end, andmissing_endthen says what every row's missing end means. TWO paths named for the same end, however clean either is: the one every value of which reads is listed first. The ONLY path named for an end, no value of which reads. And A START PATH BESIDE A PATH OF TIMELINE STRINGS ("Jan 2010 - Jan 2012"): a period is stated as two dates or as one timeline, never both, so every period key that has a candidate isnull, and you state the paths of ONE way and remove the keys of the other. A start that is asked says what removing it takes with it: an end with no start is no interval, so the end's keys go too, and the rows are then undated. The conventions (period_end,missing_end) are asked only beside a path the proposal did choose; state them with a path you state yourself. (A path named for an end is judged as that end: a timeline written into it is one more value that is no date. A path named for a START every value of which reads as a timeline or a date is the period's text — a plain date there is a timeline with no end.) - An array of objects was not fully read — every role carries a field written
job titleand the array is keyed by data, or some of its keys were not listed — and a relation whose rows are its elements is proposed from the record's own fields, or from none. This is asked of EACH relation that is not read from the array, whichever other relation is:evidence.relations.<relation>.arrays_not_readnames the array,unresolvedasks underrelations[].iteratewhich of the other relations' rows its elements are (and says which relation already reads it), and every relation proposed from the document's own fields carries"iterate": null: stateiterateand the element's paths yourself, or remove the key where the document itself states the row (a person's country, beside an array of their roles). The samenullmarks a relation read from the document's own fields while an array that also gives evidence for it was ranked second (evidence.relations.<relation>.alternatives). That one null is answered with ONE array: where the question names two, the mapping holds an entry of its own (below) for each array beyond the first — and where the document AND an array both hold the rows, removeiterateand state an entry for the array. - A container the proposal did not read (above) is named in every proposed relation's
evidence (
containers_not_read) and asked about underrelations[].iterate, and every relation proposed beside it carries a null:"iterate": nullon a relation read from the document's own fields (remove the key once you have read what that leaves out — nothing can be stated for it), an entry of its own (below) on a relation read from an array. - A relation is read from ONE place while ANOTHER may hold more of its rows. A second
array — a person's earlier roles — that gives evidence for the relation and was ranked
second, or that the relation is not read from and not all of whose keys were listed.
The record's OWN FIELDS, ranked second to an array: a
current_companybeside the roles may name an employer no element holds. Or the relation has no entry of the document's own at all (its only evidence there is an array of values) beside an array that was not fully read. The mapping then holds one MORE entry for the relation per such place,{"relation": "held_role", "iterate": null, "slot_paths": {}}, andunresolvedasks underrelations[held_role].iterate, naming each — withsources: the paths that give evidence there andcountsover thesampledrecords — for an array how many hold it empty or not at all; for the record's own fields how many hold NONE of them, and how many name there a value no element of the array holds (documents_naming_a_value_no_element_holds: where that is 0, those fields only repeat what an element states, and the answer is to remove the entry). Where the party stands in SEVERAL places of the record itself — a list of names, or fields that name different employers in one record (below) —sourceslists theentriesthat read each: state one for each, in place of the one entry asked about. A relation may be read from several places, one entry each: for each, state the entry — an array asiteratewith the element's paths, its period and its conventions (an earlier-roles array carries an end the current-roles array does not: do not copy the first entry's); the record's own fields with noiterateand each path from the document's top — or remove the ENTRY: the relation is then not read from that place, and anybody whose row stands only there is not found by it. A row read from the record's own fields carries no period unless you map one (it answers "ever", never "as of"), and a person whose record holds none of those fields is then nevernonefor the relation (above) — ONE such record is enough to withhold the licence of every answer that leaves that person unmatched, for as long as the field is blank: no convention says what a missing field means. Removing only the key leaves an entry that maps no slot, which is refused too; so is one entry left unanswered beside another that was stated. - Where sampled records hold the array a relation is read from — an array of objects
or a list of values — EMPTY, or NOT AT ALL, the
entry carries
"empty_means": null/"absent_means": nullwith the count: what either says is your statement (above). - Where sampled ELEMENTS of that array hold values and none at the entry's SLOT paths —
dated or not — the entry carries
"unread_means": nullwith the counts (elements, of how many, in how many records, how many of them carry the entry's period — those stay faults under either answer: remove the entry's period paths, and itsperiod_end/missing_end, where the relation is stated by some elements only): rows written another way (withheld), or no rows of this relation (nothing) — above. Two relations proposed over one array are each asked for themselves.
Two entries may not read the same thing. An entry that reads nothing another does not —
the same relation from the same place, no period path the other does not read, and
every slot it maps mapped there from the same path — is refused: stated twice (or with
the period left out) it reads nothing more,
and under ANOTHER convention (missing_end: unknown where the first says open) it
would state every element a second time, the looser reading winning. Two entries of one
relation over one array that read a slot from DIFFERENT keys — an employer written
under two spellings — stand, and here is what they yield: with the title and the period
mapped in both, every element is also a row with no employer under the entry whose key
it lacks, nobody is missed, and no question over the employer is licensed
(slot_unrecorded_rows_unanswerable); with the employer alone in both, each element is
one row, "who ever worked at C" is licensed, and nothing is dated. Write the field
under one key in the export if you need both. A refusal of two such entries adds that a
mapping which already loaded is, once corrected, a CHANGED mapping (replaces).
THE PARTY IN SEVERAL PLACES OF THE RECORD ITSELF. A field is evidence for the party
only where it is NAMED FOR it: the slot's name is the field's last word (company,
current_company) or is followed only by a word that says how the thing is designated
(company_name, company_id). A field that only starts with the name is an attribute
of the party — company_size, company_website, company_email — and is evidence for
nothing: unmapped[] lists it. Fields named for the party that name DIFFERENT values
in one record (current_company, previous_company) are each read, by an entry of its
own, one row a record; nothing is asked, and whichever ONE a caller named, the other's
people would read "none" under a licence. A LIST of values named for the party is read
the same way, one row an element. A list whose name only MENTIONS the party
(company_history: [..], company_tags: [..]) cannot be told apart by its name. Alone,
it is proposed as rows. Beside a field named for the party it is read by an entry of
its own only where it names values that field names somewhere in the sample (a history
of employers does; tags do not); otherwise the mapping holds an entry of nothing but
"iterate": null for it and unresolved asks, with the entry sketched (entries),
counts, per place how many sampled records name there a value no other place holds
(places), and how many hold the list empty or not at all (lists): state the entry
or remove it. BESIDE AN ARRAY THE MAPPING ASKS ABOUT (above) NOTHING OF THE RECORD IS
READ UNASKED: the array's elements may be the relation's rows, and a field or a list
of the record then repeats them — read beside the array, each such row is stated
twice. The record's fields are then one question ("slot_paths": {"<slot>": null},
the entries sketched) and each list an entry of nothing but the null. Where several
fields are each read, a field of the whole record that is evidence for another slot (a
country) is put in none of those entries — it is not a fact about one of several
rows — and unmapped[] says so. What is still a tie, and asked under
relations[<relation>].slot_paths.<slot>: several fields at which no sampled record
names different values (one field under two spellings — a record written the other way
is then silent for the relation, never "none"); an id-named field beside its name
(company_name, company_id: ONE party written twice — the name is listed first;
read both and every row is stated twice under two ids), while a third field beside
them that names another employer is read by its own entry; and a path one sampled
value of which is an object or a list (the pull sets each aside, slot_type). The
question carries counts — how many sampled records hold a value at more than one of
the paths, and how many of those name different values there — and places, and says
the third answer: where several paths each hold a row of their own, answer with one
and state an entry of its own for each other. Two nulls that stand at one key
(relations[<relation>].iterate: an array that was not read, and a list that is asked
about) are ONE question there, the second said after "ALSO AT THIS KEY".
An entry over an array of VALUES ("slot_paths": {"company": "$"}) writes one row an
element holding that ONE slot. For the party a row names — a list of employers — that
is the row. For any other slot it is a row at NO party: a person holding one is
undetermined for every question over the party they do not match, and no such answer
is licensed complete. A proposal therefore offers an array of values as rows only for
the party; another slot's is named in unmapped[] with the entry to state if its
values are rows of their own. The same holds for the record's OWN FIELDS beside the
party's array: with the employers in company_history: [..], a seniority_level on
the record is not proposed as a second entry — it would write one more row a record,
at no employer — and unmapped[] says so. A pull under a mapping that states such an
entry says so in read_errors, naming the entry.
An entry that read nothing in the whole export, four cases. (1) ITS ARRAY IS IN NO
DOCUMENT — a misspelt iterate, or an array this export does not have; the pull cannot
tell a mistake from an export that simply holds none. It is not refused: read_errors
names the entry, and while that generation is current no question over the relation is
licensed complete (unattributed_rows_withheld) — whatever the entry maps (an array of
values is a row entry too) and whatever its conventions say (stated absent_means:
none, a misspelling would otherwise read "there are none" for everybody). Correct the
entry or remove it. One record that holds the array is enough for it to have been found:
an array few records carry is ordinary, and what the others then say is absent_means.
(2) THE ARRAY IS THERE AND ITS ELEMENTS CARRY NONE OF THE ENTRY'S KEYS — the wrong array,
or a renamed key. Every element that holds something is then an element not read
(element_unread, above): past the stated rate the pull is REFUSED, naming the reason,
and under it each such person is withheld. (3) ANOTHER ENTRY OF THE SAME RELATION OVER
THAT ARRAY WROTE ROWS AND THIS ONE WROTE NONE — the second spelling of a key, the night
its last record is migrated. read_errors names the entry as one that reads nothing any
more, to remove, and says how many of the elements it passed over the other reading
read: nothing is withheld for the entry. An element NO reading of the array read is not
excused by it — that is an element not read (case 2: set aside with its person), or
nothing where every reading states so. (Another RELATION's entry reading
the elements is not this case: for this relation they are unread — case 2.) (4) THE
ENTRY STATES unread_means: "nothing" AND READS NOTHING ANYWHERE — its statement makes
every element it cannot read "no row", so no element is a fault and nothing would be
said. The pull says it instead (read_errors: a misspelt key hides every row behind
that statement) and, while that generation is current, no question over the relation
is licensed complete (unattributed_rows_withheld) — whatever else in the export was
a fault: the question is whether the ENTRY ever wrote a row. An array that is there and holds
nothing was read: empty_means says what that states, place by place.
One question has no null: where two paths tie for a relation's START date the entry
carries no period at all — its rows are then undated, which claims nothing and licenses
no "as of" answer — and unresolved asks which path holds it.
An explicit null at iterate or at a period path is refused wherever the mapping
enters, naming the entry; an absent key is simply "none". A tie between two paths is
never broken for you: its candidates name each path from the document's top, and
write_as gives the same choices as a mapping writes them, relative to the element.
What the rule cannot tell apart, and says in every proposal
(conventions.field_names): a one-word key that most of an object's records share —
the same few reviewers' surnames, usernames or staff ids on every record — reads exactly
as a field name, and is listed as one; and so, in an object that reads as a record, are
up to as many other one-word keys as it has shared ones, each on two records or more.
Where a collection keys objects by such words, read fields[] before you hand a
proposal on.
A pull is never cut to fit: an export of more than 200,000 documents, or one stating
more than 500,000 rows of a single relation, is refused whole — load it in parts, each
under its own collection label.
The line of generations. One collection label holds one line — tonight's file under a
new name is the next generation of last night's. Two guards protect it, each a 422 that
writes nothing: a pull under a CHANGED mapping (mapping_changed, naming the current
generation and each changed field — an entry by its relation and place,
relations[held_role @ experiences[]], a second reading of the same array as
… #2, in the order written) unless replaces names that generation; and a pull
holding far fewer documents than the generation it would retire (export_shrank: at
least 3 fewer and more than max_shrink_rate, 10% by default) unless allow_shrink —
what a truncated nightly file, or one shard loaded under the label of the whole
collection, looks like. The second guard also counts each mapped documents path, by
record and OVER THE RECORDS TONIGHT'S EXPORT HOLDS: when more than that share of the
records that are in the export and hold a document registered from bio carry no text
there now — tonight's job dropped the field — the pull is refused the same way
(counts.tripped: "texts", counts.path), before a single document is retired. A person
who is no longer in the export has left, not lost a field: that is the first count's
business, and their documents are retired (removed_upstream) or kept unconfirmed. A
record that still holds its text and names no usable subject for it (a documents entry
with its own subject_path, on a night the id is missing) counts the same way — its
document would be retired too — and the refusal says how many of each. One person's
texts leaving is one record, and is simply retired. The count is by record: an array of
texts CUT inside every record (each person's calls[] down to its first element) leaves
every record holding a text and is not refused — the calls that left are retired
(text_removed), and registered as new documents if they come back.
One export, one pull at a time. A pull holds its line — the workspace and the
collection label; for a table, the datasource and the table — from its first read to
the last document it registers. A second request for the same line WAITS
(EPISTEM_CONNECT_LINE_WAIT_S, 120 seconds by default) and, if the first is still
running then, is refused 409 pull_in_flight having written nothing: ask again when the
first has answered, and an unchanged export is then deduplicated. A retry sent because
a proxy gave up on a long pull is therefore harmless to the data — it writes nothing —
and the API process it lands on goes on answering while it waits (a pull, of a table as
of a document export, runs off the request loop). Two collections of one workspace
load side by side. The wait is kept by the API process and, on Postgres, by the database
as well, so several API processes over one Postgres store wait for each other; a SQLite
store is served by one process. (Where two pulls of one export do meet without waiting,
the one that finds the line changed under it writes nothing and is refused 409
pull_superseded.)
How many pulls at once. A running pull holds two of the store's database connections —
its line's lock and its writer — for as long as it runs. The store's pool is 5
connections with 10 more on demand (DB_POOL_SIZE, DB_MAX_OVERFLOW), so from about
seven simultaneous pulls in one API process the searches beside them start to wait, and
well beyond that a request that finds no connection free within the pool's wait (thirty
seconds) is refused 503 having read and written nothing — store_write_failed on a
pull of a document export or of a table with relation_mapping, store_unavailable
where a request could not even look its workspace up. Send it again. (A pull of a table
with mapping — numeric axes — writes its facts one by one, so the same shortage
part-way through it is a server error, and sending it again converges.) One to three
collections a night never meet this; a bulk load of many collections should run a few
at a time, or with a larger pool. The two numbers together must allow at least 3
connections — a pull's two and one for a read beside it — or the store refuses to start
and says so: the worker, and an API started in production or on-prem mode, exit at boot
with that sentence; an API in any other mode (development, staging) writes it to its
log, starts without a store and answers every request 503 service_unavailable until
the numbers are raised.
DB_POOL_SIZE=0 means no limit, and the database's own max_connections
is then the bound. The line's lock is a Postgres session lock: give the store a direct
connection or a session-pooling proxy — behind a transaction-pooling proxy the lock and
its release can run on different sessions.
A line is the generations the connect verb wrote under a collection label. A text
uploaded under the same name (docexport:experts) is a document of its own: it is no
part of the line and no pull is refused or retired because of it.
A generation of relation rows is one commit — for a document export and for a table
pulled with relation_mapping. Its rows and the retirement of the generation before it
are written together: a reader sees last night's export or tonight's, never both and
never neither. A pull that dies part-way — a deploy, a lost database connection (503
store_write_failed when the API is still there to say so) — leaves the current
generation answering exactly as before. At worst it leaves a generation row that was
never written: it holds nothing, refuses no mapping, and the next pull of the export
retires it (it is named in that pull's superseded). Send the pull again.
An unchanged export writes nothing — no row and no receipt is rewritten — and is
answered from the generation that already holds it. (One exception, once: after an
upgrade that changes how a pull DECIDES what it read — a relation read from more than
one place; an element or a path that used to read as "empty" or "no such array" — the
first pull of an export the change touches writes a new generation under the rule as it
now is, and its read_errors say what that rule found. The pull after it writes
nothing again. Each generation records the version of that rule it was written under,
so the exception is taken once and said only where the generation it rewrites was
written under an older one.) The data's identity covers
everything a row and a receipt carry AND what the pull set aside as not read (which
element, at which path, for which relation — never a value): an export in which a
blank element begins to hold something where no entry reads — a role written under an
older schema's keys — is a new generation although no mapped value changed, because
what the pull set aside did. It covers the TYPE of the key too: the same documents
re-exported with their ObjectIds as plain strings are a new generation, because a
receipt names its document by a typed key. What a pull may still change on an unchanged
export is what the line's identity does not cover — documents, name_path and the two
rates. The pull does what they say (a bio path added later registers every bio; a
person renamed in an otherwise unchanged export answers the new name, and names counts
it written on a deduplicated pull), and
the generation records the mapping it was LAST pulled under: that is what a proposal
returns as current_mapping.
Rows a retired generation still holds. A store written by an earlier version can hold
rows that are still current under a generation that was already retired — of a document
export, or of a table pulled with relation_mapping: two pulls of one export sent at
once could each retire the other. No re-sync retired them, so a count could include a
person the export no longer holds. Every pull of a line now looks under
its retired generations and retires what it finds — in its own commit when it writes,
on its way out when the export is unchanged — and says so: relations.stranded_retired
(generations, statements, coverage, gaps, rows) and a line in read_errors. A
healthy line never shows either. If you see it once after an upgrade, a count over that
collection taken before this pull may have been too high.
The pull knows a registered document by where it came from — the collection, the
record's key and the field — not by its words. A workspace that already ingested the
same profiles as files therefore holds each of them twice once the export's bio is
connected, and a criterion search reads both: load a directory one way.
import json, os
from docs.examples.roster import export_dir, new_workspace
ws = new_workspace(client)
client.post("/v1/public/apply_pack", json={"workspace_id": ws, "bundle_id": "bundle_expert_network"})
export = export_dir() / "experts.jsonl"
role = lambda company, title, start, end: {
"company_name": company, "career_title": title, "date_from": start, "date_till": end}
export.write_text("".join(json.dumps(person) + "\n" for person in [
{"_id": "ansel_veck", "country": "Norway", "updated_at": "2024-02-10T09:00:00Z",
"experiences": [role("Kestrel Pumps AG", "Director", "2016-03-01", "2019-01-01"),
["a role this export wrote as a list"],
role("Lumen Ridge", "Vice President", "2019-02-01", None)]},
{"_id": "britta_holm", "country": "Chile", "updated_at": "2024-03-01T09:00:00Z",
"experiences": [role("Lumen Ridge", "Analyst", "2020-05-01", "2024")]},
{"_id": "caro_lindt", "country": None, "updated_at": "2024-03-01T09:00:00Z",
"experiences": []},
{"_id": "dag_orrin", "country": "Peru", "updated_at": "2024-03-01T09:00:00Z",
"experiences": ["his one role, written as a sentence"]}]), encoding="utf-8")
def connect(**body):
r = client.post("/v1/public/connect", json={
"workspace_id": ws, "document_source": {"path": str(export), "collection": "experts"},
**body})
return r.status_code, r.json()
def search(company, **more):
body = client.post("/v1/public/search", json={"workspace_id": ws, "constraints": [
{"relation": "held_role", "slot": "company", "op": "eq", "value": company}],
**more}).json()
return body["entities"], body["coverage"].get("narrowed_by")
def roles(person):
body = client.post("/v1/public/facts", json={
"workspace_id": ws, "question": "roles", "mode": "enumerate",
"relation_query": {"relation": "held_role", "entity_id": person}}).json()
verdict = body["relations"]["absence"]["entity"]
return (body["relations"]["count"], "| licensed:", body["coverage"]["completeness_licensed"],
body["coverage"]["withheld_because"], body["enumeration"].get("narrowed_by"),
"| absence:", verdict and (verdict["kind"], [gap["reason"] for gap in verdict["withheld"]]))
mapping = {"key_path": "_id", "entity_path": "_id", "as_of_path": "updated_at", "relations": [
{"relation": "held_role", "iterate": "experiences[]",
"slot_paths": {"company": "company_name", "title": "career_title"},
"period_from_path": "date_from", "period_to_path": "date_till",
"period_end": "exclusive", "missing_end": "open"},
{"relation": "based_in", "slot_paths": {"place": "country"}}]}
status, body = connect(document_mapping=mapping)
print("-- nothing configured:", status, "|", body["detail"].split(". ")[0])
os.environ["EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS"] = str(export.parent)
try:
status, body = connect(mode="propose", sample_size=10)
print("-- propose asks:", [u["field"] for u in body["proposal"]["unresolved"]])
status, body = connect(document_mapping=mapping)
held = body["relations"]["by_relation"]["held_role"]
print("-- pull:", status, "| role rows:", held["rows"], "| kept undated:", held["undated"],
"| gaps:", body["relations"]["gaps"])
print("quarantined:", [(q["key"], q["path"], q["reason"], q["consequence"])
for q in body["quarantined"]])
print("at a company nobody is recorded at:", *search("Sable Point"))
print("at Lumen Ridge in June 2024:", *search("Lumen Ridge", as_of="2024-06"))
print("limma_facts, Ansel's roles:", *roles("ansel_veck"))
print("limma_facts, Dag's roles:", *roles("dag_orrin"))
changed = dict(mapping, relations=[dict(mapping["relations"][0], period_end="inclusive"),
mapping["relations"][1]])
status, body = connect(document_mapping=changed)
print("-- a changed mapping:", status, body["detail"]["type"],
[(c["field"], c["current"], c["sent"]) for c in body["detail"]["counts"]["changed"]])
finally:
os.environ.pop("EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS", None)
-- nothing configured: 422 | export-file sources are not enabled on this deployment
-- propose asks: ['relations[held_role].period_end', 'relations[held_role].missing_end', 'relations[held_role].empty_means', 'relations[based_in].confirmed', 'relations[held_role].confirmed']
-- pull: 200 | role rows: 3 | kept undated: {'year_only': 1} | gaps: {'withheld': 2, 'silent': 2, 'unattributed_documents': 0}
quarantined: [('ansel_veck', 'experiences.1', 'element_not_an_object', 'row_withheld'), ('dag_orrin', 'experiences.0', 'element_not_an_object', 'row_withheld')]
at a company nobody is recorded at: {'matched': [], 'none': ['britta_holm'], 'not_recorded': ['caro_lindt'], 'undetermined': ['ansel_veck', 'dag_orrin']} ['entities_not_recorded', 'withheld_rows_unanswerable:held_role']
at Lumen Ridge in June 2024: {'matched': ['ansel_veck'], 'none': [], 'not_recorded': ['caro_lindt'], 'undetermined': ['britta_holm', 'dag_orrin']} ['entities_not_recorded', 'undated_rows_unanswerable:held_role', 'withheld_rows_unanswerable:held_role']
limma_facts, Ansel's roles: 2 | licensed: False narrowed ['withheld_rows'] | absence: None
limma_facts, Dag's roles: None | licensed: False unbound ['withheld_rows'] | absence: ('not_recorded', ['element_not_an_object'])
-- a changed mapping: 422 mapping_changed [('relations[held_role @ experiences[]].period_end', 'exclusive', 'inclusive')]
Read it as an operator would. Ansel's second role could not be read, so he is
undetermined — never none — for a company no readable row of his names, and the
answer says why (withheld_rows_unanswerable:held_role); where a readable row of his
does answer (Lumen Ridge), he is a match. Britta's role ends "2024": the loader did not
turn that into a month, so in June 2024 she is undetermined, not a match. Caro's
record states no role and no country: she is known, and not_recorded. Dag's only role
could not be read: no row of his is stored, and he is still in every answer —
undetermined on a search, and on limma_facts a not_recorded verdict whose
withheld list names the source and the reason. Never none, and never missing.
limma_facts delivers Ansel's two readable roles with an exact count and withholds
these are all of them (withheld_rows).
A line the loader can tie to nobody — not JSON, longer than 64 MB (no document is; it is
stepped over without being held in memory), or holding no readable key or subject —
is quarantined as document_skipped and counted in gaps.unattributed_documents.
Whoever it was about is in no list, so every search over the relations that export
feeds withholds its licence (unattributed_rows_withheld:<relation>; withheld_rows
on limma_facts) for as long as that generation is current. Each of these holes
belongs to the generation that could not read the value: a re-sync whose file states
it readably retires the hole with the old generation, and the licence returns.
The same file under a mapping that reads its end dates the other way is refused until
replaces names the generation it would retire.
What a pull sets aside, what a proposal shows, and a collection of calls¶
Three records that are each wrong in one way a real export is — a company name holding
U+0000 and a bio that is not valid Unicode, a job description pasted into a title — and
that each carry a map keyed by an e-mail address; then the same file again, a proposal
with sample_values, the same file under a mapping whose name path moved, and a
collection of calls whose subject is not its key:
import json, os
from docs.examples.roster import export_dir, new_workspace
ws = new_workspace(client)
client.post("/v1/public/apply_pack", json={"workspace_id": ws, "bundle_id": "bundle_expert_network"})
folder = export_dir()
def write(name, documents): # json.dumps writes U+0000 and a lone surrogate as \uXXXX
(folder / name).write_text("".join(json.dumps(d) + "\n" for d in documents), encoding="ascii")
return str(folder / name)
def connect(path, collection, **body):
r = client.post("/v1/public/connect", json={
"workspace_id": ws, "document_source": {"path": path, "collection": collection}, **body})
return r.status_code, r.json()
def person(key, name, country, bio, company, title):
return {"_id": key, "name": name, "country": country, "bio": bio,
"updated_at": "2024-03-01T09:00:00Z",
"contacts": {f"{key}@mail.example": {"kind": "work"}}, # a map keyed by DATA
"experiences": [{"company_name": company, "career_title": title,
"date_from": "2016-03-01", "date_till": "2019-01-01"}]}
experts = write("experts.jsonl", [
person("ansel_veck", "Ansel Veck", "Norway",
"Ansel Veck ran pump plants for eleven years. He began at Kestrel Pumps AG, "
"and he still knows every valve in the Bergen works.",
"Kestrel Pumps AG", "Director"),
person("britta_holm", "Britta Holm", "Chile",
"Britta Holm's bio was cut inside a character \udc00 by whatever wrote it. "
"The rest of it reads perfectly well, and it is not stored.",
"Lumen\u0000 Ridge", "Analyst"),
person("caro_lindt", "Caro Lindt", "Peru",
"Caro Lindt sells cold storage across the Andes. Her title field holds a "
"whole job description, which is prose and not a title.",
"Sable Point", "Runs the regional accounts and everything near them. " * 50)])
mapping = {"key_path": "_id", "entity_path": "_id", "name_path": "name", "as_of_path": "updated_at",
"relations": [{"relation": "held_role", "iterate": "experiences[]",
"slot_paths": {"company": "company_name", "title": "career_title"},
"period_from_path": "date_from", "period_to_path": "date_till",
"period_end": "exclusive", "missing_end": "open"},
{"relation": "based_in", "slot_paths": {"place": "country"}}],
"documents": [{"path": "bio", "kind": "profile"}]}
os.environ["EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS"] = str(folder)
try:
status, body = connect(experts, "experts", document_mapping=mapping)
bios = body["documents"]["bio"]
print("-- pull:", status, "| bios registered:", bios["registered"], "of", bios["present"])
for q in body["quarantined"]:
print(" ", q["key"], "|", q["path"], "|", q["reason"], "->", q["consequence"],
"| value shown:", len(q["value"] or ""), "characters")
asked = client.post("/v1/public/search", json={"workspace_id": ws, "constraints": [
{"relation": "held_role", "slot": "company", "op": "eq", "value": "Lumen Ridge"}]}).json()
print("at Lumen Ridge:", asked["entities"], asked["coverage"].get("narrowed_by"))
status, again = connect(experts, "experts", document_mapping=mapping)
print("-- the same file again:", status, "| deduplicated:", again["deduplicated"],
"| statements written:", again["relations"]["statements"],
"| bios unchanged:", again["documents"]["bio"]["unchanged"])
status, body = connect(experts, "experts", mode="propose", sample_size=10, sample_values=True)
shown = {f["path"]: (f["samples"], f["samples_withheld"]) for f in body["proposal"]["fields"]}
print("-- propose, with sample_values:")
for path in ("country", "name", "bio", "experiences[].company_name", "updated_at"):
print(" ", path, "->", *shown[path])
print(" the paths under `contacts`:", [p for p in shown if p.startswith("contacts")],
"| keys not listed:", [u["names_not_listed"] for u in body["proposal"]["unmapped"]
if u["path"] == "contacts.*"])
print(" an address anywhere in the reply:", "@mail.example" in json.dumps(body))
status, body = connect(experts, "experts", document_mapping={**mapping, "name_path": ""})
recorded = connect(experts, "experts", mode="propose", sample_size=10)[1]["proposal"]["current_mapping"]
print("-- the same file, the name path dropped:", status, "| deduplicated:", body["deduplicated"],
"| the line now records name_path:", repr(recorded["name_path"]))
calls = write("calls.jsonl", [
{"_id": "t1", "expert_id": "ansel_veck", "call_date": "2024-03-14",
"text": "Interviewer: Which plants?\nExpert: The two in Bergen, for eleven years."},
{"_id": "t2", "call_date": "2024-03-15",
"text": "Interviewer: And then?\nExpert: Nobody recorded who this was."}])
status, body = connect(calls, "calls", document_mapping={
"key_path": "_id", "entity_path": "expert_id",
"documents": [{"path": "text", "kind": "transcript", "as_of_path": "call_date",
"turns": {"label_pattern": "^(?P<label>[A-Za-z]+):",
"interviewer_labels": ["Interviewer"]}}]})
text = body["documents"]["text"]
print("-- calls:", status, "| registered:", text["registered"], "of", text["present"],
"| set aside:", [(q["key"], q["reason"], q["consequence"]) for q in body["quarantined"]])
for note in body["read_errors"]:
print(" ", note)
finally:
os.environ.pop("EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS", None)
-- pull: 200 | bios registered: 2 of 3
britta_holm | bio | text_unstorable -> document_not_registered | value shown: 0 characters
britta_holm | experiences.0.company_name | value_unstorable -> slot_dropped | value shown: 12 characters
caro_lindt | experiences.0.career_title | value_too_long -> slot_dropped | value shown: 200 characters
at Lumen Ridge: {'matched': [], 'none': ['ansel_veck', 'caro_lindt'], 'not_recorded': [], 'undetermined': ['britta_holm']} ['slot_unrecorded_rows_unanswerable:held_role']
-- the same file again: 200 | deduplicated: True | statements written: 0 | bios unchanged: 2
-- propose, with sample_values:
country -> ['Chile', 'Norway', 'Peru'] None
name -> None proposed_as_name_path
bio -> None proposed_as_a_document
experiences[].company_name -> None not_in_a_vocabulary
updated_at -> None proposed_as_as_of_path
the paths under `contacts`: ['contacts', 'contacts.*', 'contacts.*.kind'] | keys not listed: [3]
an address anywhere in the reply: False
-- the same file, the name path dropped: 200 | deduplicated: True | the line now records name_path: ''
-- calls: 200 | registered: 1 of 2 | set aside: [('t2', 'entity_missing', 'document_skipped')]
1 document(s) could not be read or tied to a subject (see `quarantined`): nothing was registered for them. This mapping reads no relation, so no count rests on them.
The subject of every row and document is read from `expert_id`, not from the key (`_id`): the 2 document(s) this pull read name 1 distinct subject(s), and everything a subject's documents hold is filed under that ONE subject. That is the reading for a collection of roles or calls — several documents a subject. If each document here is a subject of its own (a collection of people that carries, say, an account manager's id), the subject is the key: resend with `entity_path: "_id"` and `replaces`.
1 document(s) of 'text' are stored with their text and were not read by a model (read: defer)
Read it as an operator would. Britta's employer could not be stored, so her role is kept
without it and a question about an employer reads her undetermined — never none;
her bio is not registered and says why. Caro's title is past the bound and is dropped;
her employer still answers. Nothing else in the pull was touched, and the three faults
are named with the record and the path each came from — the value itself is quoted
only up to 200 characters, and a document's text not at all. The same file again writes
nothing. The proposal shows values only for country, which this workspace's geography
vocabulary recognises; the name, the bio, the employer and the date each say why they
show none. Each record's contacts is keyed by an e-mail address: the proposal lists
ONE path for the three of them, contacts.*, counts them, and no address is anywhere
in the reply — while kind, which all three carry beneath it, is listed. Sent again
with the name path dropped, the unchanged file still writes no row, and the line
records the mapping it was last pulled under. And the calls are filed under the expert
each names, dated by the call: the one that names nobody is set aside once, whole.
What a proposal could not decide, and a text that left its record¶
Eight people. Six hold one role, the current one — a document store writes no end date
for it — so date_till is on two records in eight. First the proposal as it is; then
the same export with one stray key that does not read as a field name in one role, where
the rare key is no longer listed; then two nights of the notes each record carries:
import json, os
from docs.examples.roster import export_dir, new_workspace
ws = new_workspace(client)
client.post("/v1/public/apply_pack", json={"workspace_id": ws, "bundle_id": "bundle_expert_network"})
folder = export_dir()
def connect(documents, **body):
path = folder / "experts.jsonl"
path.write_text("".join(json.dumps(d) + "\n" for d in documents), encoding="utf-8")
r = client.post("/v1/public/connect", json={
"workspace_id": ws, "document_source": {"path": str(path), "collection": "experts"}, **body})
return r.status_code, r.json()
def person(i):
roles = [{"company_name": "Kestrel Pumps AG", "career_title": "Planner", "date_from": "2019-05-01"}]
if i >= 6: # two of the eight have a role that ENDED
roles.insert(0, {"company_name": "Lumen Ridge", "career_title": "Planner",
"date_from": "2012-01-01", "date_till": "2016-01-01"})
return {"_id": f"p{i}", "country": "Norway", "updated_at": "2024-03-01T09:00:00Z",
"experiences": roles,
"notes": [{"text": f"Note {n} of person {i}. It has two sentences."} for n in range(2)]}
def held_role(proposal):
return next(e for e in proposal["mapping"]["relations"] if e["relation"] == "held_role")
people = [person(i) for i in range(8)]
os.environ["EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS"] = str(folder)
try:
status, body = connect(people, mode="propose", sample_size=10)
entry = held_role(body["proposal"])
print("-- an end date on 2 records of 8:", entry["period_from_path"], "->", entry["period_to_path"])
people[3]["experiences"][0]["job title"] = "x"
status, body = connect(people, mode="propose", sample_size=10)
proposal, entry = body["proposal"], held_role(body["proposal"])
print("-- beside one stray key: period_to_path:", entry["period_to_path"],
"| keys not listed:", proposal["evidence"]["relations.held_role"]["keys_not_listed"],
"| the stray key in the reply:", "job title" in json.dumps(body))
print(" asked:", [u["field"] for u in proposal["unresolved"]])
print(" roles counted as having no end:",
next(u["counts"] for u in proposal["unresolved"] if u["field"].endswith("missing_end")))
mapping = proposal["mapping"]
entry["missing_end"] = "open"
mapping["documents"] = [{"path": "notes[].text", "kind": "profile"}]
status, body = connect(people, document_mapping=mapping)
print("-- sent back with its nulls:", status)
for problem in body["detail"].split(": ", 1)[1].split("; "):
print(" ", problem.split(": ")[0])
entry.update(period_to_path="date_till", period_end="exclusive")
based_in = next(e for e in mapping["relations"] if e["relation"] == "based_in")
del based_in["iterate"] # the record itself states the country
for each in mapping["relations"]: # read, checked against the data: confirmed
each["confirmed"] = True
status, body = connect(people, document_mapping=mapping)
print("-- with the end stated and the country read from the record:", status,
"| roles:", body["relations"]["by_relation"]["held_role"]["rows"],
"| notes registered:", body["documents"]["notes[].text"]["registered"])
people[0]["notes"].pop() # one note leaves p0's record
people.append({"country": "Peru", "updated_at": "2024-03-01T09:00:00Z", "experiences": [], "notes": []})
status, body = connect(people, document_mapping=mapping)
notes = body["documents"]["notes[].text"]
print("-- a note removed, beside a record with no key:", status, "| retired:", notes["retired_by_reason"],
"| kept unconfirmed:", notes["kept_unconfirmed"], "| unchanged:", notes["unchanged"])
finally:
os.environ.pop("EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS", None)
-- an end date on 2 records of 8: date_from -> date_till
-- beside one stray key: period_to_path: None | keys not listed: {'experiences[]': 2} | the stray key in the reply: False
asked: ['relations[based_in].iterate', 'relations[held_role].period_to_path', 'relations[held_role].missing_end', 'relations[].iterate', 'relations[based_in].confirmed', 'relations[held_role].confirmed']
roles counted as having no end: {'elements': 10}
-- sent back with its nulls: 422
relations[0].confirmed is null
relations[0].period_to_path is null
relations[1].confirmed is null
relations[1].iterate is null
-- with the end stated and the country read from the record: 200 | roles: 10 | notes registered: 16
-- a note removed, beside a record with no key: 200 | retired: {'text_removed': 1} | kept unconfirmed: 0 | unchanged: 15
Read it as an operator would. On two records in eight the end date is still a field of
a role, and is proposed. One stray key written with a space in one role changes that:
beside a key that does not read as a field name the proposal lists no key that few
records carry — it could be somebody's name — so the end date is counted, not named,
and the stray key itself is nowhere in the reply. What it does NOT do is propose the
roles as if they had no end: period_to_path is null, the relation's own evidence
says two keys were not listed, the question about a missing end counts only the roles,
and the mapping sent back with that null is refused in words. So is a second null,
on a relation the stray key has nothing to do with: beside an array of objects not all
of whose keys were listed, EVERY relation that is not read from it is asked whether its
rows are that array's elements — based_in, proposed from the record's own country,
carries "iterate": null until you say so. Here the record states the country, so the
key is removed. Stated, it loads. The
second night one note has left a record the pull read, and one record has no key at
all: the note is retired — the pull met its record — and nothing is held back because
of the keyless record, which could only have hidden a record the pull did NOT read.
A collection nobody wrote a pack for¶
No pack is needed. In a workspace no pack declares a relation for, mode: "propose"
also drafts the RELATIONS, from the records themselves and without a model, as
proposal.schema:
- what becomes a relation — the record's own scalar fields (nested objects' fields
too) are the slots of ONE relation named for the collection, a row a record; each
array is one relation more, a row an element (
<collection>_<array>; an array of plain values has the one slotvalue). One level of array, as a mapping reads. - what type a slot gets — whatever the pull's own reader reads of at least 95 % of
the sampled values:
boolean,number(a number written as text is NOT one:"1.10"and"1.1"are two labels),date(YYYY,YYYY-MM,YYYY-MM-DD; an instant is its UTC calendar date), and for other strings atermon a shared core axis where that axis resolves the values (countries, languages), atermwith the workspace's own vocabulary where they are a small closed set, anentity_refotherwise — an id or a name, compared exactly, never a vocabulary of thousands. Prose is registered as a document (mapping.documents), never a slot. Each slot'sevidencecounts the values, how many read as the type and how many do NOT;schema.not_readlists every field that was turned into nothing, with why. - no value of the export is in it. A term's own vocabulary is
observed: built by the PULL from the values the whole export states at the slot, growing with each re-sync, and refused (term_set_not_closed) past the slot'smax_terms. To close a set by statement send"vocabulary": "stated"withterms.
It is a draft like the mapping. Every drafted relation carries "confirmed": null
and a pull that sends one back is refused — nothing is declared by proposing.
CONFIRMING means you have read the slots and their types and what was not read, renamed
or re-typed what the sample guessed wrong, and set true; the pull that receives
document_schema beside document_mapping then DECLARES the relations for the
workspace (under the reserved origin limma_connected, which no signed bundle may use)
and loads. From there they are ordinary relations: limma_search constraints and
limma_facts name them, and a re-sync sends the mapping alone. A schema that differs
from the declared one is handled as a changed mapping is — refused (schema_changed,
naming what changed) unless replaces names the current generation — and a slot's
TYPE never changes under rows that carry it: declare the re-typed slot under a new
name. A workspace that HAS a pack gets the proposal it always got — the mapping onto
the pack's relations, whatever of the export they read; draft_schema: true asks for
the draft there, and its relations are then declared beside the pack's. A SQL table
gets no draft: export it as JSON lines and connect the file.
import json, os
from docs.examples.roster import export_dir, new_workspace
ws = new_workspace(client) # no pack is applied to it
path = export_dir() / "machines.jsonl"
os.environ["EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS"] = str(path.parent)
machines = [{"_id": f"m{n:02d}", "model": f"Press {n:02d}", "line": ["A", "B"][n % 2],
"tonnage": 40 + 10 * n, "installed": f"20{10 + n % 10}-06-01",
"leased": n % 3 == 0, "shifts": ["day", "night"][: 1 + n % 2]}
for n in range(24)]
machines[4]["tonnage"] = "sixty" # a cell that contradicts its field
del machines[9]["tonnage"] # a record that lacks the field
path.write_text("".join(json.dumps(m) + "\n" for m in machines), encoding="utf-8")
def connect(**body):
r = client.post("/v1/public/connect", json={
"workspace_id": ws, "document_source": {"path": str(path), "collection": "machines"}, **body})
return r.status_code, r.json()
proposal = connect(mode="propose")[1]["proposal"]
mapping, schema = proposal["mapping"], proposal["schema"]
for name, relation in schema["relations"].items():
print(name, "<-", relation["reads"]["place"], {slot: spec["type"] for slot, spec in relation["slots"].items()})
tonnage = schema["relations"]["machines"]["slots"]["tonnage"]["evidence"]
print("tonnage:", {k: tonnage[k] for k in ("values", "read_as_type", "not_read_as_type")})
for entry in mapping["relations"]:
entry["confirmed"] = True # the mapping: read, correct as drafted
status, body = connect(document_mapping=mapping, document_schema=schema) # the schema as it came
print(status, body["detail"][:86])
for relation in schema["relations"].values():
relation["confirmed"] = True # the schema: read, correct as drafted
status, body = connect(document_mapping=mapping, document_schema=schema)
set_aside = body["quarantined"][0]
print(status, body["schema_declared"]["declared"], body["schema_declared"]["vocabulary"])
print(set_aside["key"], set_aside["path"], set_aside["reason"], set_aside["consequence"])
def search(*constraints):
r = client.post("/v1/public/search", json={"workspace_id": ws, "constraints": list(constraints)}).json()
return (sorted(r["entities"]["matched"]), r["entities"]["undetermined"],
r["coverage"]["completeness_licensed"], r["coverage"].get("narrowed_by"))
def where(relation, slot, op, value):
return {"relation": relation, "slot": slot, "op": op, "value": value}
print(search(where("machines", "line", "eq", "B"), where("machines", "leased", "eq", True)))
print(search(where("machines", "tonnage", "gte", 240)))
print(search(where("machines_shifts", "value", "eq", "night"), where("machines", "installed", "lte", "2011")))
os.environ.pop("EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS", None)
machines <- the record {'installed': 'date', 'leased': 'boolean', 'line': 'term', 'model': 'entity_ref', 'tonnage': 'number'}
machines_shifts <- shifts[] {'value': 'term'}
tonnage: {'values': 23, 'read_as_type': 22, 'not_read_as_type': 1}
422 unusable document schema: document_schema.relations[machines].confirmed is null: this
200 ['machines', 'machines_shifts'] {'machines_line': 2, 'machines_shifts_value': 2}
m04 tonnage slot_type slot_dropped
(['m03', 'm09', 'm15', 'm21'], [], True, None)
(['m20', 'm21', 'm22', 'm23'], ['m04', 'm09'], False, ['slot_unrecorded_rows_unanswerable:machines'])
(['m01', 'm11', 'm21'], [], True, None)
The record whose tonnage was written in words and the record that has none are not
answered "no" to the threshold: each holds its row, holds no value at that slot, and
reads undetermined — and the licence is withheld, naming the relation.
A CSV or Excel export¶
A CRM, an ERP or a directory exports CSV or Excel. document_source reads such a file as
RECORDS — one row is one record, keyed by the header row — and from there nothing in
this page changes: the proposal (onto a pack, or drafting the relations without one), the
stated mapping, the identity block, generations on re-pull, the quarantine, receipts.
A row's receipt names the collection, the row by its KEY (row_key, as a JSON document
is named) and the COLUMN by its header (db_column); it is re-read from the file as it
is now when the row is delivered.
Which reader opens the file is document_source.format (csv, tsv, xlsx, json,
jsonl, json_array), or else the file's NAME — .csv, .tsv, .xlsx; .xls, .xlsb
and .ods are refused by name — and never its content: a CSV called export.txt is read
as JSON and refused, saying how to name it. An option the format does not take (a
delimiter with a workbook, a sheet with a CSV) is refused, not ignored.
proposal.source says how the file was read; it names columns by POSITION, never a name
the proposal does not list.
What a CSV cell is, and the habits it is held to — each refused BY NAME, with the row it
stopped at (detail.counts.refusal / detail.counts.row on a 422 of type
document_export_rejected, on a proposal as on a pull), and nothing written:
| habit | how it is read |
|---|---|
| encoding | UTF-8, with or without a byte-order mark. A UTF-16 / UTF-32 mark is refused naming it; any byte that is not UTF-8 is refused with its row, line and byte, saying UTF-8 was assumed (not_utf8) |
| delimiter | comma, semicolon or tab: STATED (delimiter: , ; tab; a .tsv is tab by its name; Excel's sep=; first line states it), or read from the HEADER LINE ONLY — the one of the three that stands there outside quotes. None, or two, is refused (delimiter_unknown): state it. source.delimiter_from says which |
| quoting | a quoted cell holds the delimiter, "" and line breaks. Text after a closing quote, or a quote never closed, is refused (csv_syntax) — never folded into the cell |
| empty cell | a MISSING value — the record has no such field — never the empty string; a cell of spaces too |
| header | the first row that is not blank. An empty name is refused (header_empty), a name given twice too (header_duplicate), naming the columns. Unnamed columns AT THE END (what a spreadsheet writes for a column it once formatted) are read while no row holds a value there (value_without_name). Spaces around a name are trimmed, and said |
| a row's width | every row holds as many cells as the header: one with more or fewer is refused with its row number and the line it starts on (row_width) — never padded, never cut |
| rows | numbered as a spreadsheet numbers them (the header is row 1, a blank line an empty row); quarantined[].row names a record's row. More than 200,000 rows is refused (too_many_rows), never cut |
| a cell's type | TEXT: a number in a CSV is text, as a JSON string is ("1.10" and "1.1" are two labels). The draft types such a column as text; declare the slot number and the pull reads it as one |
| a list in a cell | NEVER SPLIT unless the mapping states it: separators: {"skills": ";"} reads each cell of skills as a list (its items stripped), iterated as skills[]. A proposal over a table leaves separators.<column>: null where at least two sampled values divide on ;, \| or , (a number such as 1,200 divides into nothing) and asks; the pull refuses the null. mode: propose takes a mapping that holds ONLY separators, and drafts over the split records |
An XLSX workbook is read one SHEET a source. A workbook of several sheets is refused
unless sheet names one, listing them (sheet_required). Its header is the sheet's
first row holding a value, and must name the columns with TEXT (a number or a date there
is a row of data: header_not_text); the table may start anywhere, and a value left or
right of the named columns is refused (value_without_name). Cells keep the type the
workbook stored — numbers, booleans; a cell formatted as a date IS a date (a date-and-time
format keeps its time); a formula is its STORED result. A formula's error result
(#N/A) and a formula with no stored result (a workbook written by a program that does
not calculate) are no values: set aside where the mapping reads them (cell_error,
formula_without_value) and never read as empty. MERGED cells are reported
(source.merged_ranges, and in read_errors): a merged range holds its value in its
first cell only, and the others are read as missing. HIDDEN rows are reported, and a pull
refuses them (hidden_rows_unstated) until hidden_rows says read (records) or skip
(left out) — a hidden row may be one your export filtered out; a proposal reads them and
asks. Hidden columns are read, and said.
import os
from docs.examples.roster import export_dir, new_workspace
ws = new_workspace(client)
client.post("/v1/public/apply_pack", json={"workspace_id": ws, "bundle_id": "bundle_expert_network"})
folder = export_dir()
people = folder / "people.csv"
header = "id;name;first_name;last_name;email;company;title;country;updated_at"
rows = ["p1;Maren Tolvik;Maren;Tolvik;maren@mail.example;Nordvale;Head of Supply;Norway;2024-02-10",
"p2;Maren Tolvik;Maren;Tolvik;maren@mail.example;Nordvale;Head of Supply;Norway;2024-02-11",
'p3;Ilse Brandtner;Ilse;Brandtner;ilse@mail.example;"Brightmoor; Oslo";Buyer;Chile;2024-02-10',
"p4;Odd Hauge;Odd;Hauge;odd@mail.example;Nordvale;Buyer;Peru;"]
def write(lines): # as Excel writes "CSV UTF-8": a byte-order mark
people.write_bytes(("" + "\r\n".join(lines) + "\r\n").encode("utf-8"))
def connect(**body):
r = client.post("/v1/public/connect", json={
"workspace_id": ws, "document_source": {"path": str(people), "collection": "people"},
**body})
return r.status_code, r.json()
write([header, *rows])
os.environ["EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS"] = str(folder)
try:
status, body = connect(mode="propose")
proposal, source = body["proposal"], body["proposal"]["source"]
print("-- propose:", status, "|", source["format"], source["delimiter_name"],
source["delimiter_from"], "| byte-order mark:", source["byte_order_mark"],
"| columns:", source["columns"])
print(" key:", proposal["mapping"]["key_path"], "| asked:",
[q["field"] for q in proposal["unresolved"]])
mapping = proposal["mapping"]
for entry in mapping["relations"]:
entry["confirmed"] = True # read, checked against the data
block = proposal["identity"]["block"] # an address is one person's
block["identifiers"][0].update(shared_value_is_evidence=True, differing_values_veto=False)
block["confirmed"] = True
mapping["identity"] = block
status, body = connect(document_mapping=mapping)
merged = body["merging"]["types"]["person"]
print("-- pull:", status, "| rows read:", body["row_count"], "| records merged:",
merged["merged_records"], "into", merged["merged_entities"], "person")
found = client.post("/v1/public/search", json={"workspace_id": ws, "constraints": [
{"relation": "held_role", "slot": "company", "op": "eq", "value": "Nordvale"}]}).json()
print("at Nordvale:", found["entities"]["matched"], "|", found["counts"]["universe"],
"people,", found["counts"]["records"]["universe"], "records")
row = client.post("/v1/public/facts", json={
"workspace_id": ws, "question": "roles", "mode": "enumerate",
"relation_query": {"relation": "held_role", "entity_id": "p3"}}).json()["relations"]["rows"][0]
receipt = row["slot_receipts"]["company"]
print("p3's employer:", receipt["value"], "| column:", receipt["db_column"],
"| row key:", receipt["row_key"], "| re-read:", receipt["byte_verified"])
status, again = connect(document_mapping=mapping)
print("-- the same file again: deduplicated:", again["deduplicated"])
write([header, *rows, "p5;Ada Lind;Ada;Lind;ada@mail.example;Nordvale"])
status, body = connect(document_mapping=mapping)
detail = body["detail"]
print("-- a row cut short:", status, detail["type"], "|", detail["counts"]["refusal"],
"| row", detail["counts"]["row"])
print(" ", detail["detail"].split(":")[0])
finally:
os.environ.pop("EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS", None)
-- propose: 200 | csv semicolon sniffed | byte-order mark: True | columns: 9
key: id | asked: ['relations[based_in].confirmed', 'relations[held_role].confirmed']
-- pull: 200 | rows read: 4 | records merged: 2 into 1 person
at Nordvale: ['p1', 'p4'] | 3 people, 4 records
p3's employer: Brightmoor; Oslo | column: company | row key: {"id":"p3"} | re-read: True
-- the same file again: deduplicated: True
-- a row cut short: 422 document_export_rejected | row_width | row 6
row 6 (line 6 of the file) holds 6 cell(s) and the header holds 9
Read it as an operator would. The delimiter was read from the header line alone and the
proposal says so; the byte-order mark Excel writes is read past. The proposal is the one
the same records in JSON get. A cell that holds the delimiter inside quotes is one value
(Brightmoor; Oslo), and its receipt names the row by its key and the cell by its
column. The person entered twice is one person, so the search counts three people over
four records. The same file again writes nothing; a row cut short refuses the whole
export, naming its row, and the generation that was current still answers.
A workbook. Two sheets, so the sheet is named; a hidden row, so the pull needs to be told what it is; a date cell, a merged range and a formula with no stored result, each said:
import datetime, os
from openpyxl import Workbook
from docs.examples.roster import export_dir, new_workspace
ws = new_workspace(client) # no pack: the relations are drafted
folder = export_dir()
book = Workbook()
sheet = book.active
sheet.title = "Contracts"
sheet.append(["contract_id", "counterparty", "kind", "signed", "value", "region"])
for n in range(1, 21):
sheet.append([f"C-{n}", ["Halden Freight", "Lumen Ridge", "Sable Point", "Corvid"][n % 4],
["lease", "supply", "service"][n % 3], datetime.date(2020 + n % 4, 1 + n % 12, 1),
10000 * (1 + n % 5), ["North", "South"][n % 2]])
sheet["E4"] = "=E2*2" # C-3: a formula, and no stored result
sheet.merge_cells("F2:F3") # C-1's region shown across C-2's row
sheet.row_dimensions[21].hidden = True # C-20, filtered out of the view
book.create_sheet("Notes")
book.save(folder / "contracts.xlsx")
def connect(**body):
r = client.post("/v1/public/connect", json={
"workspace_id": ws, "document_source": {"path": str(folder / "contracts.xlsx"),
"collection": "contracts",
**body.pop("source", {})}, **body})
return r.status_code, r.json()
os.environ["EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS"] = str(folder)
try:
status, body = connect(mode="propose")
print("-- unnamed:", status, body["detail"]["counts"]["refusal"], "|",
body["detail"]["detail"].split(":")[0])
status, body = connect(mode="propose", source={"sheet": "Contracts"})
source = body["proposal"]["source"]
print("-- propose:", status, "| hidden rows:", source["hidden_rows"]["rows"],
"| merged:", source["merged_ranges"]["ranges"],
"| formulas with no result:", source["formulas_without_value"]["cells"])
proposal = body["proposal"]
print(" slots:", {slot: spec["type"] for slot, spec in
proposal["schema"]["relations"]["contracts"]["slots"].items()})
mapping, schema = proposal["mapping"], proposal["schema"]
for entry in mapping["relations"]:
entry["confirmed"] = True
for relation in schema["relations"].values():
relation["confirmed"] = True
status, body = connect(source={"sheet": "Contracts"}, document_mapping=mapping,
document_schema=schema)
print("-- pull, hidden rows unstated:", status, body["detail"]["counts"]["refusal"],
"| row", body["detail"]["counts"]["row"])
status, body = connect(source={"sheet": "Contracts", "hidden_rows": "skip"},
document_mapping=mapping, document_schema=schema)
print("-- pull, hidden rows skipped:", status, "| rows read:", body["row_count"])
for q in body["quarantined"]:
print(" ", q["key"], q["path"], q["reason"], "row", q["row"], "->", q["consequence"])
for note in body["read_errors"]:
print(" ", note.split(": ")[0])
found = client.post("/v1/public/search", json={"workspace_id": ws, "constraints": [
{"relation": "contracts", "slot": "value", "op": "gte", "value": 40000}]}).json()
print("value of 40,000 or more:", sorted(found["entities"]["matched"]),
"| undetermined:", found["entities"]["undetermined"])
finally:
os.environ.pop("EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS", None)
-- unnamed: 422 sheet_required | the workbook holds 2 sheets ('Contracts', 'Notes')
-- propose: 200 | hidden rows: [21] | merged: ['F2:F3'] | formulas with no result: ['E4']
slots: {'counterparty': 'term', 'kind': 'term', 'region': 'term', 'signed': 'date', 'value': 'number'}
-- pull, hidden rows unstated: 422 hidden_rows_unstated | row 21
-- pull, hidden rows skipped: 200 | rows read: 19
C-3 value formula_without_value row 4 -> slot_dropped
sheet 'Contracts' merges cells in 1 range(s) (F2:F3)
sheet 'Contracts' hides 1 row(s) holding values (row 21)
sheet 'Contracts' holds 1 formula cell(s) with no stored result (E4)
value of 40,000 or more: ['C-13', 'C-14', 'C-18', 'C-19', 'C-4', 'C-8', 'C-9'] | undetermined: ['C-3']
C-3's value is a formula the file holds no result for: it is set aside by name, its
row kept, and a search over the value reads it undetermined. C-2 sits in the second
row of the merged range: its region is read as missing, as the file stores it, and the
pull says so — never copied down. C-20 was hidden, and skip left it out.
The audit event of a correction¶
limma_correct retires one relation row and emits exactly ONE audit event through the
API's AuditEmitter; the response's audit_event_id is that event's id. An event is
{event_id, event_type, timestamp, workspace_id, actor_id, action, target, result,
metadata} and never contains source content:
| field | on a correction |
|---|---|
event_type |
correction |
action |
correct_relation_row |
target |
the row id |
actor_id |
the actor the caller named |
timestamp |
the same instant as the row's retired_at |
result |
success |
metadata |
{reason, retired_at, relation, entity_id, verb: "correct"} |
A refused correction — a blank reason or actor, an unknown row, a row already corrected — emits no event.
Where the event goes. The API's emitter fans every event out to each sink it holds:
| sink | holds | survives a restart | read with |
|---|---|---|---|
the store (StoreAuditSink, wired whenever the workspace store is) |
one audit_events row per event, written before the verb returns |
yes — it is a table in the deployment's database (DATABASE_URL) |
GET /v1/audit/events, with an API key |
the in-memory buffer (app.state.audit_buffer) |
the most recent 50,000 events of this process | no | GET /v1/audit/{workspace_id} (and /export): a browser session only — an API key alone is 401 |
S3, when LIMMA_AUDIT_SINK=s3 and LIMMA_AUDIT_S3_BUCKET are set |
one object per event | yes | the bucket |
GET /v1/audit/events lists the durable trail oldest first, filtered by workspace_id,
verb, since, request_id and event_type — public_verb_call by default, so ask for
event_type=correction (or * for every type). What a caller may read is one rule, and
the response names it in scope.reads:
- an admin key reads its organisation's events (
organization), within the workspaces its key is scoped to when it is scoped — the correction events included. An admin key scoped to documents reads as the next bullet: another caller's events can name documents it may not read; - any other key reads only the calls it made (
own_calls). A correction event is filed under the key that AUTHENTICATED the call (caller_key_id), never under theactorthe caller typed — that stated actor is kept beside it, in its ownactorcolumn, so a key cannot file a correction under another key's id. Such a key therefore lists both its owncorrectverb call, whosedomain_event_idsholds the correction event's id, and the correction event itself. A read-only (viewer) key cannot correct at all (403); - a deployment with no keys configured (development mode) reads everything (
all).
The example corrects a row with the docs' analyst key, reads the trail with that key, then
with an admin key of the same organisation — added to EPISTEM_API_KEYS for the one call
here; in a deployment it is one more entry of that variable — and finally reads the stored
row over a new database connection, which is what a restarted process reads:
import os
from sqlalchemy import create_engine, text
from docs.examples.roster import new_roster
r = new_roster(client)
row_id = client.post("/v1/public/facts", json={
"workspace_id": r.workspace_id, "question": "roles", "mode": "enumerate",
"relation_query": {"relation": "held_role", "entity_id": "teodor_brask"}}).json()["relations"]["rows"][0]["row_id"]
body = client.post("/v1/public/correct", json={
"workspace_id": r.workspace_id, "row_id": row_id,
"reason": "wrong function recorded", "actor": "ops@example.org"}).json()
own = client.get("/v1/audit/events", params={
"workspace_id": r.workspace_id, "verb": "correct"}).json()
print("analyst key reads:", own["scope"]["reads"], "| its correct calls:", own["count"],
"| links the correction:", own["events"][0]["domain_event_ids"] == [body["audit_event_id"]])
print("correction events listed to the analyst key:", client.get("/v1/audit/events", params={
"workspace_id": r.workspace_id, "event_type": "correction"}).json()["count"])
ADMIN = "sk_docs_admin:org_docs:admin"
configured = os.environ["EPISTEM_API_KEYS"]
os.environ["EPISTEM_API_KEYS"] = configured + "," + ADMIN
try:
trail = client.get("/v1/audit/events", headers={"X-API-Key": ADMIN}, params={
"workspace_id": r.workspace_id, "event_type": "correction"}).json()
finally:
os.environ["EPISTEM_API_KEYS"] = configured
event = trail["events"][0]
print("admin key reads:", trail["scope"]["reads"], "| correction events:", trail["count"],
"| the response's id:", event["event_id"] == body["audit_event_id"])
print(event["event_type"], event["action"], event["result"], "| actor:", event["actor_id"])
print("target is the row:", event["target"] == row_id,
"| timestamp is retired_at:", event["timestamp"] == body["correction"]["retired_at"])
print("metadata:", {k: v for k, v in event["metadata"].items() if k != "retired_at"})
with create_engine(os.environ["DATABASE_URL"]).connect() as db:
stored = db.execute(text("SELECT event_type, verb, caller_key_id, actor FROM audit_events "
"WHERE event_id = :id"), {"id": body["audit_event_id"]}).fetchall()
print("stored row:", [tuple(row) for row in stored])
print("buffer route with only an API key:", client.get(f"/v1/audit/{r.workspace_id}").status_code)
analyst key reads: own_calls | its correct calls: 1 | links the correction: True
correction events listed to the analyst key: 1
admin key reads: organization | correction events: 1 | the response's id: True
correction correct_relation_row success | actor: ops@example.org
target is the row: True | timestamp is retired_at: True
metadata: {'caller_key_id': 'key_5055511a4c29f818', 'entity_id': 'teodor_brask', 'reason': 'wrong function recorded', 'relation': 'held_role', 'verb': 'correct'}
stored row: [('correction', 'correct', 'key_5055511a4c29f818', 'ops@example.org')]
buffer route with only an API key: 401
To answer "who changed this fact, when and why" later: GET /v1/public/corrections lists
the correction with the row's slots and retired_by / retired_at / retired_reason
(scoped by the caller's document ACL), and the audit event with the same id and instant
is the record that it happened through the verb. Erasing the document removes the
correction from that listing but not its audit row, which still names the entity
(data handling: what a document delete leaves).