One thing, entered twice¶

A list that says "these are all of them" has to count things — people, companies, contracts, products, accounts — not records. A real collection holds the same thing more than once ("Dr. Maren Q. Tolvik" and "Maren Tolvik", entered a year apart; "Solvane K.K." and "SOLVANE KABUSHIKI KAISHA" in two systems; one contract exported twice) and different things that share a name. With an identity block on a connect, the workspace recognises two records of the export's entity type as one thing, and every search over that type then counts and lists things.

The two mistakes it can make are not alike, and the whole design follows from that:

A wrong merge joins two histories into one thing, which then matches questions neither of them satisfies. Nothing downstream can tell a joined history from a real one.
A missed merge counts one thing twice. Both records are in the list, and merging them later fixes it.

So two records are merged only on evidence a different thing cannot plausibly share, never on a name. Everything weaker is a proposal: you see it, and nothing is merged until somebody confirms it. A contradiction between the records always wins over evidence. No row is ever rewritten — a merge is applied when an answer is assembled, so it is undone by one decision.

State how records are recognised — one block per entity type¶

The block is part of the document mapping, and it is about the export's ENTITY TYPE (the subject_kind of its relations: person, company, contract, …). Each type is merged under its own block; a record is never compared with a record of another type. It is stated, never inferred:

key	what you state
`name_kind`	how the type's names are compared — required. `person`: given, middle and family names, titles, credentials. `organisation`: the legal form is read apart from the name ("Solvane" and "Solvane K.K." may be one company; "ACME LIMITED" and "Acme Ltd." are one name; "Ferrum Systems AG", "Ferrum Systems Pvt Ltd" and "Ferrum Systems Ltd" are three companies — a different legal form is never evidence of sameness). `label`: a product name, a contract title — the same text (case, accents and separators ignored), or different.
`given_name_path`, `family_name_path`	a person's own name fields, when the record has them (both or neither). The display name is the mapping's `name_path`. A name is needed to compare two records at all — and never merges them.
`identifiers`	fields that identify a thing — `[{path, kind, compare_as, shared_value_is_evidence, differing_values_veto}]`; `emails[]` reads an array. Every key is required.
`identifiers[].kind`	your word for what it is: `registration_number`, `lei`, `vat_id`, `web_domain`, `contract_number`, `sku`, `gtin`, `medical_record_number`, `email`, `profile_url`. Values of two kinds never match each other.
`identifiers[].compare_as`	how two values are compared: `email` (case ignored; a shared mailbox such as `info@…` is never evidence), `url` (scheme, `www.`, query, trailing slash and case ignored), `domain` (the host of what is written), `code` (an issued number: case and every separator ignored — `GB 123-456` is `gb123456`), `text` (case, accents and separators ignored).
`identifiers[].shared_value_is_evidence`	`true`: one value is never held by two different things, so a value two records share is evidence. `false` for an office telephone or a shared inbox.
`identifiers[].differing_values_veto`	`true`: one thing never holds two values, so two different values rule a merge out (a registration number, a contract number, a profile URL). `false` for e-mail addresses or web domains — a person changes jobs, a company has several sites.
`corroborate`	relations whose shared rows are evidence: both records state the same row (the named party and one other slot equal, nothing disagreeing, periods that can coincide) — the same role at the same company, the same party to the same contract.
`exclusive`	`[relation, slot]` pairs one thing holds one value of, such as where it is based. Two values the vocabulary places apart rule a merge out; "UK" and "United Kingdom" are one place.

An identifier's value is never stored or returned: the workspace keeps a key (a hash) and the cell it came from. A key is still personal data — it is erased with the pull that read it.

An entity id two types' exports both use (a people export and a company export both keyed 1, 2, …) is merged by neither — which thing would the merge be about? — and the connect reply counts it (merging.ids_in_two_types).

What merges, and what is only proposed¶

The same table for every type:

the two records share	result
an identifier you declared as evidence and a `corroborate` row	merged
two `corroborate` rows, at least one with overlapping dates	merged
a strong identifier — declared both ways (`true`, `true`): a GTIN, a registration number, a contract number — under names that can be one thing's (the same name; "Solvane" / "Solvane K.K."), with nothing contradicting, and the two records' rows not disagreeing: one record has no row to compare, or they share a row (even one many records share — one product at one big retailer)	merged
a strong identifier, but both records state rows and share none (the signature of a number pasted onto another thing: one contract number on another contract with other parties)	proposed (`identifier_unconfirmed`)
an identifier that is evidence but not strong (an e-mail address, a web domain) and no rare row beside it	proposed (`identifier_unconfirmed`)
an identifier under two names that are not the same thing's ("Supply Agreement" / "Supply Agreement (Amended)", "Ferrum Systems AG" / "Ferrum Systems Ltd")	proposed (`identifier_names_differ`)
one `corroborate` row	proposed (`one_shared_fact`)
two `corroborate` rows, none dated	proposed (`shared_facts_undated`)
only rows that many records share	proposed (`common_facts_only`)
a name, in a small group of that name	proposed (`same_name`)
an identifier you declared as evidence and a contradiction	proposed (`conflicting_evidence`)
rows and two values of an `exclusive` relation placed apart	proposed (`conflicting_evidence`)
two different values of an identifier declared `differing_values_veto` (two contract numbers, two profile URLs), no shared identifier, and shared rows that would merge the two by themselves (two or more, one with overlapping dates)	proposed (`conflicting_evidence`)
two different values of an identifier declared `differing_values_veto`, no shared identifier, and shared rows that would not merge them by themselves (one row, or rows with no dates)	decided: two things
a contradiction and nothing else	decided: two things

A contradiction is two different values of an identifier a thing holds one of, two values of an exclusive relation the vocabulary places apart, or somebody's decision that the two are different. A value many records hold — a switchboard, a template URL, a GTIN printed on three differently named products — is never evidence, and neither is a shared mailbox such as info@….

Two different values of an identifier a thing holds only one of decide "two things" even when the records share rows — unless a shared identifier stands beside them, or the shared rows would merge the two by themselves. A wrong merge cannot be undone by anything downstream; a missed one is counted twice and undone by one same_entity decision. So two contracts of one title that share a party and hold two numbers are two contracts: a search about one of them reads none for the other, not undetermined. And a true duplicate whose contract number was mistyped on the copy — its parties shared, but parties carry no dates — is counted twice and listed nowhere as a proposal: never joined to another contract. Where the shared rows are strong enough to merge on their own — one person, two profile URLs, two dated roles in common — the pair is a real conflict and stays a proposal: a duplicate nothing proposes is invisible to counts.matched_lower_bound, which would then claim more things than there are.

The example names its export by a path on the API host. On a deployment you cannot write a file to, upload it first and pass {"export_id": ..., "collection": "experts"} instead — the same mapping, the same merge (upload it, then connect its id).

import json, os
from docs.examples.roster import export_dir, new_workspace

ws = new_workspace(client)
client.post("/v1/public/apply_pack", json={"workspace_id": ws, "bundle_id": "bundle_expert_network"})
role = lambda company, title, start, end: {
    "company_name": company, "career_title": title, "date_from": start, "date_till": end}
career = [role("Nordvale", "Head of Supply", "2015-01-01", "2019-06-01"),
          role("Brightmoor", "VP Operations", "2019-07-01", None)]
person = lambda key, name, roles, country, url="": {
    "_id": key, "name": name, "first_name": "Maren", "last_name": "Tolvik",
    "experiences": roles, "country": country, "linkedin_url": url, "emails": [],
    "updated_at": "2024-02-10T09:00:00Z"}
export = export_dir() / "experts.jsonl"
export.write_text("".join(json.dumps(record) + "\n" for record in [
    person("rec_1", "Maren Tolvik", career, "Norway", "https://profiles.example/in/mt-1"),
    person("rec_2", "Dr. Maren Q. Tolvik", career[:1], "Norway", "http://profiles.example/in/MT-1/"),
    person("rec_3", "MAREN TOLVIK", career[1:], None),
    person("rec_4", "Maren Tolvik", [role("Oakhollow", "Buyer", "2012-01-01", None)], "India")]),
    encoding="utf-8")

mapping = {"key_path": "_id", "entity_path": "_id", "name_path": "name", "as_of_path": "updated_at",
    "relations": [
        {"relation": "held_role", "iterate": "experiences[]",
         "slot_paths": {"company": "company_name", "title": "career_title"},
         "period_from_path": "date_from", "period_to_path": "date_till",
         "period_end": "exclusive", "missing_end": "open"},
        {"relation": "based_in", "slot_paths": {"place": "country"}}],
    "identity": {"name_kind": "person",
        "given_name_path": "first_name", "family_name_path": "last_name",
        "identifiers": [{"path": "linkedin_url", "kind": "profile_url", "compare_as": "url",
                         "shared_value_is_evidence": True, "differing_values_veto": True}],
        "corroborate": ["held_role"], "exclusive": [["based_in", "place"]]}}

os.environ["EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS"] = str(export.parent)
try:
    pulled = client.post("/v1/public/connect", json={
        "workspace_id": ws, "document_mapping": mapping,
        "document_source": {"path": str(export), "collection": "experts"}}).json()
finally:
    del os.environ["EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS"]
people = pulled["merging"]["types"]["person"]
print("resolved:", people["status"], "| merged", people["merged_records"], "records into",
      people["merged_entities"], "person |", people["merged_pairs_by_kind"])
print("proposed:", people["proposals_by_kind"], "| decided different:", people["vetoed_pairs"])

def search(company):
    return client.post("/v1/public/search", json={"workspace_id": ws, "detail": "full",
        "constraints": [{"relation": "held_role", "slot": "company", "op": "eq", "value": company}]}).json()

found = search("Nordvale")
match = found["matches"][0]
print("-- worked at Nordvale:", found["counts"]["matched"], "person;",
      found["counts"]["records"]["matched"], "records | unit:", found["merging"]["unit"])
print("   ", match["entity_id"], "=", match["name"], "| members:",
      [(m["entity_id"], m["name"]) for m in match["members"]])
print("    merged by:", match["merged_by"][0]["kind"], "|",
      [e["kind"] for e in match["merged_by"][0]["evidence"]])
print("    one role, stated twice:", sorted((row["entity_id"], len(row["same_fact_as"])) for row in match["rows"]))
for pair in found["possible_duplicates"]:
    print("-- possible duplicate:", pair["entity_ids"], pair["kind"], pair["statuses"])
print("   rec_3 is", [name for name in ("none", "undetermined") if "rec_3" in found["entities"][name]],
      found["groups"][0]["undetermined_reasons"], "| licensed:", found["coverage"]["completeness_licensed"])

decided = client.post("/v1/public/resolve", json={"workspace_id": ws, "entity_type": "person",
    "decisions": [{"entity_ids": ["rec_1", "rec_3"], "decision": "same_entity",
                   "reason": "confirmed with her by phone", "actor": "ops@example"}]}).json()
print("-- confirmed:", decided["entities"][0]["members"], "| audited:", bool(decided["decisions"][0]["audit_event_id"]))
undone = client.post("/v1/public/resolve", json={"workspace_id": ws, "entity_type": "person",
    "decisions": [{"entity_ids": ["rec_1", "rec_2"], "decision": "different_entities",
                   "reason": "two people after all", "actor": "ops@example"}]}).json()
print("-- a merge undone:", undone["entities"][1]["members"], "| replaced:", repr(undone["decisions"][0]["previous"]))
print("   worked at Nordvale now:", search("Nordvale")["counts"]["matched"], "people")

resolved: complete | merged 2 records into 1 person | {'identifier_and_shared_facts': 1}
proposed: {'one_shared_fact': 1, 'same_name': 1} | decided different: 2
-- worked at Nordvale: 1 person; 2 records | unit: person
    rec_1 = Maren Tolvik | members: [('rec_1', 'Maren Tolvik'), ('rec_2', 'Dr. Maren Q. Tolvik')]
    merged by: identifier_and_shared_facts | ['shared_identifier', 'shared_fact']
    one role, stated twice: [('rec_1', 1), ('rec_2', 1)]
-- possible duplicate: ['rec_1', 'rec_3'] one_shared_fact {'rec_1': 'matched', 'rec_3': 'undetermined'}
-- possible duplicate: ['rec_3', 'rec_4'] same_name {'rec_3': 'undetermined', 'rec_4': 'undetermined'}
   rec_3 is ['undetermined'] ['possible_duplicate'] | licensed: True
-- confirmed: ['rec_1', 'rec_2', 'rec_3'] | audited: True
-- a merge undone: ['rec_2'] | replaced: ''
   worked at Nordvale now: 2 people

rec_1 and rec_2 share a profile URL (written two ways) and a role, so they are one person — rec_1, under one name, with both records as members. rec_3 shares the name and one role with her: a proposal. rec_4 has the same name, another career and another country: against rec_1 and rec_2 that is a contradiction with no evidence beside it, so it is decided — two people. Against rec_3, which states no country, there is only the name: a same_name proposal. The search still licenses "these are all": a record that may be a listed person's duplicate adds nobody who is not listed.

Companies are recognised the same way, under their own block — here an organisation name, a registration number that is one company's and never shared, and a web domain a company may hold several of. (No pack is needed: the pull declares its relation.)

import json, os
from docs.examples.roster import export_dir, new_workspace

ws = new_workspace(client)
companies = export_dir() / "companies.jsonl"
company = lambda key, name, reg_no, website, sector: {
    "_id": key, "name": name, "reg_no": reg_no, "website": website, "sector": sector}
companies.write_text("".join(json.dumps(record) + "\n" for record in [
    company("co_1", "Solvane K.K.", "JP-1234-5", "https://www.solvane.example", "chemicals"),
    company("co_2", "SOLVANE KABUSHIKI KAISHA", "jp 1234 5", "", "chemicals"),
    company("co_3", "Ferrum Systems AG", "DE-77", "ferrum.example", "steel"),
    company("co_4", "Ferrum Systems Ltd", "GB-99", "ferrum.example/uk", "steel")]),
    encoding="utf-8")
schema = {"relations": {"company_profile": {"confirmed": True, "subject_kind": "company",
    "slots": {"sector": {"type": "term", "vocabulary": "observed", "max_terms": 50}}}}}
company_mapping = {"key_path": "_id", "entity_path": "_id", "name_path": "name",
    "relations": [{"relation": "company_profile", "slot_paths": {"sector": "sector"}}],
    "identity": {"name_kind": "organisation", "identifiers": [
        {"path": "reg_no", "kind": "registration_number", "compare_as": "code",
         "shared_value_is_evidence": True, "differing_values_veto": True},
        {"path": "website", "kind": "web_domain", "compare_as": "domain",
         "shared_value_is_evidence": True, "differing_values_veto": False}]}}
os.environ["EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS"] = str(companies.parent)
try:
    pulled = client.post("/v1/public/connect", json={
        "workspace_id": ws, "document_mapping": company_mapping, "document_schema": schema,
        "document_source": {"path": str(companies), "collection": "companies"}}).json()
finally:
    del os.environ["EPISTEM_CONNECT_ALLOWED_EXPORT_DIRS"]
for kind, account in sorted(pulled["merging"]["types"].items()):
    print(kind, "|", account["status"], "| merged", account["merged_records"], "records into",
          account["merged_entities"], "|", account["proposals_by_kind"])
steel = client.post("/v1/public/search", json={"workspace_id": ws, "detail": "full",
    "constraints": [{"relation": "company_profile", "slot": "sector", "op": "eq", "value": "chemicals"}]}).json()
print("-- chemicals:", steel["counts"]["matched"], steel["merging"]["unit"], "|",
      [m["entity_id"] for m in steel["matches"][0]["members"]], "|", steel["matches"][0]["merged_by"][0]["kind"])

company | complete | merged 2 records into 1 | {}
-- chemicals: 1 company | ['co_1', 'co_2'] | identifier

co_1 and co_2 write one legal form two ways ("K.K." is "Kabushiki Kaisha") and share the registration number; co_2 states nothing else, so the number — one company's, never shared — merges them. co_3 and co_4 share a web domain, but "AG" and "Ltd" are two legal forms and their registration numbers differ: two companies, decided — not even proposed. (Had they shared only the domain and no number contradicted it, the pair would be shown as a proposal, identifier_names_differ, and still never merged.) In a workspace holding people too, each type is resolved under its own block in the same run, and merging.types reports each.

What a search says¶

Where the workspace merges records of the type a search counts, limma_search counts merged things:

merging — {status, entity_type, unit, merged_entities, merged_records, proposals, possible_duplicates_total, …}. unit is the entity type (person, company) when merges are applied: what every count counts.
counts are of merged things; counts.records is the same partition counted per record.
A match is one thing: members are the records merged as it, merged_by the evidence each merge rests on, name one name. Each row keeps the record that states it, and same_fact_as marks a fact two records state — one role, not two.
possible_duplicates lists the proposals that touch the answer ({entity_ids, resolved_ids, kind, statuses, evidence, vetoes}). Nothing in it is merged: each side is counted as itself.
A record proposed as the duplicate of a thing that matched (or is open) is never none — it reads undetermined, reason possible_duplicate. If it is that thing, "no" would be false.
counts.matched_lower_bound / matched_upper_bound bound the number of things: the count if every proposal among the matches were confirmed, and the count if every unlisted group of possible duplicates that could together match were one more. In that second case the completeness licence is withheld (possible_duplicate_unanswerable:<relation>). Neither bound knows a duplicate that nothing proposed.
The compact reply lists each thing once — one entry of matches (and at most one of close.matches), with records where several were merged — and says what matched.total counts under counted.unit.

merging.status says whether the fold is current. While a resolution is pending, running or failed — or the type's collections state two different blocks (config_conflict) — a reply counts records (merging.unit: "record"): a duplicate is counted twice and nothing is merged. It is never the other way round. A type no identity block was stated for gets the reply it always did, with no merging block.

limma_facts enumerates a relation's rows per record; each row carries resolved_id, and asked about one record, relations.resolved names the thing it belongs to and every other member.

Answer a proposal, or undo a merge¶

resolve records a decision about two records of ONE entity type (entity_type) and re-resolves the workspace: same_entity merges them from then on; different_entities splits them whatever the evidence, and they are not proposed again. The latest decision on a pair wins, each one is an audit event, and no row is touched. A pair of records of two types, or of a type with no block, is refused. limma_state lists the merged things per type (merging.entities) and every pending proposal (entity_duplicate_proposals, kind: entity, with its entity_type).

A thing an operator joins across a contradiction — one record based in Norway, the other in India — is merged, and is then undetermined for questions about where it is based (members_disagree): the records do not say.

What it does not do¶

It does not merge on a name, at any similarity. "Bob" and "Robert" are two names; "Ferrum Systems AG" and "Ferrum Systems Ltd" are two companies; "Master Services Agreement" names many contracts.
It reads only what a connect declared. A value a model extracted from a document is never evidence that two records are one thing.
A duplicate with nothing to compare — another spelling of the name, no shared identifier, no shared row — is not found, and nothing says so.
mode: propose drafts the block from field names and offers it beside the proposed mapping (proposal.identity.block), every judgement null and "confirmed": null (and name_kind null unless it found a given and a family name field). The proposed mapping itself merges nothing. Add the block to the mapping as identity to have records merged; a pull refuses it until you have answered each question.
S3's people block is retired: a connect that sends it is refused with the reason. Send the same block as identity, with "name_kind": "person" and each identifier's compare_as. A workspace loaded under a people block keeps resolving as it did.