Loading a sync tool's output

A sync tool (Airbyte, or one like it) copies a source system — a CRM, a ticketing system, an HR system — into a file or a database on a schedule. Limma reads that copy through the doors it already has. No connector is written for the source itself.

Choose an output Limma reads

The sync tool writes	Limma reads it with
JSONL or CSV files with each record's fields at the top level (Airbyte: a file or S3 destination with root-level flattening)	connect with a document_source (A document-store export, A CSV or Excel export)
Tables in Postgres or another SQL database (Airbyte: a database destination's final tables)	connect with database_url + table (Which databases it may open)
Records wrapped in one object per line (a raw format)	choose a flattened output instead: the proposal is drafted over a record's top-level fields

Key each record by the source's own id — never the tool's

A sync tool adds its own columns to every record. Airbyte adds _airbyte_raw_id, _airbyte_extracted_at, _airbyte_meta and _airbyte_generation_id. _airbyte_raw_id is a new value for the same record on every sync. Keyed by it, every sync would load every record as a new one, and the previous generation's records would all read as removed.

Limma does not choose for you when a file has more than one field that could be the key: the proposal asks, and lists them — the source's own id and _airbyte_raw_id among them. Answer with the source's own id. The tool's other columns carry nothing about the record; leave them unmapped.

Keep it in step

Point each scheduled pull at the sync tool's latest output. A pull compares what it reads with the generation before it: a changed record is a new generation of that record, and a record that is gone is said to be gone (Generations and replaces).