Connectors — a folder, a bucket, Google Drive or SharePoint, kept current with one command¶

limma sync loads a folder of files, an S3, Azure Blob or Google Cloud Storage bucket, a Google Drive folder, or a SharePoint or OneDrive for Business library into a workspace — and keeps it current: run it again and a new file is added, a changed file replaces its old version, an unchanged one costs nothing. It needs no third-party account and no connector set up on the server: it runs on your machine, reads your files with your credentials, and sends them through the same public doors any integration uses (Sync and access). Hosted and inside your network alike.

Run it¶

pip install limma-ai installs the command (limma) and the Python SDK (import limma) — the client alone, with httpx. Add the extra of the source you read: "limma-ai[s3]", "limma-ai[azure]", "limma-ai[gcs]", "limma-ai[gdrive]" or "limma-ai[sharepoint]". The distribution is limma-ai, never limma: that name on PyPI is somebody else's package. Inside Limma's image, or a checkout of it, the same command is python -m epistem.sdk sync.

The command reads the API key from LIMMA_API_KEY — never from a flag, so the key does not land in your shell history. Use a key that may load documents (role service, or analyst; see Keys and roles):

pip install limma-ai
read -rs LIMMA_API_KEY && export LIMMA_API_KEY      # type or paste the key; nothing is echoed
limma sync ./contracts --url https://limma.example.com

--url is the Limma API (or set LIMMA_BASE_URL). Without --workspace the files go to the key's default workspace — the one every call that names no workspace is served from — and each run's first line names it. The default is created by the first such call of an admin or analyst key; a service key, the usual key for a sync, finds it there, or is told (exit 2) to have one made first or to give --workspace <workspace_id>, which keeps a named workspace instead. Inside your network, run it from the image on the deployment's network, with the folder mounted read-only (the image's value is LIMMA_IMAGE in your .env):

docker run --rm --network limma_limma_egress -e LIMMA_API_KEY \
  -v "$PWD/contracts:/sync/contracts:ro" \
  <LIMMA_IMAGE from .env> python -m epistem.sdk sync /sync/contracts \
  --url http://api:8000

-e LIMMA_API_KEY with no value hands the container your shell's value; it is not written on the command line.

A folder¶

The first run sends every file Limma reads and says what it skipped (these runs name the workspace, ws_contracts):

$ limma sync ./contracts --url https://limma.example.com --workspace ws_contracts
added        2024/msa-northwind.pdf
added        2024/nda-halden.docx
added        2025/renewal-terms.md
skipped      archive/old-deck.ppt  (unsupported type: .ppt — save it as .pptx)
added        board/minutes-2025-03.txt
added        scans/signed-msa.png
limma sync: 5 added, 0 updated, 0 unchanged, 0 deleted, 3 skipped (1 hidden, 1 temporary file, 1 unsupported type), 0 failed

The three skipped are the .ppt (listed, because you can act on it), the folder's .DS_Store and Word's lock file ~$nda-halden.docx (counted; --verbose lists them).

Then renewal-terms.md is edited and the March minutes are removed from the folder. The next run sends the one changed file — the other three cost nothing — and lists the file that is gone, without deleting it:

$ limma sync ./contracts --url https://limma.example.com --workspace ws_contracts
updated      2025/renewal-terms.md
skipped      archive/old-deck.ppt  (unsupported type: .ppt — save it as .pptx)
kept         board/minutes-2025-03.txt  (not on the source: --delete removes it)
limma sync: 0 added, 1 updated, 3 unchanged, 0 deleted, 3 skipped (1 hidden, 1 temporary file, 1 unsupported type), 0 failed; 1 not on the source, kept (--delete removes them)

updated means the old version was erased when the new one was registered — its facts stop answering at once (what happens).

--dry-run says what a run would do and sends nothing; --delete makes the workspace a mirror of the folder:

$ limma sync ./contracts --url https://limma.example.com --workspace ws_contracts --delete --dry-run
skipped      archive/old-deck.ppt  (unsupported type: .ppt — save it as .pptx)
would delete board/minutes-2025-03.txt
limma sync (dry run, nothing sent): 0 to add, 0 to update, 4 unchanged, 1 to delete, 3 skipped (1 hidden, 1 temporary file, 1 unsupported type), 0 failed

$ limma sync ./contracts --url https://limma.example.com --workspace ws_contracts --delete
skipped      archive/old-deck.ppt  (unsupported type: .ppt — save it as .pptx)
deleted      board/minutes-2025-03.txt
limma sync: 0 added, 0 updated, 4 unchanged, 1 deleted, 3 skipped (1 hidden, 1 temporary file, 1 unsupported type), 0 failed

A deletion is the provable delete of Sync and access: every generation of the document and its bytes, read back.

A bucket¶

The same command takes a bucket instead of a folder:

pip install "limma-ai[s3]"      # or "limma-ai[azure]", "limma-ai[gcs]"
limma sync s3://acme-docs/contracts --url https://limma.example.com
limma sync az://contracts/2025 --url https://limma.example.com
limma sync gs://acme-docs/contracts --url https://limma.example.com

The prefix is a folder: s3://acme-docs/contracts reads contracts/… and never contracts-old/…; no prefix reads the whole bucket. Each object is read with the credentials of the machine the command runs on — Limma stores no cloud credential, and nothing is set up on the server:

source	credentials, from this machine's environment	package
`s3://bucket/prefix`	boto3's own chain: `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`, `AWS_PROFILE`, SSO, an instance or task role; `AWS_ENDPOINT_URL` for an S3-compatible store that is not AWS	`boto3` (`limma-ai[s3]`; in the Limma image)
`az://container/prefix`	`AZURE_STORAGE_CONNECTION_STRING`; or `AZURE_STORAGE_ACCOUNT` with `AZURE_STORAGE_SAS_TOKEN` or `AZURE_STORAGE_KEY` — or with neither, an Azure sign-in (`az login`, a managed identity, a service principal), which needs `azure-identity`	`azure-storage-blob` (`limma-ai[azure]`, with `azure-identity`)
`gs://bucket/prefix`	Application Default Credentials: `GOOGLE_APPLICATION_CREDENTIALS`, `gcloud auth application-default login`, an attached service account	`google-cloud-storage` (`limma-ai[gcs]`)

A missing package is named, with the line that installs it (pip install "limma-ai[azure]"); a folder never needs any of them. The identity reading the bucket needs only to list and read objects (s3:ListBucket and s3:GetObject; Storage Blob Data Reader; Storage Object Viewer): the command never writes to, moves or deletes anything in your bucket.

Google Drive¶

A folder in My Drive, or in a shared drive, and every folder under it. Limma hosts no OAuth app: the command signs in as this machine does (Google's Application Default Credentials), and asks for read access to Drive only (drive.readonly).

Install the extra: pip install "limma-ai[gdrive]".
Sign in, one of two ways:
- A service account (for a server, or a schedule). In a Google Cloud project with the Google Drive API enabled, create a service account and a JSON key for it, and point GOOGLE_APPLICATION_CREDENTIALS at the key file. Share the Drive folder with the service account's e-mail address as Viewer; for a shared drive, add the service account to the shared drive as a member.
- Yourself. Google's own gcloud sign-in grants no Drive scope, so create an OAuth client of your own in your project (type Desktop app), download it as client_secret.json, and run gcloud auth application-default login --client-id-file=client_secret.json --scopes=https://www.googleapis.com/auth/drive.readonly,https://www.googleapis.com/auth/cloud-platform.
Find the folder's id: open the folder in Drive; it is the last part of the address, drive.google.com/drive/folders/<folder-id>. A shared drive's own id names its top folder.
Look first, then sync: limma sync gdrive://<folder-id> --url https://limma.example.com --dry-run, then the same without --dry-run.

A Drive file is held under its Drive id, not its path: Drive lets two files of one folder share a name, and a rename or a move keeps the id. Each line of the report prints the file's path, then its id:

$ limma sync gdrive://1BoardPapersFolder --url https://limma.example.com --workspace ws_board
added        Board minutes.docx  [gdrive:1Kq2BoardMinutesDoc]
added        Budget 2025.xlsx  [gdrive:1Lm3Budget2025Sheet]
added        Contract.pdf  [gdrive:1Nn4ContractPdf]
skipped      Survey  [gdrive:1Pp5BoardSurveyForm]  (unsupported type: a Google Forms file — Google Drive exports it to no format Limma reads)
added        notes.txt  [gdrive:1Qq6NotesFirst]
added        notes.txt  [gdrive:1Rr7NotesSecond]
limma sync: 5 added, 0 updated, 0 unchanged, 0 deleted, 1 skipped (1 unsupported type), 0 failed

Then Contract.pdf is renamed Contract (signed).pdf, and one of the two notes.txt is moved to the trash. The rename costs nothing — the same id, the same bytes — and the trash is a deletion:

$ limma sync gdrive://1BoardPapersFolder --url https://limma.example.com --workspace ws_board --delete
skipped      Survey  [gdrive:1Pp5BoardSurveyForm]  (unsupported type: a Google Forms file — Google Drive exports it to no format Limma reads)
deleted      gdrive:1Rr7NotesSecond
limma sync: 0 added, 0 updated, 4 unchanged, 1 deleted, 1 skipped (1 unsupported type), 0 failed

A file gone from the folder is named by its id alone: its path is no longer listed.

Google's own files are exported to the format Limma reads best: a Doc as Word (.docx), a Sheet as Excel (.xlsx, every sheet), a Slides deck as PowerPoint (.pptx, its tables and speaker notes kept), a Drawing as PDF. A Form, a Site, a My Maps map or an Apps Script project has no such format and is skipped, by name. A shortcut to a file is read where it points, under the shortcut's own id; a shortcut to a folder is not followed. A file in the trash is a file gone. A folder that is itself in the trash is refused, never mirrored as empty.

SharePoint and OneDrive for Business¶

A document library of a SharePoint site — or a folder in one — and every folder under it, read through Microsoft Graph with an app registration's own permissions (client credentials). A OneDrive for Business is the library of its owner's /personal/ site.

Install the extra: pip install "limma-ai[sharepoint]".
In Microsoft Entra ID, register an app (App registrations → New registration) and add a client secret to it (Certificates & secrets).
Give it the Microsoft Graph application permission Sites.Read.All, and grant an administrator's consent. To reach only the sites you choose, give it Sites.Selected instead and grant it read on each site (an administrator does so through Graph's site permissions).
Put the app's tenant, id and secret in this machine's environment — the variables az:// signs in with too (a certificate, or a managed identity on Azure, works as well):
```
export AZURE_TENANT_ID=<directory (tenant) id> AZURE_CLIENT_ID=<application (client) id>
read -rs AZURE_CLIENT_SECRET && export AZURE_CLIENT_SECRET
```
Name the library: sharepoint://<tenant>.sharepoint.com/sites/<site>/<library>[/<folder>]. The library is named by its title (Documents) or as its address writes it (Shared Documents, or Shared%20Documents copied from the browser); a Teams site is /teams/<site>, a OneDrive for Business sharepoint://<tenant>-my.sharepoint.com/personal/<user>/Documents.
Look first, then sync: limma sync "sharepoint://contoso.sharepoint.com/sites/finance/Shared Documents" --url https://limma.example.com --dry-run, then the same without --dry-run.

A SharePoint file is held under its item id (sharepoint:<id>), and its version is its cTag — the tag of its CONTENT, which a rename, a move or an edited column does not change, so none of those sends it again. A OneNote notebook is skipped; a shortcut to a file in another library is read where it points, a shortcut to a folder is not followed.

What is synced, and how a change is seen¶

	a folder	a bucket	Google Drive	SharePoint
a file's id in Limma (`external_id`)	its path relative to the folder, with `/` between folders on every system, in Unicode NFC — the same folder synced from a Mac or from Linux names each file the same way	its key, relative to the prefix, exactly	`gdrive:<file id>` — a shortcut's own id for a file read through it	`sharepoint:<item id>`
its version (`external_version`)	the SHA-256 of its bytes — read, never guessed from a timestamp	the object's ETag (its content hash where the store gives none)	a file's `md5Checksum`, checked against the bytes read; a Google file's Drive `version` and export format (`v42.docx`), read again after the export	the item's `cTag`, read again after the download
its name in Limma	the same as its id	the same as its id	its path, with the export's extension (`Board minutes.docx`), or its type's where its name has none	its path, with its type's extension where its name has none
a rename or a move inside the source	a new id: the old one deleted with `--delete`, the bytes not read again	the same	nothing sent (the document keeps the name it was last sent under)	the same

What the workspace holds is read from the server's reconcile listing on every run — never from a state file on your machine, which would be wrong the first time anything else changed the workspace. A file at the version the listing holds is not sent; a file at another version is sent as a new version; an id the workspace holds that the source no longer has is listed, and deleted only with --delete. The command sends first and deletes after, so a renamed or moved file is never read twice: its bytes are the document the old name held.

Sent: the types Limma reads — PDF (.pdf), Word (.docx, .doc), PowerPoint (.pptx), Excel (.xlsx, .xlsm, .xls), OpenDocument (.odt, .ods, .odp), HTML (.html, .htm), e-mail (.eml, Outlook .msg, with their attachments), RTF (.rtf), XML and XBRL (.xml, .xbrl), CSV and TSV (.csv, .tsv), text (.txt, .text, .log, .md, .markdown, .json), zip archives (.zip, every file in it read by its own type), and images (.png, .jpg, .jpeg, .tif, .tiff, .bmp, .gif, .webp), which the deployment reads by OCR or refuses by name when it has none. The list is the server's own: a test holds it to the reader.

Not sent, and said:

any other type — .ppt and .xlsb with the format to save them as; every other extension, and a file with none, by its extension;
a hidden file or folder (a name beginning with .) and an Office lock file (~$…) — counted on the summary line, listed with --verbose;
a linked folder (not followed: a link can loop, or lead out of the folder) — a linked file is read where it points; a Drive or SharePoint shortcut alike;
a Google file of a kind Drive exports to nothing Limma reads (a Form, a Site), and a OneNote notebook in SharePoint;
a file that changed while it was being read (it is sent on the next run), and a name the server cannot hold as an id — a control character, more than 1,024 characters, or two files whose names are the same once normalised — each a failure, with why.

A file larger than the deployment's upload limit (100 MB unless the operator set LIMMA_MAX_UPLOAD_BYTES) is refused by the server and reported as a failure. Records exports you want as records rather than as documents — a CSV of contracts, a JSONL of people — go through the connect path instead.

Safety¶

Nothing is deleted without --delete. Without it, a run lists what --delete would remove.
Nothing is deleted when the source could not be listed completely — a folder that could not be opened is not a folder whose files are gone: the run reports it as a failure and withholds every deletion.
A mirror of nothing is refused. With --delete, a source that holds no file Limma reads (an unmounted share, a mistyped prefix) would delete every document the workspace holds; the run refuses, says so, and fails. To empty a workspace on purpose, delete its documents through the API.
Each source owns its ids. Without --id-prefix, the command treats every external id of the workspace as its own. To sync several folders or buckets into one workspace, give each its own name: --id-prefix contracts holds that source's files as contracts/<path>, and its runs — --delete included — touch nothing outside it.

Options¶

option
`--workspace ID`	the workspace to keep current; left out, the key's default workspace (named on each run's first line). Given and empty is refused, never read as "left out"
`--access user:alice@x.com,group:legal`	who may see every file sent (access lists). Changed on a later run, every document's list is replaced without reading it again. Left out, each document keeps the list it has — a new version keeps the old version's. An empty entry, or one with a space at an end, is refused before anything is sent.
`--delete`	mirror mode: delete what is gone from the source
`--dry-run`	say what would be sent and deleted; send nothing
`--watch SECONDS`	run again every SECONDS until stopped (each run's summary line starts with the time)
`--id-prefix NAME`	hold this source's ids under `NAME/`
`--concurrency N`	files sent at once: 4 by default, at most 16
`--retry-failed`	read again the documents the workspace holds and could not read
`--verbose`	also list unchanged files and hidden or temporary ones

The result¶

One line for each file added, updated, deleted, kept, skipped or failed (with why), then one summary line: added, updated, unchanged, deleted, skipped and failed, the skips and failures counted by reason. A request the server asks to repeat (429, using the Retry-After it names), a server error and a dropped connection are tried up to five times before the file is counted as failed.

exit status
`0`	the workspace holds every supported file of the source, at its version, and could read it
`1`	something failed — each is listed with why. A document the workspace holds and could not read fails every run until it is fixed or read again (`--retry-failed`); a document just sent is read in the background, so a failure to read it is reported by the next run
`2`	the command cannot work as given — an argument, `LIMMA_API_KEY` unset, a missing package or cloud credentials, a key the API refuses, a workspace that does not exist, no default workspace for a key that names none. `--watch` stops
`130`	interrupted

With --watch, a run that fails is reported and the next one runs on time; only status 2 stops it. For a schedule instead, run the same command from cron or a systemd timer.

Permissions¶

The key needs to load documents: the command reads the reconcile listing, uploads through POST /v1/workspaces/{id}/documents and deletes with POST /v1/public/delete_document — writes, refused to a read-only (viewer) key or a key confined to documents. The call is made by your platform as itself: it asserts no end user's principals.
The source needs only to be read: Drive's drive.readonly scope (a folder shared as Viewer), Graph's Sites.Read.All or Sites.Selected with read.
Who sees each document is the --access list; with a workspace that sets access_required, a document sent without one is visible to no end user (access_required).

Limits¶

It finds changes by listing, not by a change feed. Every run lists the whole source and, for a folder, reads every file to hash it; a bucket's, Drive's and SharePoint's listings carry their versions. Drive and SharePoint are listed one folder (and one page) per request, so a large tree costs as many requests; their throttling (a 429 or 503, and Drive's rate-limit 403s) is waited out for the Retry-After named, else 1, 2, 4, 8 seconds.
A Google file is sent again when its Drive version moves — which Drive moves for every change, a comment or a sharing change included (it is the one field no client can set back). The export is sent; when its bytes are the ones the workspace holds, nothing new is read. A Google file Drive will not export (its API exports at most 10 MB of one) fails with Drive's own words. A document renamed or moved keeps, in Limma, the name it had when its content last changed.
Deletions are paced by the server: delete_document and the listing count against the API's rate limit (60 a minute per key by default), so a run that deletes thousands of documents takes minutes. The command waits as the server asks.
Memory: each file is read whole before it is sent, so a run holds up to --concurrency files at once.
Azure Blob and Google Cloud Storage are read through their official client libraries, tested against a local stand-in of each service's API rather than a live account — run --dry-run against yours first. S3 is tested against an in-process S3 (moto). A listing is read page by page, as each library pages it; the stand-ins serve one page.
Google Drive and SharePoint are read over their REST APIs, tested against stand-ins of those APIs built from their published references (paging, throttling, export, the trash, renames, moves, names shared by two files, permission errors) — not yet against a live account. Run --dry-run against yours first.