Delta Lake¶

Delta Lake is an open table format for data lakes. A Delta Lake table stores data files together with a transaction log. The transaction log provides the table metadata and the table version history.

omniload supports Delta Lake as a source and destination. It reads Delta Lake tables with Polars and writes Delta Lake tables with dlt.

Supported URI schemes¶

Use a Delta Lake URI scheme for the storage system that contains the Delta Lake catalog.

URI scheme	Source	Destination	Description
`file+delta://`	Yes	Yes	Local file system
`s3+delta://`	Yes	Yes	Amazon S3 or an S3-compatible object store
`gs+delta://`	Yes	Yes	Google Cloud Storage
`az+delta://`	Yes	Yes	Azure Storage
`hdfs+delta://`	Yes	Yes	Hadoop Distributed File System
`lakefs+delta://`	Yes	Yes	lakeFS
`uc+delta://`	Yes	No	Unity Catalog

The +delta suffix selects the Delta Lake connector. omniload removes this suffix before it accesses the storage system.

URI format¶

For storage-backed Delta Lake catalogs, use a URI that identifies the catalog root.

<storage>+delta://<catalog-root>?<storage-option>=<value>

The --source-table and --dest-table values identify a table within that catalog.

<schema>.<table>

omniload appends <schema>/<table> to the catalog URI when it reads a source table. It writes the destination table to the dataset named by <schema>.

For a local catalog, use the following format.

file+delta:///path/to/catalog

For an S3 catalog, use the following format.

s3+delta://<bucket>/<catalog-prefix>?<storage-option>=<value>

For a Google Cloud Storage catalog, use the following format.

gs+delta://<bucket>/<catalog-prefix>?<storage-option>=<value>

For an Azure Storage catalog, use the following format.

az+delta://<container>/<catalog-prefix>?<storage-option>=<value>

For an HDFS catalog, use the following format.

hdfs+delta://<host>:<port>/<catalog-prefix>?<storage-option>=<value>

For source reads, omniload passes URI query parameters to polars.scan_delta as storage options. Use the storage options that apply to your storage system and authentication method.

For destination writes to S3, Google Cloud Storage and Azure Storage, omniload builds storage credentials from the URI query parameters and passes the query-free URI to the dlt filesystem destination. Supply the options your storage system requires, or supply none of them and let the environment provide the credentials. See the relevant storage guide for the option names:

hdfs+delta:// and lakefs+delta:// destinations take no options from the query; see Limitations.

Table layout¶

A storage-backed Delta Lake catalog uses this layout.

<catalog-root>/
└── <schema>/
    └── <table>/
        ├── _delta_log/
        └── part-*.parquet

The _delta_log directory is required. Do not use a Parquet directory that does not contain this directory as a Delta Lake table.

Note

On write, dlt normalizes the schema and table names before it builds the directory layout. analytics.events writes to analytics/events, and delta-schema.delta-table writes to delta_schema/delta_table. The source does not normalize; it reads the directory the table name spells. Names that survive normalization therefore address the same directory on both sides, and a catalog omniload wrote is read back by its normalized name.

Read a Delta Lake table¶

The following command reads analytics.events from a local Delta Lake catalog and loads the data into DuckDB.

omniload ingest \
    --source-uri 'file+delta:///data/delta-catalog' \
    --source-table 'analytics.events' \
    --dest-uri 'duckdb:///analytics.duckdb' \
    --dest-table 'public.events'

This command reads the Delta Lake table at:

/data/delta-catalog/analytics/events

Write a Delta Lake table¶

The following command loads a DuckDB table into a local Delta Lake catalog.

omniload ingest \
    --source-uri 'duckdb:///analytics.duckdb' \
    --source-table 'public.events' \
    --dest-uri 'file+delta:///data/delta-catalog' \
    --dest-table 'analytics.events'

This command creates or updates the Delta Lake table at:

/data/delta-catalog/analytics/events

The destination uses the Delta table format. The destination table name must contain both a schema and a table name.

Read from object storage¶

For object storage, use the same table-name format. Add the storage authentication options to the source URI.

omniload ingest \
    --source-uri 's3+delta://my-bucket/lakehouse?access_key_id=<access-key-id>&secret_access_key=<secret-access-key>' \
    --source-table 'analytics.events' \
    --dest-uri 'duckdb:///analytics.duckdb' \
    --dest-table 'public.events'

The connector reads this Delta Lake table:

s3://my-bucket/lakehouse/analytics/events

Use the equivalent gs+delta://, az+delta://, hdfs+delta://, or lakefs+delta:// URI for the corresponding storage system.

Unity Catalog¶

uc+delta:// is available as a source only. The connector passes the Unity Catalog URI to Polars after it removes the +delta suffix.

Use a Unity Catalog URI and authentication options that polars.scan_delta supports.

Incremental loading¶

The Delta Lake source reads the whole table on every run, so there is no cursor for --incremental-key to advance and the source rejects it.

DeltaLake takes care of incrementality on its own, you should not provide incremental_key

Each run of a Delta Lake source replaces the destination table with what it read. Use --incremental-strategy append to add the rows to the destination instead.

The connector reads source data in streaming batches. --page-size sets the batch size.

Limitations¶

A storage-backed table name must name both a schema and a table. Quote a component that contains a dot, as in "my.schema".events.
uc+delta:// is not available as a destination. As a source it addresses the table in the URI, so --source-table only names the dlt resource.
The lakefs+delta:// and hdfs+delta:// destinations take no storage options from the URI query, because dlt models no credentials for those protocols.
The merge, delete-and-insert, and SCD2 strategies need an incremental or merge key, which a whole-table read does not expose.