seon99il

Data Engineering

Delta Lake with Polars: How to handle versioning and ACID on file systems

7 min read

Today, I’ll explore how Delta Lake works and its key benefits. We’ll also walk through managing Delta files in a local environment using Polars.

Disadvantages of Existing Systems Before Delta Lake

1. Lack of ACID Transactions

Using file formats like Parquet or CSV alone can lead to write failures, duplicate writes, or read-write contention

  • During appends, multiple writes to the same file can create duplicate data
  • If another process reads at the same time, it risks reading a partial (incomplete) write

2. Problems with Schema Evolution / Enforcement

Traditional data lakes struggled with schema changes and management

  • Changing the schema required regenerating the entire Parquet file
  • If files with different schemas were mixed, the read engine would break
  • Engines that couldn’t process added columns would fail

3. Difficulty with Upserts and Deletes

In traditional Parquet-based systems, modification operations were virtually impossible. To change just one row in a Parquet file with 100 rows, you had to rewrite all 100 rows

  • Parquet is immutable → partial row deletes/updates are not possible
  • Solution: “Rewrite the entire file” → highly inefficient
  • Risk of conflicts if files are rewritten concurrently in a parallel environment

4. No Data Versioning & Time Travel

Time Travel — the ability to ask “What did the data look like yesterday?”, “I need to revert to a previous version,” or “Track which job corrupted the data” — was not possible

Delta Lake

Delta Lake is a storage format and a transaction layer. It provides ACID transactions on top of your data lake. It offers performance advantages and Versioning (Time Travel) for modifications (updates, inserts, deletes) to data that is stored in an immutable format (like Parquet)

Delta Lake Integrations — Use the following frameworks, Delta Sharing clients, managed services, and/or community integrations for Delta Lake and…

Delta Lake Structure

.
└──── delta # Delta Lake directory
    └── users # Delta Table
        ├── _delta_log
        ├── part-00000-....snappy.parquet
        └── part-00000-....snappy.parquet

As shown above, Delta Lake records transaction logs in JSON files inside the _delta_log directory. Furthermore, it can be structured as follows (often called a Medallion Architecture):

S3 / HDFS / Blob Storage
 ├── Raw Zone        (Original source, append only)
 │   └── users
 │       ├── _delta_log
 │       └── part-00000-....snappy.parquet
 │
 ├── Bronze Zone     (Minimal processing)
 ├── Silver Zone     (Cleansed & normalized data)
 └── Gold Zone       (BI-optimized aggregates)

What is a Medallion Architecture? — A medallion architecture is a data design pattern used to logically organize data in a lakehouse, with the goal of…

Now, let’s explore how Delta Lake solves the problems we discussed earlier.

How Delta Lake Handles CRUD Operations

1. No ACID Guarantee → Solved by Delta’s Transaction Log

Delta implements ACID by recording the following in the _delta_log directory

Existing Problem: S3/HDFS are file systems and lack atomic writes, rollback, and consistency

Delta Lake’s Solution:

  • Each commit is one JSON file = 1 transaction (e.g., 000000000000000010.json)
  • When a write finishes, the commit file is atomically renamed. → The rename operation is atomic in HDFS/S3
  • The log includes information on file additions/deletions, data statistics, etc.

Read engines like Spark, Polars, and DuckDB read this log and combine the files to construct the most recent version of the table

2. Mixed Schemas, Broken Data → Schema Enforcement/Evolution

Schema Enforcement: The write operation fails if the schema does not match. Schema Evolution: Allows schema changes (e.g., mergeSchema=true).

  • Adding columns
  • Different column order → automatically matched
  • Supports changes to nullable properties, etc.

Delta records file metadata in the log, which fundamentally prevents files with mismatched schemas from being created

3. No Updates/Deletes → File-level Write + Logical Deletion

Parquet is immutable, so it doesn’t support row-level updates.

  • Delta’s Solution:

    1. It removes the file(s) containing the rows to be updated or deleted. This is marked as a removeFile action in the _delta_log.
    2. It writes a new Parquet file containing the new data.

    → Delta records both of these steps as a single atomic commit.

This process of “file-level rewrite + logical remove” is bundled into a single transaction

4. No Versioning → Time Travel & Snapshot

Since Delta’s log is effectively a version history, you can revert to any previous version at any time

5. Concurrency Issues → Optimistic Concurrency Control (OCC)

All writers start by reading the current version (N). At commit time, Delta checks for conflicts (e.g., if the same file was modified) and can reject & retry the operation

Delta Lake With Polars

Now, let’s learn how to add, modify, and delete rows using local files and see how it actually works

GitHub - seon99il/deltalake-with-polars — This is an example of how to read and write DeltaLake files in a local environment using Polars …

$ pip install deltalake pyarrow polars

Reading a Delta Lake

You can read data in the same way you would read a standard Parquet file

df = pl.read_delta("delta/users")
df_v0 = pl.read_delta("delta/users", version=0) # Time Travel

Writing Delta Lake Files: Append

This adds newly generated data

Without Delta Lake, if you wanted to add 2 new rows to a file with 100 rows, the typical method would be to overwrite the original file with a new, single file containing 102 rows. However, Delta Lake allows you to track and manage these changes using the _delta_log

df_new_row = pl.DataFrame({
    "name": ["Sumin", "Lee"],
    "user_id": [6, 7],
    "age": [29, 31],
    "score": [88.5, 92.0],
})

df_new_row.write_delta("./delta/users", mode="append")  # Append Mode

New parts, like part-1.snappy.parquet and part-2.snappy.parquet, are added. When the data is read, both files are combined

Modifying Delta Lake Files: Overwrite

This method overwrites the existing data. If the table previously consisted of files 1 and 2, an overwrite operation will ignore files 1 and 2 and read only the new file 3

filtered_df = df.filter(pl.col("age") > 30)
filtered_df.write_delta("delta/users", mode="overwrite")
  • Now, the delta/users directory will contain part-1.snappy.parquet, part-2.snappy.parquet, and part-3.snappy.parquet. On future reads, files 1 and 2 are ignored, and only version 3 (the new file) is read

This is what enables the Time Travel feature

When deleting data, you can use Deletion Vectors to efficiently manage file storage through logical deletion.

What are deletion vectors? | Databricks on AWS — Learn how Databricks leverages deletion vectors to accelerate deletes and updates to data stored in Delta tables.

Modifying Delta Lake Files: Merge Mode

This might seem to produce the same result as overwrite, but its advantages become clear if the source data is partitioned

An overwrite writes a new version (new files) for all partitions, whereas merge only creates new versions for the partitions that have matching data

df.write_delta(
    "./delta/users", # target
    mode="merge",
    delta_merge_options={
        "predicate": "target.user_id = source.user_id",
        "source_alias": "source",
        "target_alias": "target"
    }
).action().execute()
  • predicate: Sets the key to merge on. It targets rows with the same user_id
  • source & target: The DataFrame (df) is referred to as the source
  • action: Defines how to handle data that matches or does not match the key. Multiple action methods can be chained.
new_users_df = pl.DataFrame({
    "user_id": [1, 10, 4],
    "name": ["Alice2", "Bob2", "Charlie2"],
    "age": [30, 25, 40]
})

Merge Mode Action: when_matched_update_all()

Updates all data that matches the key with the source data.

Values in the existing table that don’t match are preserved, but data from the source that doesn’t have a match in the target is not inserted

Meaning, if the target only has users with IDs 1 and 4, the user data for ID 10 (from the source) will be lost (not inserted)

new_users_df.write_delta(
    "./delta/users",
    mode="merge",
    delta_merge_options={
        "predicate": "target.user_id = source.user_id",
        "source_alias": "source",
        "target_alias": "target"
    }
).when_matched_update_all().execute()

Merge Mode Action: when_not_matched_insert()

Inserts data that does not have a matching ID as a new row. If used with the when_matched_update_all action, the user data for ID 10 will not be lost and will be added

new_users_df.write_delta(
    "./delta/users",
    mode="merge",
    delta_merge_options={
        "predicate": "target.user_id = source.user_id",
        "source_alias": "source",
        "target_alias": "target"
    }
).when_matched_update_all().when_not_matched_insert_all().execute()

Merge Mode Action: when_not_matched_by_source

This allows you to modify data in the target that is not present in the source (df)

For example, it changes the name of users in the target other than IDs 1 and 4 to “Unknown”.

new_users_df.write_delta(
    "./delta/users",
    mode="merge",
    delta_merge_options={
        "predicate": "target.user_id = source.user_id",
        "source_alias": "source",
        "target_alias": "target"
    }
).when_not_matched_by_source_update(
    updates={
        "name": "'Unknown'",  # Must use single quotes for string literals
    }
).execute()

Feature With Merge Mode

Requirement: Update data not in the source to have the name “UNKNOWN”, and “UPSERT” the source data (change names for 1 & 4, and add 10)

(new_users_df.write_delta(
    "./delta/users",
    mode="merge",
    delta_merge_options={
        "predicate": "target.user_id = source.user_id",
        "source_alias": "source",
        "target_alias": "target"
    }
).when_not_matched_by_source_update(
    updates={
        "name": "'Unknown'",
    }
).when_not_matched_insert_all()
 .when_matched_update_all().execute()
 )
  • when_not_matched_by_source_update: Change the name for data not in the source
  • when_not_matched_insert_all: Insert data that is not in the target (e.g., ID 10)
  • when_matched_update_all(): Update data that matches the ID with the data from the source (e.g., IDs 1 and 4)

Previous posts

View all posts →

Comments