Delta Lake with Polars: How to handle versioning and ACID on file systems
7 min read

Today, I’ll explore how Delta Lake works and its key benefits. We’ll also walk through managing Delta files in a local environment using Polars.
Disadvantages of Existing Systems Before Delta Lake
1. Lack of ACID Transactions
Using file formats like Parquet or CSV alone can lead to write failures, duplicate writes, or read-write contention
- During appends, multiple writes to the same file can create duplicate data
- If another process reads at the same time, it risks reading a partial (incomplete) write
2. Problems with Schema Evolution / Enforcement
Traditional data lakes struggled with schema changes and management
- Changing the schema required regenerating the entire Parquet file
- If files with different schemas were mixed, the read engine would break
- Engines that couldn’t process added columns would fail
3. Difficulty with Upserts and Deletes
In traditional Parquet-based systems, modification operations were virtually impossible. To change just one row in a Parquet file with 100 rows, you had to rewrite all 100 rows
- Parquet is immutable → partial row deletes/updates are not possible
- Solution: “Rewrite the entire file” → highly inefficient
- Risk of conflicts if files are rewritten concurrently in a parallel environment
4. No Data Versioning & Time Travel
Time Travel — the ability to ask “What did the data look like yesterday?”, “I need to revert to a previous version,” or “Track which job corrupted the data” — was not possible
Delta Lake
Delta Lake is a storage format and a transaction layer. It provides ACID transactions on top of your data lake. It offers performance advantages and Versioning (Time Travel) for modifications (updates, inserts, deletes) to data that is stored in an immutable format (like Parquet)
Delta Lake Structure
.
└──── delta # Delta Lake directory
└── users # Delta Table
├── _delta_log
├── part-00000-....snappy.parquet
└── part-00000-....snappy.parquet
As shown above, Delta Lake records transaction logs in JSON files inside the _delta_log directory. Furthermore, it can be structured as follows (often called a Medallion Architecture):
S3 / HDFS / Blob Storage
├── Raw Zone (Original source, append only)
│ └── users
│ ├── _delta_log
│ └── part-00000-....snappy.parquet
│
├── Bronze Zone (Minimal processing)
├── Silver Zone (Cleansed & normalized data)
└── Gold Zone (BI-optimized aggregates)
Now, let’s explore how Delta Lake solves the problems we discussed earlier.
How Delta Lake Handles CRUD Operations
1. No ACID Guarantee → Solved by Delta’s Transaction Log
Delta implements ACID by recording the following in the
_delta_logdirectory
Existing Problem: S3/HDFS are file systems and lack atomic writes, rollback, and consistency
Delta Lake’s Solution:
- Each commit is one JSON file = 1 transaction (e.g.,
000000000000000010.json) - When a write finishes, the commit file is atomically renamed. → The
renameoperation is atomic in HDFS/S3 - The log includes information on file additions/deletions, data statistics, etc.
Read engines like Spark, Polars, and DuckDB read this log and combine the files to construct the most recent version of the table
2. Mixed Schemas, Broken Data → Schema Enforcement/Evolution
Schema Enforcement: The write operation fails if the schema does not match. Schema Evolution: Allows schema changes (e.g.,
mergeSchema=true).
- Adding columns
- Different column order → automatically matched
- Supports changes to
nullableproperties, etc.
Delta records file metadata in the log, which fundamentally prevents files with mismatched schemas from being created
3. No Updates/Deletes → File-level Write + Logical Deletion
Parquet is immutable, so it doesn’t support row-level updates.
-
Delta’s Solution:
- It removes the file(s) containing the rows to be updated or deleted. This is marked as a
removeFileaction in the_delta_log. - It writes a new Parquet file containing the new data.
→ Delta records both of these steps as a single atomic commit.
- It removes the file(s) containing the rows to be updated or deleted. This is marked as a
This process of “file-level rewrite + logical remove” is bundled into a single transaction
4. No Versioning → Time Travel & Snapshot
Since Delta’s log is effectively a version history, you can revert to any previous version at any time
5. Concurrency Issues → Optimistic Concurrency Control (OCC)
All writers start by reading the current version (N). At commit time, Delta checks for conflicts (e.g., if the same file was modified) and can reject & retry the operation
Delta Lake With Polars
Now, let’s learn how to add, modify, and delete rows using local files and see how it actually works
$ pip install deltalake pyarrow polars
Reading a Delta Lake
You can read data in the same way you would read a standard Parquet file
df = pl.read_delta("delta/users")
df_v0 = pl.read_delta("delta/users", version=0) # Time Travel
Writing Delta Lake Files: Append
This adds newly generated data
Without Delta Lake, if you wanted to add 2 new rows to a file with 100 rows, the typical method would be to overwrite the original file with a new, single file containing 102 rows. However, Delta Lake allows you to track and manage these changes using the _delta_log
df_new_row = pl.DataFrame({
"name": ["Sumin", "Lee"],
"user_id": [6, 7],
"age": [29, 31],
"score": [88.5, 92.0],
})
df_new_row.write_delta("./delta/users", mode="append") # Append Mode
New parts, like part-1.snappy.parquet and part-2.snappy.parquet, are added. When the data is read, both files are combined
Modifying Delta Lake Files: Overwrite
This method overwrites the existing data. If the table previously consisted of files 1 and 2, an
overwriteoperation will ignore files 1 and 2 and read only the new file 3
filtered_df = df.filter(pl.col("age") > 30)
filtered_df.write_delta("delta/users", mode="overwrite")
- Now, the
delta/usersdirectory will containpart-1.snappy.parquet,part-2.snappy.parquet, andpart-3.snappy.parquet. On future reads, files 1 and 2 are ignored, and only version 3 (the new file) is read
This is what enables the Time Travel feature
When deleting data, you can use Deletion Vectors to efficiently manage file storage through logical deletion.
Modifying Delta Lake Files: Merge Mode
This might seem to produce the same result as
overwrite, but its advantages become clear if the source data is partitioned
An overwrite writes a new version (new files) for all partitions, whereas merge only creates new versions for the partitions that have matching data
df.write_delta(
"./delta/users", # target
mode="merge",
delta_merge_options={
"predicate": "target.user_id = source.user_id",
"source_alias": "source",
"target_alias": "target"
}
).action().execute()
- predicate: Sets the key to merge on. It targets rows with the same
user_id - source & target: The DataFrame (
df) is referred to as thesource - action: Defines how to handle data that matches or does not match the key. Multiple action methods can be chained.
new_users_df = pl.DataFrame({
"user_id": [1, 10, 4],
"name": ["Alice2", "Bob2", "Charlie2"],
"age": [30, 25, 40]
})
Merge Mode Action: when_matched_update_all()
Updates all data that matches the key with the
sourcedata.
Values in the existing table that don’t match are preserved, but data from the source that doesn’t have a match in the target is not inserted
Meaning, if the target only has users with IDs 1 and 4, the user data for ID 10 (from the source) will be lost (not inserted)
new_users_df.write_delta(
"./delta/users",
mode="merge",
delta_merge_options={
"predicate": "target.user_id = source.user_id",
"source_alias": "source",
"target_alias": "target"
}
).when_matched_update_all().execute()
Merge Mode Action: when_not_matched_insert()
Inserts data that does not have a matching ID as a new row. If used with the
when_matched_update_allaction, the user data for ID 10 will not be lost and will be added
new_users_df.write_delta(
"./delta/users",
mode="merge",
delta_merge_options={
"predicate": "target.user_id = source.user_id",
"source_alias": "source",
"target_alias": "target"
}
).when_matched_update_all().when_not_matched_insert_all().execute()
Merge Mode Action: when_not_matched_by_source
This allows you to modify data in the target that is not present in the source (
df)
For example, it changes the name of users in the target other than IDs 1 and 4 to “Unknown”.
new_users_df.write_delta(
"./delta/users",
mode="merge",
delta_merge_options={
"predicate": "target.user_id = source.user_id",
"source_alias": "source",
"target_alias": "target"
}
).when_not_matched_by_source_update(
updates={
"name": "'Unknown'", # Must use single quotes for string literals
}
).execute()
Feature With Merge Mode
Requirement: Update data not in the source to have the name “UNKNOWN”, and “UPSERT” the source data (change names for 1 & 4, and add 10)
(new_users_df.write_delta(
"./delta/users",
mode="merge",
delta_merge_options={
"predicate": "target.user_id = source.user_id",
"source_alias": "source",
"target_alias": "target"
}
).when_not_matched_by_source_update(
updates={
"name": "'Unknown'",
}
).when_not_matched_insert_all()
.when_matched_update_all().execute()
)
when_not_matched_by_source_update: Change the name for data not in the sourcewhen_not_matched_insert_all: Insert data that is not in the target (e.g., ID 10)when_matched_update_all(): Update data that matches the ID with the data from the source (e.g., IDs 1 and 4)