By date
December
Apache Spark Deep Dive: repartition & coalesce
The difference between repartition and coalesce, the two ways to change the number of partitions in Spark, and how each works.
Apache Spark Deep Dive: Shuffle
A look at the stages a Spark Shuffle goes through and how the number of partitions changes at each stage.
November
Airflow TriggerRule & Watcher Pattern
Sorting out the confusing Trigger_Rule, and solving a problem it causes with the Watcher Pattern.
Airflow Best Practices
A translated summary of the Best Practices section of the official Airflow documentation.
Airflow UnitTest with DAG BAG
Explains DAG_BAG-based Airflow unit testing and CI setup with GitHub Actions.
Delta Lake with Polars: How to handle versioning and ACID on file systems
How Delta Lake works and its key benefits, plus managing Delta files in a local environment using Polars.
Column-oriented Store: What's The Benefit?
A detailed look at row vs column in-memory layouts, the internals of Apache Arrow, Parquet, and Polars, and what actually happens when data is modified.
Airflow Scheduling and How Execution Date Works
A summary of how Execution Date differs between DAGs run by the scheduler and DAGs run by backfill or trigger in Apache Airflow.
October
August
Python's with Statement and Context Managers: The Elegant Alternative to try-finally
You probably use with open(...) all the time without a second thought. But have you ever wondered how the file gets closed automatically?
List vs Generator in Python: Why Use Generators When Both Work in a For Loop?
Lists and generators look interchangeable in a for loop, but they differ fundamentally in evaluation strategy and memory usage.