Data Engineering
Apache Spark Memory Deep Dive: The Betrayal of Broadcast Variables and Resolving OOM
Struggling with PySpark OOMs? We expose the hidden memory duplication of Broadcast Variables and explore the ultimate Hash Join strategy.
Great Expectations (GX): to check your data quality
No matter how sophisticated your dashboards or how advanced your machine learning models are, the output becomes worthless if the underlying data is bad.
Apache Spark Deep Dive: Memory
A walkthrough of Apache Spark's memory layout, the Unified Memory Manager, spills, OOM analysis, and how memory is allocated during shuffles.
Apache Spark Deep Dive: repartition & coalesce
The difference between repartition and coalesce, the two ways to change the number of partitions in Spark, and how each works.
Apache Spark Deep Dive: Shuffle
A look at the stages a Spark Shuffle goes through and how the number of partitions changes at each stage.
Airflow TriggerRule & Watcher Pattern
Sorting out the confusing Trigger_Rule, and solving a problem it causes with the Watcher Pattern.
Airflow Best Practices
A translated summary of the Best Practices section of the official Airflow documentation.
Delta Lake with Polars: How to handle versioning and ACID on file systems
How Delta Lake works and its key benefits, plus managing Delta files in a local environment using Polars.
Column-oriented Store: What's The Benefit?
A detailed look at row vs column in-memory layouts, the internals of Apache Arrow, Parquet, and Polars, and what actually happens when data is modified.
Airflow Scheduling and How Execution Date Works
A summary of how Execution Date differs between DAGs run by the scheduler and DAGs run by backfill or trigger in Apache Airflow.