#data-engineering
Apache Spark Memory Deep Dive: The Betrayal of Broadcast Variables and Resolving OOM
Struggling with PySpark OOMs? We expose the hidden memory duplication of Broadcast Variables and explore the ultimate Hash Join strategy.
Great Expectations (GX): to check your data quality
No matter how sophisticated your dashboards or how advanced your machine learning models are, the output becomes worthless if the underlying data is bad.
Apache Spark Deep Dive: Memory
A walkthrough of Apache Spark's memory layout, the Unified Memory Manager, spills, OOM analysis, and how memory is allocated during shuffles.
Apache Spark Deep Dive: Shuffle
A look at the stages a Spark Shuffle goes through and how the number of partitions changes at each stage.
Delta Lake with Polars: How to handle versioning and ACID on file systems
How Delta Lake works and its key benefits, plus managing Delta files in a local environment using Polars.