#apache-spark
Apache Spark Memory Deep Dive: The Betrayal of Broadcast Variables and Resolving OOM
Struggling with PySpark OOMs? We expose the hidden memory duplication of Broadcast Variables and explore the ultimate Hash Join strategy.
Apache Spark Deep Dive: Memory
A walkthrough of Apache Spark's memory layout, the Unified Memory Manager, spills, OOM analysis, and how memory is allocated during shuffles.
Apache Spark Deep Dive: repartition & coalesce
The difference between repartition and coalesce, the two ways to change the number of partitions in Spark, and how each works.
Apache Spark Deep Dive: Shuffle
A look at the stages a Spark Shuffle goes through and how the number of partitions changes at each stage.