Apache Spark Deep Dive: Memory
8 min read

Memory allocation in Apache Spark is a key factor that determines the performance of a Spark job. Spark memory is divided into several components to handle computation, caching, and system overhead.

- Driver Memory: Memory allocated to the driver program, which is responsible for task scheduling and overall management of the Spark job. The Driver is the main process that creates the SparkSession/SparkContext,
submits jobs, and coordinates task execution across Workers. - Executor Memory: Memory allocated to each Executor running on a Worker Node. This is where the actual data processing and storage (caching) take place.
Executor Memory Architecture
When a Spark Executor (JVM) runs, its memory is broadly divided into the JVM Heap, Off-Heap (outside the JVM), and Overhead regions.
JVM Heap Memory (Inside of JVM)
The Unified Memory Manager, introduced in Spark 1.6, does not split memory statically; instead it adjusts the boundaries flexibly as needed. The memory inside an Executor container is divided into three main regions.
Since Spark 1.6.0, the Unified Memory Manager is Spark’s default memory manager. In Spark 1.6 and later, static memory management can be enabled with the spark.memory.useLegacyMode=true parameter.
1. Reserved Memory (Hardcoded to 300MB as of spark 1.6)
Since Spark 1.6 this is hardcoded to 300MB. This space is used neither by the user nor for data processing.
If the heap size is smaller than 1.5 times this reserved memory (450MB), Spark refuses to start and exits.
2. User Memory
This region is not managed directly by Spark. It stores custom objects defined in user code, UDFs, and global variables (such as Broadcast Variables).
- Formula:
(Java Heap — 300MB) X (1 — spark.memory.fraction) - Default: about 40% of the total heap
If this region is shrunk too much, an OOM can occur while running UDFs.
3. Spark Memory (Unified Region)
This is where Spark’s core activity actually happens. It is further divided into two sub-regions that are shared with each other. Default: spark.memory.fraction (default 0.6, i.e. 60%)
- Execution Memory: The space where actual computation such as Join, Shuffle, Aggregation, and Sort is performed. Temporary buffers created during a Shuffle are allocated here.
- Storage Memory: The space where RDDs or DataFrames that have been
cache()d orpersist()ed are stored. It also stores Shuffle data and RDD lineage information to guarantee fault tolerance.
Off-Heap Memory (Outside of JVM)
Spark actively uses memory outside the JVM heap to avoid GC (Garbage Collection) overhead.
- Configuration: Enabled with
spark.memory.offHeap.enabled=trueandspark.memory.offHeap.size. - Tungsten Engine: Tungsten, Spark’s optimization engine, converts data into binary form rather than Java objects and stores it directly off-heap. It is not affected by GC and offers C++-level memory management efficiency.
Off-Heap memory is by default used for serialized data storage and network transfer buffers, but when you enable spark.memory.offHeap.enabled=true, even the execution buffers of the Shuffle & Sorting stage are allocated in the Off-Heap region, which eliminates GC overhead.
How the Unified Memory Manager Works
In the Unified Memory architecture, the Execution and Storage regions share a single memory pool without a hard partition, following a dynamic occupancy mechanism in which each flexibly borrows the other’s idle resources according to runtime workload demand.
Dynamic Occupancy Policy
The Unified Memory Manager maximizes memory efficiency through mutual borrowing and preemption between the Execution and Storage regions.
- Dynamic Borrowing: If the Execution region has idle space, Storage borrows it for caching; conversely, Execution occupies Storage’s idle space for computation.
- Preemptive Eviction in favor of Execution: If memory runs short during computation (Execution), the memory blocks that Storage has borrowed are immediately evicted to free up space.
Losing cached data only incurs recomputation cost, whereas losing data that is being computed leads directly to Task failure. Spark therefore puts computational stability first.
- Immunity Threshold for Storage: The region set by
spark.memory.storageFraction(default 0.5) is treated as an Immune Zone. Cached data stored within this threshold is not forcibly evicted even when Execution memory is short.
How Disk Spill Occurs and What It Costs
When a Task runs, each thread (Active Slot) occupies Execution Memory to process its partition. When available memory reaches a threshold, the Spill (to disk) mechanism is triggered.
The Spill Process (Step-by-Step)
If Execution Memory is 500MB and there are 2 running cores (
N = 2), each task must operate within a memory range from at least 125MB (500 / 4 (2N)) to at most 250MB (500 / 2).
If a task cannot secure the minimum memory, it waits until space becomes available via
acquireMemory().
- In-Memory Buffer allocation: When performing operations such as
SortMergeJoin, Spark loads data into in-memory page buffers such asExternalAppendOnlyMaporExternalSorter. - Resource request and shortage detection: When the buffer becomes saturated, it asks the
MemoryManagerfor an additional page allocation (acquireMemory()). - Spill Trigger: When the available space in Unified Memory is exhausted and there is no space left to evict from Storage, a Spill is triggered.
- Serialization & I/O Overhead: The data resident in memory is sorted, serialized, and written to a temporary directory on the local disk (
spark.local.dir).- Performance impact: this causes CPU load from serialization and Disk I/O latency at the same time, so processing speed drops sharply.
- Merge Sort: When the Task completes, the spill files scattered on disk and the data remaining in memory are read back and merge-sorted.
Spill is a defense mechanism against OOM (Out Of Memory), but excessive spilling can create a serious bottleneck. You can monitor the Spill (Memory) and Spill (Disk) metrics in the Stage tab of the Spark UI.
OOM Analysis and Performance Optimization
Case 1: Executor OOM
java.lang.OutOfMemoryError: Java heap space
- Cause A (Data Skew): Data is concentrated on specific partition keys, so the amount of data a single Task must process exceeds the Executor’s heap memory limit.
- → Apply key salting, or redistribute the data with
repartition().
- → Apply key salting, or redistribute the data with
- Cause B (High Memory Footprint): Many expensive objects (large Maps, Lists, etc.) are created in the User Memory region.
- → Optimize application logic, or lower
spark.memory.fractionto free up User Memory space.
- → Optimize application logic, or lower
Case 2: Container killed by YARN for exceeding memory limits
- Cause: Off-Heap Overhead used outside the JVM Heap (Netty buffers, thread stacks, etc.) exceeds the container’s configured limit.
- → Raise the
spark.executor.memoryOverheadparameter.
- → Raise the
Rule: Max(spark.executor.memory * 0.10, 384MB)
Secure headroom with something like --conf spark.executor.memoryOverhead=2048.
Case 3: Driver OOM
- Cause: A
collect()call sends a large result set from the Executors to the Driver all at once.- → Use
toLocalIterator()instead ofcollect()to stream the data. - → Change the
spark.driver.maxResultSizesetting (default 1GB). - → Increase Driver memory resources (
spark.driver.memory).
- → Use
Case 4: Increased Latency from Excessive Spill
- Cause: There are too few partitions for the data size, so the size per partition exceeds the Executor’s memory capacity.
- → Raise
spark.sql.shuffle.partitions(default 200) to keep the data size per partition at a reasonable level (128MB~200MB).
- → Raise
Memory Occupation and Allocation During a Shuffle
Shuffling is broadly divided into two phases: Shuffle Write (Map Side) and Shuffle Read (Reduce Side). In this process Spark uses the Execution Memory region, and the Tungsten engine’s page-based management comes fully into play.
1. Map Side: Memory Allocation in the Shuffle Write Phase
After processing data, a Map Task must classify and sort it by partition so that the Reducers of the next stage can fetch it. The In-Memory Buffer plays the key role in this process.
1-1. Data Structures: PartitionedAppendOnlyMap & PartitionedPairBuffer
Spark does not write incoming records to disk immediately; it first fills a memory buffer. Depending on the type of operation, it is allocated one of two data structures.
PartitionedAppendOnlyMap: Used when Aggregation is needed, such asreduceByKeyoraggregateByKey. Like a hash table, it updates the value when the Key is the same, using memory efficiently.PartitionedPairBuffer: Used when only simple sorting or grouping is needed, such assortByKeyorgroupByKey. It is simply an array that stacks data one after another.
1-2. How Memory Is Occupied (Pointer + Page)
These buffers don’t simply store Java objects; they occupy memory the Tungsten way.
- Pointer Array:
- Allocates a
Longarray that stores only the data’s address (pointer) and the partition ID. - When sorting is needed, instead of moving the actual heavy data, it speeds things up by quick-sorting only this lightweight pointer array.
- Allocates a
- Data Pages:
- The actual Key and Value data are written contiguously, in binary form, into memory blocks allocated in OS Page units (64MB by default, etc.).
- This prevents memory fragmentation and improves CPU cache locality.
1-3. Allocation and the Spill Threshold
As data keeps coming in, the memory manager repeats the following process.
- Check: Check whether the size of the current buffer (Pointer Array + Data Pages) has exceeded the threshold.
- Request: Ask the
UnifiedMemoryManagerfor additional Execution Memory (acquireMemory()). - Spill: If the request is rejected (out of memory) or the pointer array inside the collection reaches its limit, a Spill is triggered.
- The data in memory is sorted by partition ID and written to disk, and the memory is freed.
2. Reduce Side: Memory Allocation in the Shuffle Read Phase
A Reducer fetches, over the network, the data blocks of the partitions assigned to it from multiple Mappers.
2-1. Fetch & Buffer
- ExternalAppendOnlyMap: If the Reduce stage also needs to aggregate or sort data, it allocates an
ExternalAppendOnlyMapsimilar to the one on the Map Side. - Network buffer: Data arriving over the network first passes through a network buffer (Netty Buffer, Off-Heap) and is then copied into the Executor’s Heap (or Off-Heap) Execution Memory.
2-2. Merge Sort and Memory
It would be ideal if all the data could fit in memory without spilling, but for large workloads a Memory-Disk hybrid approach kicks in.
- In-Memory Merge: Data blocks fetched from Mappers are put into an in-memory map while being aggregated (merged) in real time.
- Spill to Disk: As on the Map Side, when
Execution Memoryis full, the contents of the map aggregated so far are spilled to disk. - Global Merge: Finally, the multiple files spilled to disk and the data currently remaining in memory are merged into one.