seon99il

Data Engineering

Apache Spark Deep Dive: Memory

8 min read

Memory allocation in Apache Spark is a key factor that determines the performance of a Spark job. Spark memory is divided into several components to handle computation, caching, and system overhead.

  • Driver Memory: Memory allocated to the driver program, which is responsible for task scheduling and overall management of the Spark job. The Driver is the main process that creates the SparkSession/SparkContext, submits jobs, and coordinates task execution across Workers.
  • Executor Memory: Memory allocated to each Executor running on a Worker Node. This is where the actual data processing and storage (caching) take place.

Executor Memory Architecture

When a Spark Executor (JVM) runs, its memory is broadly divided into the JVM Heap, Off-Heap (outside the JVM), and Overhead regions.

JVM Heap Memory (Inside of JVM)

The Unified Memory Manager, introduced in Spark 1.6, does not split memory statically; instead it adjusts the boundaries flexibly as needed. The memory inside an Executor container is divided into three main regions.

Since Spark 1.6.0, the Unified Memory Manager is Spark’s default memory manager. In Spark 1.6 and later, static memory management can be enabled with the spark.memory.useLegacyMode=true parameter.

1. Reserved Memory (Hardcoded to 300MB as of spark 1.6)

Since Spark 1.6 this is hardcoded to 300MB. This space is used neither by the user nor for data processing.

If the heap size is smaller than 1.5 times this reserved memory (450MB), Spark refuses to start and exits.

2. User Memory

This region is not managed directly by Spark. It stores custom objects defined in user code, UDFs, and global variables (such as Broadcast Variables).

  • Formula: (Java Heap — 300MB) X (1 — spark.memory.fraction)
  • Default: about 40% of the total heap

If this region is shrunk too much, an OOM can occur while running UDFs.

3. Spark Memory (Unified Region)

This is where Spark’s core activity actually happens. It is further divided into two sub-regions that are shared with each other. Default: spark.memory.fraction (default 0.6, i.e. 60%)

  • Execution Memory: The space where actual computation such as Join, Shuffle, Aggregation, and Sort is performed. Temporary buffers created during a Shuffle are allocated here.
  • Storage Memory: The space where RDDs or DataFrames that have been cache()d or persist()ed are stored. It also stores Shuffle data and RDD lineage information to guarantee fault tolerance.

Off-Heap Memory (Outside of JVM)

Spark actively uses memory outside the JVM heap to avoid GC (Garbage Collection) overhead.

  • Configuration: Enabled with spark.memory.offHeap.enabled=true and spark.memory.offHeap.size.
  • Tungsten Engine: Tungsten, Spark’s optimization engine, converts data into binary form rather than Java objects and stores it directly off-heap. It is not affected by GC and offers C++-level memory management efficiency.

Off-Heap memory is by default used for serialized data storage and network transfer buffers, but when you enable spark.memory.offHeap.enabled=true, even the execution buffers of the Shuffle & Sorting stage are allocated in the Off-Heap region, which eliminates GC overhead.

How the Unified Memory Manager Works

In the Unified Memory architecture, the Execution and Storage regions share a single memory pool without a hard partition, following a dynamic occupancy mechanism in which each flexibly borrows the other’s idle resources according to runtime workload demand.

Dynamic Occupancy Policy

The Unified Memory Manager maximizes memory efficiency through mutual borrowing and preemption between the Execution and Storage regions.

  • Dynamic Borrowing: If the Execution region has idle space, Storage borrows it for caching; conversely, Execution occupies Storage’s idle space for computation.
  • Preemptive Eviction in favor of Execution: If memory runs short during computation (Execution), the memory blocks that Storage has borrowed are immediately evicted to free up space.

Losing cached data only incurs recomputation cost, whereas losing data that is being computed leads directly to Task failure. Spark therefore puts computational stability first.

  • Immunity Threshold for Storage: The region set by spark.memory.storageFraction (default 0.5) is treated as an Immune Zone. Cached data stored within this threshold is not forcibly evicted even when Execution memory is short.

How Disk Spill Occurs and What It Costs

When a Task runs, each thread (Active Slot) occupies Execution Memory to process its partition. When available memory reaches a threshold, the Spill (to disk) mechanism is triggered.

The Spill Process (Step-by-Step)

If Execution Memory is 500MB and there are 2 running cores (N = 2), each task must operate within a memory range from at least 125MB (500 / 4 (2N)) to at most 250MB (500 / 2).

If a task cannot secure the minimum memory, it waits until space becomes available via acquireMemory().

  1. In-Memory Buffer allocation: When performing operations such as SortMergeJoin, Spark loads data into in-memory page buffers such as ExternalAppendOnlyMap or ExternalSorter.
  2. Resource request and shortage detection: When the buffer becomes saturated, it asks the MemoryManager for an additional page allocation (acquireMemory()).
  3. Spill Trigger: When the available space in Unified Memory is exhausted and there is no space left to evict from Storage, a Spill is triggered.
  4. Serialization & I/O Overhead: The data resident in memory is sorted, serialized, and written to a temporary directory on the local disk (spark.local.dir).
    • Performance impact: this causes CPU load from serialization and Disk I/O latency at the same time, so processing speed drops sharply.
  5. Merge Sort: When the Task completes, the spill files scattered on disk and the data remaining in memory are read back and merge-sorted.

Spill is a defense mechanism against OOM (Out Of Memory), but excessive spilling can create a serious bottleneck. You can monitor the Spill (Memory) and Spill (Disk) metrics in the Stage tab of the Spark UI.

OOM Analysis and Performance Optimization

Case 1: Executor OOM

java.lang.OutOfMemoryError: Java heap space

  • Cause A (Data Skew): Data is concentrated on specific partition keys, so the amount of data a single Task must process exceeds the Executor’s heap memory limit.
    • → Apply key salting, or redistribute the data with repartition().
  • Cause B (High Memory Footprint): Many expensive objects (large Maps, Lists, etc.) are created in the User Memory region.
    • → Optimize application logic, or lower spark.memory.fraction to free up User Memory space.

Case 2: Container killed by YARN for exceeding memory limits

  • Cause: Off-Heap Overhead used outside the JVM Heap (Netty buffers, thread stacks, etc.) exceeds the container’s configured limit.
    • → Raise the spark.executor.memoryOverhead parameter.

Rule: Max(spark.executor.memory * 0.10, 384MB)

Secure headroom with something like --conf spark.executor.memoryOverhead=2048.

Case 3: Driver OOM

  • Cause: A collect() call sends a large result set from the Executors to the Driver all at once.
    • → Use toLocalIterator() instead of collect() to stream the data.
    • → Change the spark.driver.maxResultSize setting (default 1GB).
    • → Increase Driver memory resources (spark.driver.memory).

Case 4: Increased Latency from Excessive Spill

  • Cause: There are too few partitions for the data size, so the size per partition exceeds the Executor’s memory capacity.
    • → Raise spark.sql.shuffle.partitions (default 200) to keep the data size per partition at a reasonable level (128MB~200MB).

Memory Occupation and Allocation During a Shuffle

Shuffling is broadly divided into two phases: Shuffle Write (Map Side) and Shuffle Read (Reduce Side). In this process Spark uses the Execution Memory region, and the Tungsten engine’s page-based management comes fully into play.

1. Map Side: Memory Allocation in the Shuffle Write Phase

After processing data, a Map Task must classify and sort it by partition so that the Reducers of the next stage can fetch it. The In-Memory Buffer plays the key role in this process.

1-1. Data Structures: PartitionedAppendOnlyMap & PartitionedPairBuffer

Spark does not write incoming records to disk immediately; it first fills a memory buffer. Depending on the type of operation, it is allocated one of two data structures.

  • PartitionedAppendOnlyMap: Used when Aggregation is needed, such as reduceByKey or aggregateByKey. Like a hash table, it updates the value when the Key is the same, using memory efficiently.
  • PartitionedPairBuffer: Used when only simple sorting or grouping is needed, such as sortByKey or groupByKey. It is simply an array that stacks data one after another.

1-2. How Memory Is Occupied (Pointer + Page)

These buffers don’t simply store Java objects; they occupy memory the Tungsten way.

  1. Pointer Array:
    • Allocates a Long array that stores only the data’s address (pointer) and the partition ID.
    • When sorting is needed, instead of moving the actual heavy data, it speeds things up by quick-sorting only this lightweight pointer array.
  2. Data Pages:
    • The actual Key and Value data are written contiguously, in binary form, into memory blocks allocated in OS Page units (64MB by default, etc.).
    • This prevents memory fragmentation and improves CPU cache locality.

1-3. Allocation and the Spill Threshold

As data keeps coming in, the memory manager repeats the following process.

  1. Check: Check whether the size of the current buffer (Pointer Array + Data Pages) has exceeded the threshold.
  2. Request: Ask the UnifiedMemoryManager for additional Execution Memory (acquireMemory()).
  3. Spill: If the request is rejected (out of memory) or the pointer array inside the collection reaches its limit, a Spill is triggered.
  4. The data in memory is sorted by partition ID and written to disk, and the memory is freed.

2. Reduce Side: Memory Allocation in the Shuffle Read Phase

A Reducer fetches, over the network, the data blocks of the partitions assigned to it from multiple Mappers.

2-1. Fetch & Buffer

  • ExternalAppendOnlyMap: If the Reduce stage also needs to aggregate or sort data, it allocates an ExternalAppendOnlyMap similar to the one on the Map Side.
  • Network buffer: Data arriving over the network first passes through a network buffer (Netty Buffer, Off-Heap) and is then copied into the Executor’s Heap (or Off-Heap) Execution Memory.

2-2. Merge Sort and Memory

It would be ideal if all the data could fit in memory without spilling, but for large workloads a Memory-Disk hybrid approach kicks in.

  1. In-Memory Merge: Data blocks fetched from Mappers are put into an in-memory map while being aggregated (merged) in real time.
  2. Spill to Disk: As on the Map Side, when Execution Memory is full, the contents of the map aggregated so far are spilled to disk.
  3. Global Merge: Finally, the multiple files spilled to disk and the data currently remaining in memory are merged into one.

Previous posts

View all posts →

Comments