seon99il

Data Engineering

Apache Spark Memory Deep Dive: The Betrayal of Broadcast Variables and Resolving OOM

12 min read

The Catalyst: How Simple Logic Caused a Pipeline Bottleneck

Recently, while building a data pipeline based on Apache Spark, I had to develop logic to extract and aggregate non-standardized country names (e.g., Korea, Ko, ko) into standardized country codes (e.g., Korea).

The logic itself was incredibly simple. It involved defining mapping rules as a Python dictionary using a library, and replacing the values via a UDF (.map). However, as the volume of data grew, this simple dictionary mapping logic became a severe bottleneck, hogging Executor memory and causing massive performance degradation.

This led to a lingering question: “Broadcasted variables are stored once per Executor in the Storage Memory area. But since they are deserialized when used by Python workers, doesn’t the data get duplicated in memory as many times as the number of workers?”

To resolve and verify this, I brainstormed three different approaches for passing dictionary data to the Executors:

  1. Encapsulating as a Python Closure variable and calling it inside .map
  2. Explicitly passing it as a Spark Broadcast Variable.
  3. Converting the mapping rules into a DataFrame and performing a Broadcast Hash Join (BHJ) at the JVM level.

In this post, I will share how I set up hypotheses for these three methods and dug into the internal logs to understand how data is serialized/deserialized between the Driver and Executors in a PySpark environment, and how memory is ultimately consumed.

You can find the full code at the bottom of this post.

Test Environment

  • Spark: 3.5.1 (PySpark)
  • Python: 3.12
  • Cluster: local-cluster[2,2,1024] (2 Executors, 2 Cores, processing 16 partitions)

Background Knowledge: PySpark’s Execution Structure and Memory Sharing Limits

Because PySpark’s Executor processes are JVM-based, they cannot execute Python code directly. Therefore, Executors must exchange data with separate Python Worker processes via local socket communication.

PySpark Python UDF Workflow

  1. When the JVM Executor receives a task, it checks if a pyspark.daemon process exists on the node.
  2. The Executor requests the daemon to spawn a worker. The pyspark.daemon calls Linux’s os.fork() to clone itself and create a Python Worker Process.
  • Here, I formulated a hypothesis: If we pass a massive dictionary to the Python worker, can we benefit from Linux’s COW (Copy-On-Write) and share the memory?
  • To give you the conclusion first: it is impossible. The pyspark.daemon does not hold the actual data at the time it forks. The data is individually loaded only after the worker has been created.

So, how does the memory footprint of the Python worker change depending on the method used to pass the data? To find out, I wrote a custom class (Tracker) to trace memory usage (RSS) and Object IDs, and began my verification.

Hypothesis Test 1: The Memory Lifecycle of Python Closure Variables

This is the most common anti-pattern: referencing an external variable (Closure). When a globally declared large dictionary or object is directly referenced inside a map function, how does Spark handle it?

spark = (
    SparkSession.builder.master("local-cluster[2,2,1024]")
    .appName("ClosureTest")
    .config("spark.python.worker.reuse", "true")
    .getOrCreate()
)
sc = spark.sparkContext
closure_obj = Tracker("Closure", payload_size_mb=100)

...

def task(partition_idx, iterator):
    before = rss_mb()
    obj = closure_obj
    obj.counter += 1
    after = rss_mb()
    
    list(iterator)
    
    return [
        {
          ...
        }
    ]

result = rdd.mapPartitionsWithIndex(task).collect()

At first, I suspected, “When hundreds of Tasks (partitions) are executed, does the Driver send the variable via RPC communication every single time?” However, after digging into Spark’s DAGScheduler logs, my assumption was only half right.

# Tracker Log

    DRIVER
    pid=2077
    host=Seonggils-MacBook-Pro.local

            CREATE
            mode=Closure
            pid=2077
            host=Seonggils-MacBook-Pro.local
            obj_id=4528186352

            SERIALIZE
            mode=Closure
            serialize_idx=1
            pid=2077
            host=Seonggils-MacBook-Pro.local
            obj_id=4528186352
            size=100.00MB

...

# DAGScheduler Log
# --- Executor ---

INFO BlockManagerInfo: Added broadcast_0_python on disk on <ip>:50002 (size: 100.0 MiB)
INFO BlockManagerInfo: Added broadcast_0_python on disk on <ip>:50003 (size: 100.0 MiB)

To prevent Driver network bottlenecks caused by sending thousands of tasks via RPC, Spark optimizes this by creating a Broadcast variable out of the serialized byte array of the Task and Closure (taskBinary), sending it to each Executor (JVM) exactly once.

However, the real problem started in the Python Worker process.

# Closure Result

================================================================================
RESULT
================================================================================
{'partition': 0, 'pid': 9044, 'object_id': 4452912544, 'counter': 1, 'rss_before': 234.15625, 'rss_after': 234.21875, 'rss_delta': 0.0625}
{'partition': 1, 'pid': 9045, 'object_id': 4452912544, 'counter': 1, 'rss_before': 234.21875, 'rss_after': 234.265625, 'rss_delta': 0.046875}
{'partition': 2, 'pid': 9040, 'object_id': 4398518144, 'counter': 1, 'rss_before': 234.1875, 'rss_after': 234.25, 'rss_delta': 0.0625}
{'partition': 3, 'pid': 9041, 'object_id': 4398518144, 'counter': 1, 'rss_before': 234.21875, 'rss_after': 234.28125, 'rss_delta': 0.0625}
{'partition': 4, 'pid': 9042, 'object_id': 4555782432, 'counter': 1, 'rss_before': 234.1875, 'rss_after': 234.234375, 'rss_delta': 0.046875}
{'partition': 5, 'pid': 9043, 'object_id': 4454748032, 'counter': 1, 'rss_before': 234.21875, 'rss_after': 234.265625, 'rss_delta': 0.046875}
{'partition': 6, 'pid': 9041, 'object_id': 4506131472, 'counter': 1, 'rss_before': 272.328125, 'rss_after': 272.328125, 'rss_delta': 0.0}
{'partition': 7, 'pid': 9040, 'object_id': 4506131424, 'counter': 1, 'rss_before': 272.40625, 'rss_after': 272.40625, 'rss_delta': 0.0}
{'partition': 8, 'pid': 9041, 'object_id': 4506128976, 'counter': 1, 'rss_before': 272.765625, 'rss_after': 272.765625, 'rss_delta': 0.0}
{'partition': 9, 'pid': 9040, 'object_id': 4506128928, 'counter': 1, 'rss_before': 272.890625, 'rss_after': 272.890625, 'rss_delta': 0.0}
{'partition': 10, 'pid': 9040, 'object_id': 4506130752, 'counter': 1, 'rss_before': 273.0, 'rss_after': 273.0, 'rss_delta': 0.0}
{'partition': 11, 'pid': 9041, 'object_id': 4506128112, 'counter': 1, 'rss_before': 272.859375, 'rss_after': 272.859375, 'rss_delta': 0.0}
{'partition': 12, 'pid': 9040, 'object_id': 4506130176, 'counter': 1, 'rss_before': 273.15625, 'rss_after': 273.15625, 'rss_delta': 0.0}
{'partition': 13, 'pid': 9041, 'object_id': 4506130560, 'counter': 1, 'rss_before': 272.859375, 'rss_after': 272.859375, 'rss_delta': 0.0}
{'partition': 14, 'pid': 9040, 'object_id': 4506129552, 'counter': 1, 'rss_before': 273.328125, 'rss_after': 273.328125, 'rss_delta': 0.0}
{'partition': 15, 'pid': 9041, 'object_id': 4506130704, 'counter': 1, 'rss_before': 272.859375, 'rss_after': 272.859375, 'rss_delta': 0.0}

Even though worker reuse is enabled (spark.python.worker.reuse=true) and multiple partitions are processed on the same PID (9040), the Object ID changes for every single partition, and the Counter is always initialized to 1.

The reason lies in PySpark’s deserialization mechanism. Every time a Task is assigned from the JVM to the Python Worker, the worker uses cloudpickle.loads() to newly unpickle (deserialize) the received closure byte array and load it into memory. In short, if you have a 300MB closure and 50 partitions (Tasks) to process, the Python worker creates and discards that 300MB object 50 times. This is the primary culprit behind extreme Garbage Collection (GC) overhead and OOM errors.

Hypothesis Test 2: Broadcast Variable — Caching Logic at the Worker Level

So, how does the Worker’s behavior change if we explicitly use a Broadcast Variable?

spark: SparkSession = (
    SparkSession.builder.master("local-cluster[2,2,1024]")
    .appName("BroadcastTest")
    .config("spark.python.worker.reuse", "true")
    .getOrCreate()
)
sc = spark.sparkContext
broadcast_obj = Tracker("Broadcast", payload_size_mb=100)
bc = sc.broadcast(broadcast_obj)

...

def task(partition_idx, iterator):
    before = rss_mb()
    obj = bc.value
    obj.counter += 1
    after = rss_mb()
    
    list(iterator)
    
    return [
        {
          ...
        }
    ]

result = rdd.mapPartitionsWithIndex(task).collect()

The results showed a stark difference. Looking at the same PID (Worker), memory (RSS) only increased once when the broadcast variable was initially loaded. For all subsequent partitions processed by that worker, the memory delta was 0. Furthermore, the Object ID remained identical, and the Counter value accumulated, proving that the object was perfectly cached in memory and reused.

# Broadcast Result

================================================================================
RESULT
================================================================================
{'partition': 0, 'pid': 9108, 'object_id': 4566676736, 'counter': 1, 'rss_before': 30.640625, 'rss_after': 133.984375, 'rss_delta': 103.34375}
{'partition': 1, 'pid': 9109, 'object_id': 4565405856, 'counter': 1, 'rss_before': 30.8125, 'rss_after': 134.171875, 'rss_delta': 103.359375}
{'partition': 2, 'pid': 9107, 'object_id': 4565564832, 'counter': 1, 'rss_before': 30.90625, 'rss_after': 134.359375, 'rss_delta': 103.453125}
{'partition': 3, 'pid': 9106, 'object_id': 4554549072, 'counter': 1, 'rss_before': 30.8125, 'rss_after': 134.28125, 'rss_delta': 103.46875}
{'partition': 4, 'pid': 9107, 'object_id': 4565564832, 'counter': 2, 'rss_before': 172.515625, 'rss_after': 172.515625, 'rss_delta': 0.0}
{'partition': 5, 'pid': 9106, 'object_id': 4554549072, 'counter': 2, 'rss_before': 172.59375, 'rss_after': 172.59375, 'rss_delta': 0.0}
{'partition': 6, 'pid': 9107, 'object_id': 4565564832, 'counter': 3, 'rss_before': 173.09375, 'rss_after': 173.09375, 'rss_delta': 0.0}
{'partition': 7, 'pid': 9106, 'object_id': 4554549072, 'counter': 3, 'rss_before': 173.0625, 'rss_after': 173.0625, 'rss_delta': 0.0}
{'partition': 8, 'pid': 9106, 'object_id': 4554549072, 'counter': 4, 'rss_before': 173.171875, 'rss_after': 173.171875, 'rss_delta': 0.0}
{'partition': 9, 'pid': 9107, 'object_id': 4565564832, 'counter': 4, 'rss_before': 173.109375, 'rss_after': 173.109375, 'rss_delta': 0.0}
{'partition': 10, 'pid': 9107, 'object_id': 4565564832, 'counter': 5, 'rss_before': 173.234375, 'rss_after': 173.234375, 'rss_delta': 0.0}
{'partition': 11, 'pid': 9106, 'object_id': 4554549072, 'counter': 5, 'rss_before': 173.1875, 'rss_after': 173.1875, 'rss_delta': 0.0}
{'partition': 12, 'pid': 9109, 'object_id': 4565405856, 'counter': 2, 'rss_before': 172.546875, 'rss_after': 172.546875, 'rss_delta': 0.0}
{'partition': 13, 'pid': 9108, 'object_id': 4566676736, 'counter': 2, 'rss_before': 172.40625, 'rss_after': 172.40625, 'rss_delta': 0.0}
{'partition': 14, 'pid': 9107, 'object_id': 4565564832, 'counter': 6, 'rss_before': 173.234375, 'rss_after': 173.234375, 'rss_delta': 0.0}
{'partition': 15, 'pid': 9106, 'object_id': 4554549072, 'counter': 6, 'rss_before': 173.1875, 'rss_after': 173.1875, 'rss_delta': 0.0}

This happens because, internally in PySpark, when .value is called for the first time, it reads the broadcast file from the local disk, deserializes it, and saves it in the worker’s global context (_broadcastRegistry). The inefficiency of recreating the object for every partition (like the closure method) is entirely eliminated.

However, I could confirm my initial suspicion: the data is still duplicated in memory as many times as there are Python workers (i.e., the number of cores). (Note: If you disable Python worker reuse with "spark.python.worker.reuse": "false", it behaves identically to the Closure variable with the counter remaining at 1).

Hypothesis Test 3: The Ultimate Answer for Large-Scale Data — Broadcast Hash Join (BHJ)

Finally, I tested the scenario where, instead of passing the dictionary as a Python object, I converted it into a DataFrame and executed a Broadcast Hash Join through the Spark SQL engine.

In this case, the data never crosses over into the Python worker’s memory space. Spark’s physical execution engine (Tungsten) utilizes its own internal memory format (UnsafeRow) to process the join entirely within the JVM.

Executor Heap Histogram Analysis Results:

# Executor Heap Histogram
# Rank | Instances | Bytes | Class Name

1882 | 1 | 24 | org.apache.spark.sql.execution.joins.LongHashedRelation

As the results show, exactly one instance of the LongHashedRelation object is created in the JVM heap space of each Executor, and all Cores (Tasks) share it.

This is the most ideal state: there is absolutely zero serialization/deserialization communication cost between Python and the JVM, and the data is not duplicated per core.

Conclusion & Summary

Based on the logging and code analysis above, we can outline the following architectural strategies for passing reference metadata in a PySpark environment:

If you need to transmit Column/Row-shaped data or large datasets, you should strongly consider processing it with a Broadcast Hash Join depending on the context, rather than simply passing it as a Broadcast Variable.

Closure Variable

  • Network Transmission: Broadcasted once to the JVM.
  • Python Memory Load: Freshly unpickled for every Task (partition).
  • Drawbacks: The object is created and destroyed repeatedly as many times as the number of partitions processed (causes extreme GC).

Broadcast Variable

  • Network Transmission: Broadcasted once to the JVM.
  • Python Memory Load: Loaded exactly once per Worker Process (PID).
  • Drawbacks: Data is replicated in memory according to the number of Cores (Workers).

Broadcast Hash Join (BHJ)

  • Network Transmission: Broadcasted once to the JVM.
  • Python Memory Load: Data is never sent to the Python Worker.
  • Drawbacks: Network I/O occurs when managing it as a table; puts a burden on the Driver if dynamically generated.

References

  • Task Broadcast Logic in DAGScheduler: The optimization logic inside submitMissingTasks that broadcasts the serialized Task Binary object just once to prevent Driver network bottlenecks.
  • Process Forking via pyspark.daemon: Checking the behavior of the background daemon, which takes requests from the JVM Executor and spawns the actual Python worker processes via Linux’s os.fork().
  • Python Worker Deserialization (Unpickle): How the Python worker, upon receiving a Task from the JVM, unpickles the byte array back into a Python object (using cloudpickle, etc.) inside the read_command function to load it into memory.
  • LongHashedRelation in Broadcast Hash Join: The hash table structure that the Tungsten engine instantiates just once within the JVM to process joins and reuse across tasks.

Full Code

  • tracker.py
import os
import pickle
import socket

class Tracker:
    serialize_count = 0

    def __init__(self, mode_name, payload_size_mb=100):

        self.mode_name = mode_name

        self.counter = 0

        self.payload = b"x" * payload_size_mb * 1024 * 1024

        print(
            f"""
            CREATE
            mode={mode_name}
            pid={os.getpid()}
            host={socket.gethostname()}
            obj_id={id(self)}
            """
        )

    def __getstate__(self):

        Tracker.serialize_count += 1

        size_mb = len(pickle.dumps(self.__dict__)) / 1024 / 1024

        print(
            f"""
            SERIALIZE
            mode={self.mode_name}
            serialize_idx={Tracker.serialize_count}
            pid={os.getpid()}
            host={socket.gethostname()}
            obj_id={id(self)}
            size={size_mb:.2f}MB
            """
        )

        return self.__dict__
  • test_closure.py
import os
import socket

import psutil
from tracker import Tracker

from pyspark.sql import SparkSession

def rss_mb():
    return psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024

spark = (
    SparkSession.builder.master("local-cluster[2,2,1024]")
    .appName("ClosureTest")
    .config("spark.python.worker.reuse", "true")
    .getOrCreate()
)

sc = spark.sparkContext
sc.setLogLevel("DEBUG")

sc.addPyFile("tracker.py")

print(
    f"""
    DRIVER
    pid={os.getpid()}
    host={socket.gethostname()}
    """
)

closure_obj = Tracker("Closure", payload_size_mb=1)

rdd = sc.parallelize(range(16), 16)

def task(partition_idx, iterator):

    before = rss_mb()

    obj = closure_obj

    obj.counter += 1

    after = rss_mb()

    list(iterator)

    return [
        {
            "partition": partition_idx,
            "pid": os.getpid(),
            "host": socket.gethostname(),
            "object_id": id(obj),
            "counter": obj.counter,
            "rss_before": before,
            "rss_after": after,
            "rss_delta": after - before,
        }
    ]

result = rdd.mapPartitionsWithIndex(task).collect()

print("\n")
print("=" * 80)
print("RESULT")
print("=" * 80)

for row in sorted(result, key=lambda x: x["partition"]):
    print(row)

spark.stop()
  • test_broadcast.py
import os
import socket

import psutil
from tracker import Tracker

from pyspark.sql import SparkSession

def rss_mb():
    return psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024

spark = (
    SparkSession.builder.master("local-cluster[2,2,1024]")
    .appName("BroadcastTest")
    .config("spark.python.worker.reuse", "true")
    .getOrCreate()
)

sc = spark.sparkContext
sc.setLogLevel("DEBUG")

sc.addPyFile("tracker.py")

print(
    f"""
    DRIVER
    pid={os.getpid()}
    host={socket.gethostname()}
    """
)

broadcast_obj = Tracker("Broadcast", payload_size_mb=1)
bc = sc.broadcast(broadcast_obj)

rdd = sc.parallelize(range(16), 16)

def task(partition_idx, iterator):

    before = rss_mb()

    obj = bc.value

    obj.counter += 1

    after = rss_mb()

    list(iterator)

    return [
        {
            "partition": partition_idx,
            "pid": os.getpid(),
            "host": socket.gethostname(),
            "object_id": id(obj),
            "counter": obj.counter,
            "rss_before": before,
            "rss_after": after,
            "rss_delta": after - before,
        }
    ]

result = rdd.mapPartitionsWithIndex(task).collect()

print("\n")
print("=" * 80)
print("RESULT")
print("=" * 80)

for row in sorted(result, key=lambda x: x["partition"]):
    print(row)

spark.stop()

Previous posts

View all posts →

Comments