seon99il

Data Engineering

Apache Spark Deep Dive: Shuffle

7 min read

Let’s look at the stages a Shuffle goes through in Spark and how partitions change at each stage.

Spark Executor & Slot(Core)

Spark consists of a Driver plus N Executors and M Slots (Cores).

  • --num-executors : Number of Executors
  • --executor-cores : Number of cores per Executor
  • --executor-memory : Memory allocated to each Executor

If you set 3 Executors with 3 Cores each, the number of partitions that can be processed in parallel at once is 9 (N × M = 9).

Each Core processes one Partition

Executor Memory

The memory (JVM) allocated through --executor-memory, i.e. the Heap memory region, is shared by all Cores.

When you create one Executor (memory: 1GB), the server (Worker) allocates memory as follows.

  • Executor Memory (Heap): 1024MB (the size you set)
  • Memory Overhead (Off-heap): 384MB (max( (heap/10), 384 ))

Overhead (Off-heap)

This off-heap memory is used for the following essential tasks.

  • NIO Buffers: Buffers used to send and receive data over the network during a Shuffle
  • Thread Stacks: Stack memory used by each thread
  • Direct Memory: Used when OS memory is used directly, bypassing the JVM heap
  • PySpark: (When using Python) part of the memory used by the Python process may be included in this region.

How Spark Shuffle Works

Back to the main topic: a Spark Shuffle is actually completed in 2 phases.

  • Map Phase (Shuffle Write)
  • Reduce Phase (Shuffle Read)
user_schema = StructType([
    StructField("user_id", IntegerType(), False),
    StructField("name", StringType(), True),
    StructField("city", StringType(), True),
    StructField("age", IntegerType(), True)
])

user_data = [
    (1, "Alice", "Seoul", 28),
    (2, "Bob", "Busan", 35),
    (3, "Charlie", "Seoul", 42),
    (4, "David", "Incheon", 22),
    (5, "Eve", "Busan", 31)
]

users_df = spark.createDataFrame(user_data, user_schema)

.config("spark.sql.adaptive.enabled", "false") To make things easier to follow, I proceeded without AQE.

Stage 1: Map Side (Shuffle Write)

This stage occurs on the sending Core to prepare for the Shuffle, and it Hash Partitions according to the configured number of partitions (default: 200).

This means that if you group by("userId") on user data with IDs 1, 2, 3, 4, 5, each user ID is placed into a specific partition based on its hash value (hash(userId) % 200).

Partitioning here is deterministic: when you partition with the same number of partitions and the same key, a given key always lands in the same partition number.

If you partition the user with userId 1 by userId across 200 partitions, it always lands in the same partition ID (43).

result.select(
    F.spark_partition_id().alias("partition_id")
    , F.col("user_id")
    , F.col("count")).show()

In other words, whichever table you use (an orders table, a user address table), if you Hash Partition under the same condition of user ID + 200 partitions, you will not find data with user_id = 1 in any partition other than partition 43. When doing a Join or an aggregation, you only need to look at partition 43 for user_id=1.

File Structure of Shuffle Write

When a Shuffle Write is done on 3 upstream partitions, files are created on disk in chunks through the PartitionedPairBuffer (shuffle buffer), as follows.

spark.local.dir/blockmgr-{uuid}/
  shuffle_0_0.data    <- All partitions from map task 0, concatenated
  shuffle_0_0.index   <- Byte offsets for quick lookup
  shuffle_0_1.data    <- All partitions from map task 1
  shuffle_0_1.index
  shuffle_0_2.data    <- All partitions from map task 2
  shuffle_0_2.index

Each Task writes 2 files (an index file and a data file), and the .index file stores the offsets defined above (numOfpartition: 200) so the Reduce side can quickly find the partition it wants.

The MapOutput written to disk (HashPartition(user_id, 200)) can be tracked through the MapOutputTrackerMaster in the Spark Driver, which records where the Map Side (repartition) result is stored. Thanks to this, when processing Reduce Side (count, write) work based on that partition, the redundant Repartition (Map Stage) only needs to be done once. (Skipped Stage)

Stage 2: Reduce Side (Shuffle Read)

Shuffle Read occurs on the Core (Slot) that gathers a specific partition in one place and processes it, reading all the data of the partition it is responsible for.

As mentioned earlier, one partition is handled by one Core. If it is assigned partition 43, it goes around all the distributed servers, gathers the data of partition 43 from each, and runs the aggregation (count, sum) to get the result for user ID 1.

This is called Shuffle Read.

Q: How many partitions are created?

A: By default, 200 are created. (5 + 195 empty partitions)

There are only 5 user IDs, but the default 200 partitions are created, which degrades performance because 195 Tasks run on empty partitions.

result.rdd.getNumPartitions()

# 200

Reducing Tasks on Small Partitions with AQE (Adaptive Query Execution)

If AQE is enabled here (it is by default), AQE’s intervention can prevent the performance degradation.

Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan, which is enabled by default since Apache Spark 3.2.0. Spark SQL can turn on and off AQE by spark.sql.adaptive.enabled as an umbrella configuration.

Performance Tuning — Spark offers many techniques for tuning the performance of DataFrame or SQL workloads. Those techniques, broadly…

As in the example above, small partitions are reduced under the name AQEShuffleRead coalesced so that they can be read in a single Task.

Web UI — AQE, result.select(partition_id, …)

It Coalesces the 200 split partitions into one, tuning things so that one Task runs instead of 200 Tasks across all those partitions.

In the Shuffle Write stage, 200 Partitions are still created as before. However, AQE can be expected to improve performance at Shuffle Read.

Shuffle Write: Hash Aggregate

Hash Aggregate is a way to reduce network cost by putting pre-grouped results into the partitions during Shuffle Write, instead of partitioning the original data in full.

Count by user ID

Instead of putting user_id | name | city | age into the partitions and sending them, each core computes in advance on the partition it is responsible for and prepares the computed results in the partitions.

[{
  user_id: 1,
  count:  1,
},
{
  user_id: 2,
  count:  1,
},
{
  user_id: 3,
  count:  1,
}
...
]

Then, when the data moves (Shuffle Read), network IO can be reduced.

About the Number Of Output Rows in Hash Aggregate

Let’s group by user ID on the order data.

order_data = [
    (101, 1, "Laptop", 1200.00, datetime(2025, 11, 15, 10, 0, 0)),
    (102, 2, "Mouse", 25.50, datetime(2025, 11, 15, 12, 30, 0)),
    (103, 1, "Monitor", 350.99, datetime(2025, 11, 16, 9, 0, 0)),
    (104, 3, "Keyboard", 75.00, datetime(2025, 11, 16, 15, 0, 0)),
    (105, 5, "Webcam", 50.00, datetime(2025, 11, 17, 11, 0, 0)),
    (106, 1, "Headset", 99.99, datetime(2025, 11, 17, 14, 0, 0)),
    (107, 6, "Charger", 15.00, datetime(2025, 11, 18, 10, 0, 0))
]

orders_df = spark.createDataFrame(order_data, order_schema)
orders_df.rdd.getNumPartitions()

# Partition_cnt: 12
orders_df.groupBy("user_id").count().show()

Q: In this case, how many Rows are sent after the Hash Aggregate? A: The answer is 5 to 7.

result of orders_df.groupBy(“user_id”).count().show()

The reason lies in the number of partitions. If all the data were in one partition, it could be grouped into 5 rows, 1, 2, 3, 5, 6, and sent.

But what if the values are in different partitions? If the data is distributed as Partition1(1, 2, 1), Partition2 (3, 5, 1, 6), then 2 rows move from partition 1 and 4 rows from partition 2, for a total of 6 rows.

(orders_df.groupBy(F.spark_partition_id())
          .agg(F.collect_list("user_id"))
          .show(truncate=False)
)

The data is split across 7 partitions, so the Hash Aggregate result is 7 rows

Does merging all partitions into one with coalesce(1) cause a Shuffle?

Let’s compare how shuffling occurs when there is one partition after spark.read versus when partitions are merged into one with coalesce(1).

orders_df = orders_df.coalesce(1)
users_df = users_df.coalesce(1)

result = orders_df.join(users_df, on="user_id", how="inner")
result.show()

As a result, no Shuffle occurs.

It’s true that user data and order data on different servers move to a single server, but Spark does not call this a shuffle.

Join Between Tables with 1 Partition after spark.read

Now let’s check whether a Shuffle occurs, as above, when there is 1 partition after spark.read.

order_df = spark.read.schema(order_schema).parquet("./data/work/orders")
user_df = spark.read.schema(user_schema).parquet("./data/work/users")

order_df.rdd.getNumPartitions(), user_df.rdd.getNumPartitions()

# (1, 1)
result = order_df.join(user_df, on="user_id", how="inner")
result.show()

Result

Spark’s intervention turned this into a Broadcast Hash Join, but without that intervention (e.g. spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)), a Shuffle would have occurred. Why does the same single partition give a different result?

Inspecting the Partitioning Rule

They look the same (1 partition), but if you check internally, you can see they have different partitioning rules.

# spark.read (partition : 1)
order_df = spark.read.schema(order_schema).parquet("./data/work/orders")
print("Plain:", order_df._jdf
      .queryExecution()
      .executedPlan()
      .outputPartitioning())

# Plain: UnknownPartitioning(0)
# coalesce(1) (partition: 1)
order_df = order_df.coalesce(1)
print("Plain:", order_df._jdf
      .queryExecution()
      .executedPlan()
      .outputPartitioning())

# Plain: SinglePartition

For a DataFrame that simply ended up with 1 partition by being read, Spark cannot guarantee SinglePartition, so even DataFrames with the same single partition produce different results as above.

Previous posts

View all posts →

Comments