Apache Spark Deep Dive: repartition & coalesce
6 min read

When working with Spark, you may end up with too many or too few partitions. In such cases, you can adjust the number of partitions with repartition and coalesce.
repartition and coalesce are both functions that set the number of partitions, but the difference is that coalesce can only reduce the number of partitions.
To make things easier to follow, let’s check with the AQE setting and Broadcast set to false.
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.types import *
spark = SparkSession.builder \
.master("local[*]") \
.appName("PartitionKeyExample") \
.config("spark.sql.adaptive.enabled", "false")\
.getOrCreate()
When AQE is on, Spark wraps the Physical Plan in a wrapper node called AdaptiveSparkPlanExec. So calling repartition plans a Shuffle, but AQE stands by, ready to dynamically reduce the number of partitions (coalesce) or change the join strategy based on runtime statistics, which can make it hard to verify things precisely.
Repartition
Using
numPartitionsorcols, you can split one partition into N partitions.
# def repartition( # type: ignore[misc]
# self, numPartitions: Union[int, "ColumnOrName"], *cols: "ColumnOrName"
# ) -> "DataFrame":
users_df = users_df.repartition(F.col("user_id"))
If you only specify the column name as above, the data is repartitioned into 200 partitions following the default setting (number of partitions).
# Print partition Num
users_df.rdd.getNumPartitions()
# 200
# Print partition type
print("Plain after partition:", users_df
._jdf.queryExecution()
.executedPlan()
.outputPartitioning()
)
# Plain after partition: hashpartitioning(user_id#41, 200)
With repartition, you can either increase or decrease the number of partitions.
Group by After Repartition
Let’s group by (agg) the data partitioned by user_id.
To make things easier to follow, we’ll repartition into 100 partitions.
users_df = users_df.repartition(100, F.col("user_id"))
users_df.rdd.getNumPartitions()
# 100
result = users_df.groupBy("user_id").count()
result.show()

number of partitions: 100
You can see that repartition(100, "user_id") from earlier works as expected.
Join After Repartition
Now let’s repartition the user information by
user_idand then Join it with the order information onuser_id.
orders_df = spark.read.parquet("./data/work/orders")
orders_df.rdd.getNumPartitions()
# 1
What result do we get when we Join user data that is already partitioned by user_id with order data?
result = orders_df.join(users_df, on="user_id", how="inner")
result.show()

Exchange
You can see that both tables were partitioned into 200 partitions and then SortMergeJoined.
To unify on the default of 200, Spark discarded the repartition(100, "user_id") defined earlier.
In a Spark Join, two partitionings are treated as the same strategy only when both the Partition key (user_id) and the number of Partitions (100) are identical.
orders_df = orders_df.repartition(100, F.col("user_id"))
print("Plain:", orders_df
._jdf
.queryExecution()
.executedPlan()
.outputPartitioning())
# Plain: hashpartitioning(user_id#243, 100)
print("Plain:", users_df
._jdf.queryExecution()
.executedPlan()
.outputPartitioning()
)
# Plain: hashpartitioning(user_id#41, 100)
In the case above, the keys don’t match because of the #41, #243 after user_id, so it is shuffled into 200 partitions as well.
Coalesce
Coalesce can only reduce the number of Partitions. You can only set the target count, and you cannot specify a particular column.
# def coalesce(self, numPartitions: int) -> "DataFrame":
df.coalesce(2)
Let’s go back to the example of grouping user information by user_id.
users_df = users_df.coalesce(2)
result = users_df.groupBy("user_id").count()
result.show()

Query
Looking at the executed Query, you can see that coalesce has been added, unlike the earlier Repartition. One slight difference, though, is that it is still partitioned by user_id.
Repartition vs Coalesce (Map Side vs Reduce Side)
Repartition and Coalesce both change the number of partitions, but they operate at different places.
A Spark Shuffle works through two Stages, the Map Side and the Reduce Side.
On the Map Side, data is shuffled so that a given key exists in only one partition. On the Reduce Side, one Task (slot) is assigned per partition, and each Task fetches only the partition it is responsible for from all upstreams and processes it.

Map Side: Repartition (Shuffle Write)
Repartition operates on the Map Side and decides into how many partitions, or by which key, the data coming from upstream is split.
Because of this, you can increase or decrease the number of partitions regardless of the previous partition count, and you can choose which key to partition by.
Reduce Side: Coalesce (Shuffle Read)
Coalesce, on the other hand, operates on the Reduce Side. Normally the partitions created on the Map Side would require running Tasks 1:1 as Partition = Task, but with coalesce you can decide how far to reduce that number.
If you coalesce(10) a Partition set that was split into 200, it is still split into 200, but the data is bundled in groups (e.g., task1: [0 ~ 19], task2: [20 ~ 39]) so that fewer Tasks are created to process it.
Because it works this way, you cannot set the number of partitions to a value greater than the upstream partition count (200).
AQE (Adaptive Query Execution): Shuffle Read (Coalesce)
With AQE, the coalesce at the Shuffle Read stage is set dynamically based on the data size.

At the Shuffle Read stage, 200 Tasks would normally have to run, but you can see that AQE looked at the data size and changed the query plan so that all partitions are processed in a single Task.
AQE: Configuration for Dynamically Coalescing Shuffle Partitions
spark.sql.adaptive.coalescePartitions.enabled (default: True): Dynamically reduces the number of partitions after a shuffle, trying to match the partition size to the variable below (advisoryPartitionSizeInBytes)spark.sql.adaptive.advisoryPartitionSizeInBytes (default: 64MB): The target partition size when reducing the number of partitions after shufflingspark.sql.adaptive.coalescePartitions.parallelismFirst (default: True): To guarantee parallelism, the target size above is ignored and only the minPartitionSize is guaranteedspark.sql.adaptive.coalescePartitions.initialPartitionNum (default: None): The number of partitions before coalescing. If unset, spark.sql.shuffle.partitions is used
Things to Consider When Using Coalesce
When you use Coalesce, the earlier partitioning record (key, numOfPartition) is lost, so even if you process by the same key and numOfPartition, partitioning may happen once more.
print("Plain after partition:", users_df
._jdf.queryExecution()
.executedPlan()
.outputPartitioning()
)
# Plain after partition: hashpartitioning(user_id#423, 100)
It is guaranteed to be hash-partitioned by user_id#423, 100. If you JOIN with another table partitioned by user_id#423, 100, no shuffling occurs
users_df = users_df.coalesce(2)
print("Plain after partition:", users_df
._jdf.queryExecution()
.executedPlan()
.outputPartitioning()
)
# Plain after partition: UnknownPartitioning(2)
It becomes UnknownPartitioning, so if you JOIN with a table partitioned by user_id#423, 100, it has to be partitioned again
If you merge all partitions into one with coalesce(1), it becomes SinglePartition rather than UnknownPartitioning.
When joining, no Shuffle occurs if both tables are SinglePartition, but if the other table’s partitioning conditions (key, num) differ, shuffling still occurs under a unified partitioning condition.

When only one side is coalesced, both are shuffled by user ID

When both sides are made SinglePartition(coalesce(1)), no shuffle occurs
Using Coalesce When Writing Files
Coalesce is beneficial when you finally
writea file.
By default, Spark creates partitions following the partitioning created earlier.
When Spark reads user data that exists as 2 small files (10MB each), it reads them as 2 Partitions following the existing partitioning, rather than merging them into 1 Partition.
This causes the small file problem and can degrade performance. To prevent it, it is best to use coalesce when finally writing files, so that files are written in appropriately sized Partitions.