Handling dataskew without salting the join key in spark - scala

Question

I am trying to inner Join a million rows dataframe with a 30 rows dataframe and both the tables have same join key, spark is trying to perform sort merge join and due to which all my data ends up in the same executor and Job never finishes, for example
DF1(million rows dataframe registered as TempView DF1)
+-------+-----------+
| id | price |
+-------+-----------+
| 1 | 30 |
| 1 | 10 |
| 1 | 12 |
| 1 | 15 |
+-------+-----------+
DF2(30 rows dataframe registered as TempView DF2)
+-------+-----------+
| id | Month |
+-------+-----------+
| 1 | Jan |
| 1 | Feb |
+-------+-----------+
I tried following
Broadcasting
spark.sql("Select /*+ BROADCAST(Df2) */ Df1.* from Df1 inner join Df2 on Df1.id=Df2.id").createTempView("temp")
Repartitioned
Df1.repartition(200)
Query Execution Plan
00 Project [.......................]
01 +- SortMergeJoin [.............................],Inner
02 :- Project [.............................]
03 : +-Filter is notnull[JoinKey]
04 : +- FileScan orc[..........................]
05 +-Project [.............................]
06 +-BroadcastHashJoin [..........................], LeftOuter, BuildRight
07 :- BroadCastHashJoin [......................],LeftSemi, BuildRight
Output of the number of partitions
spark.table("temp").withColumn("partition_id",spark_partition_id).groupBy
("partition_id").count
+-------+---------------+
| 21 |300,00,000 |
+-------+---------------+
Even though i re-partition/broadcast the data, spark is bringing all the data to one executor while joining and data gets skewed at one executor. I also tried turning off the spark.sql.join.preferSortMergeJoin to false. But i still see my data getting skewed at one executor. Can anyone help me ?

Answer 1

Just doing it like this, it works fine. Data is as is, no partitioning as such.
import org.apache.spark.sql.functions.broadcast
// Simulate some data
val df1 = spark.range(1000000).rdd.map(x => (1, "xxx")).toDF("one", "val")
val df2 = spark.range(30).rdd.map(x => (1, "yyy")).toDF("one", "val2")
// Data is as is, has no partitioning applied
val df3 = df1.join(broadcast(df2), "one")
df3.count // An action to kick it all along
// Look at final counts of partitions
val rddcounts = df3.rdd.mapPartitions(iter => Array(iter.size).iterator, true)
rddcounts.collect
returns:
res26: Array[Int] = Array(3750000, 3750000, 3750000, 3750000, 3750000, 3750000, 3750000, 3750000)
This relies on default parallelism, 8 on a CE Databricks cluster.
Broadcast should work in any event as the small table is SMALL.
Even with this:
val df = spark.range(1000000).rdd.map(x => (1, "xxx")).toDF("one", "val")
val df1 = df.repartition(50)
It works in parallel with 50 partitions. This is round-robin partitioning meaning the cluster will get partitions distributed over N Workers with at least N Executors. It is not hashed, the hash is invoked by specifying a column causing skewness if all values same. I.e. the same partition on 1 Worker for all the data.
QED: So, not all working on only one Executor, unless you have only one Executor for the Spark App or hashing applied.
I ran afterwards on my experimental laptop with local[4] and the data was being serviced by 4 cores, thus 4 Executors as it were. No salting, parallel 4. So, it is odd you cannot get that, unless you hashed.
You can see 4 parallel Tasks and thus not all on 1 Executor if on a real cluster.

Answer 2

why dose all data move to one executor ? If you only have same
id(id:1) in DF1 and use id to join DF2 . according the
HashPartitioner the data with id=1 will always move together .
have you occur Broadcast join ? check it in spark UI