Reducing shuffle disk usage in Spark aggregations - scala

Question

I have a table in hive which is 100 GB of size. I am trying to group, count function and storing the result as hive table..my hard disk has 600 GB of space, by the time job reaches to 70% all of the disk space is being occupied.
so my job got fails...How can I minimize the shuffle data writes
hiveCtx.sql("select * from gm.final_orc")
.repartition(300)
.groupBy('col1, 'col2).count
.orderBy('count desc)
.write.saveAsTable("gm.result")
spark_memory

Answer 1

In cloud-based execution environments adding more disk is usually a very easy and cheap option. If your environment does not allow this and you've verified that your shuffles settings are reasonable, e.g., compression (on by default) is not changed, then there is only one solution: implement your own staged map-reduce using the fact that counts can be re-aggregated via sum.
Partition your data in any way that seems fit (by date, by directory, by number of files, etc.)
Perform the counting by col1 and col2 as separate Spark actions.
Re-group and re-aggregate.
Sort.
For simplicity, let's assume that col1 is an integer. Here is how I'd break up processing into 8 separate jobs, re-aggregating their output. If col1 is not an integer, you can hash it or you can use another column.
def splitTableName(i: Int) = s"tmp.gm.result.part-$i"
// Source data
val df = hiveCtx.sql("select col1, col2 from gm.final_orc")
// Number of splits
val splits = 8
// Materialize partial aggregations
val tables = for {
i <- 0 until splits
tableName = splitTableName(i)
// If col1 % splits will create very skewed data, hash it first, e.g.,
// hash(col1) % splits. hash() uses Murmur3.
_ = df.filter('col1 % splits === i)
// repartition only if you need to, e.g., massive partitions are causing OOM
// better to increase the number of splits and/or hash to un-skew skewed data
.groupBy('col1, 'col2).count
.write.saveAsTable(tableName)
} yield hiveCtx.table(tableName)
// Final aggregation
tables.reduce(_ union _)
.groupBy('col1, 'col2)
.agg(sum('count).as("count"))
.orderBy('count.desc)
.write.saveAsTable("gm.result")
// Cleanup temporary tables
(0 until splits).foreach { i =>
hiveCtx.sql(s"drop table ${splitTableName(i)}")
}
If col1 and col2 are so diverse and/or so large that the partial aggregation storage is causing disk space issues then you have to consider one of the following:
Smaller number of splits will generally use less disk space.
Sorting on col1 will help (because of Parquet run length encoding) but that would slow down execution.
How to create splits that are independent, e.g., find distinct values of col1, partition those into groups.
If you are extremely short on disk space you'd have to implement multi-step re-aggregation. The simplest approach is to generate the splits one at a time and keep a running aggregate. The execution would be much slower but it will use a lot less disk space.
Hope this helps!