How to read Hive Table with Spark-Sql efficiently - scala

Question

I have a table with 20GB data in hive, I am reading the table using spark with hive context and I am able to see the data and schema as expected.
However it is taking around 40 mins to read the data, is there any alternative to read the data from hive table efficiently.
Hive Table Sample_Table - 20 GB, No partitions, using ORC Snappy Compression. (data exploded to 120 GB while reading from spark)
spark = SparkSession.builder().enableHiveSupport()getOrCreate()
val spark_table = spark.sql(select * from Sample_Table)
Environment Details -
Not using any cloud
Nodes - Around 850, total memory - 160 TB, 80 V cores per node, upto 300GB memory per node, 22 Disks per node
Spark-Submit command -
/mapr/spark/bin/spark-submit \
--verbose
--num-executors=30 \
--conf spark.locality.wait=60s \
--conf spark.network.timeout=14080s \
--driver-memory=20G \
--executor-memory=15G \
--conf spark.blacklist.enabled=true \
--conf spark.shuffle.service.enabled=true \
--master yarn \
--name=Sample_xxx \
--conf spark.default.parallelism=25 \
--conf spark.task.cpus=3 \
--conf spark.broadcast.compress=true \
--conf spark.io.compression.codec=lz4 \
--conf spark.shuffle.compress=true \
--conf "spark.executor.cores=3" \
--conf spark.shuffle.spill.compress=true \
--conf spark.rdd.compress=true \
--conf spark.sql.shuffle.partitions=1000 \
--conf spark.yarn.executor.memoryOverhead=3G \
--conf spark.sql.tungsten.enabled=true \
--queue sample1XX \
--class XXX yy.jar
I am reading multiple tables and performing multiple transformations, that's the reason I have below configurations in spark submit command

Answer 1

20GB should not take very long, though if it decompresses to 120GB that is a more substantial workload; please describe what kind of hardware you are using.
I guess from the path to spark-submit that you are using the MapR distribution of Hadoop. Does this include a user interface giving additional insight into performance? Look particularly for memory usage / garbage collection.
Are there any other jobs running on your cluster that might be taking resources?
Is the 40 minutes merely to load the data, or does this include your processing (bear in mind that Spark loads data lazily, so timings can sometimes be misleading)?
The main thing I notice is that spark.yarn.executor.memory is not a valid spark setting, so your executors may be short of memory.
Use the --executor-memory setting instead.
Also add the --verbose flag and study the output when you run your job, to ensure that the settings are correct and are being parsed in the way you expect.
Also check the executor logs to ensure that there are no errors, particularly due to running out of memory.
Minor observations:
You are setting spark.shuffle.spill.compress twice (harmless but unnecessary)
You are setting both --num-executors and spark.dynamicAllocation.enabled=true, which are contradictory, and normally result in a warning message saying that dynamic allocation will be disabled.