Efficient way of reading large dataset (10TB) from S3 using EMR Spark, performing filtering operations on partitioned columns, and writing back to S3 - scala

Question

I'd like to read a large (~10TB per day) log dataset from S3 using EMR Spark, do some filtering based on partitioned and non-partitioned columns, and write the results back to S3. After filtering, there should be less than 1TB data left in each day.
The dataset is partitioned in this format day/hour/col1/col2. There is relatively similar size of data in each hour for every day. That's why I chunk up the data and read day by day and partition the query by hour. Here is a sample query:
for(date <- 1 to 30){
var output_path = "s3a://dest-bucket/logs/day=%d/".format(date)
spark.table("s3a://source-bucket/logs/day=%d/")
.filter('col1.isin("A", "B", "C")) // col1 is partitioned in source S3
//.filter('col3.isin("X", "Y", "Z")) --> Point 4
.withColumn("hid", hash($"id") % 2000) --> Point 5
.select("col1","col2","col3","col4")
// .repartition(col("hid")) --> Point 5
.repartition(col("hour")) --> Point 1
.write
// .partitionBy("hid")
.partitionBy("hour") --> Point 2
.option("header","true").mode("overwrite").parquet(output_path)
}
The objective is to perform the query as fast as possible and with least operation issues (OOM, lost/failed executors, etc). Currently, my query takes 10-15 hours per log day. And sometimes it fails due to too many lost executors. However, copying the entire files (no filtering) using aws cli takes the same amount on a single node. So, I'm hoping to expedite the query using a large cluster.
In order to reach this objective, I have several questions that can shed light on the problem:
For Spark (v2.4.8) to understand that the source is partitioned on a column, do I need a repartition(col("hour")) statement? What are the pros and cons of doing repartition on a partitioned data source? Even if Spark does Partition Discovery and figures out that the data is partitioned on multiple columns, is there any benefit to doing a repartition based on knowing that the data is fairly uniformly distributed over one column (hour).
In order to expedite the write process, I have a partitionBy("hour") statement, to make write go 24x faster. Otherwise, Spark writes one output file at a time (essentially the move part out of _temporary folder is done sequentially). I don't understand the reason. How could I make the executors perform write operation in parallel?
If I have a partitionBy in write, would I benefit from having a repartition statement?
I can forgo of the condition on the non-partitioned column col3, if that could significantly increase the query performance.
My ultimate goal is to hash-partition the users based on their IDs and rewrite the data with that partition (a new column called hid). However, the reshuffling makes the cluster fail. I have 200 nodes in the cluster each with 128 GB of memory (m5.8xlarge). There should be enough memory to hold the data. Is there any way to optimize the reshuffling based on a new unpartitioned column?
Please see below for the spark configuration I use.
spark-shell \
--master yarn \
--conf "spark.executor.instances=1199" \
--conf "spark.default.parallelism=5995" \
--conf "spark.sql.shuffle.partitions=5995" \
--conf "spark.driver.cores=5" \
--conf "spark.driver.maxResultSize=20g" \
--conf "spark.driver.memory=18g" \
--conf "spark.driver.memoryOverhead=3g" \
--conf "spark.executor.cores=5" \
--conf "spark.executor.memory=18g" \
--conf "spark.executor.memoryOverhead=3g" \
--conf "spark.executor.heartbeatInterval=3600" \
--conf "spark.hadoop.orc.overwrite.output.file=true" \
--conf "spark.hadoop.parquet.enable.summary-metadata=false" \
--conf "spark.sql.execution.arrow.pyspark.enabled=true" \
--conf "spark.sql.execution.arrow.pyspark.fallback.enabled=true" \
--conf "spark.sql.parquet.mergeSchema=false" \
--conf "spark.sql.parquet.int96RebaseModeInRead=CORRECTED" \
--conf "spark.sql.debug.maxToStringFields=100" \
--conf "spark.hadoop.fs.s3.connection.maximum=1000" \
--conf "spark.hadoop.fs.s3.connection.timeout=300000" \
--conf "spark.hadoop.fs.s3.threads.core=250" \
--conf "spark.hadoop.fs.s3a.connection.maximum=1000" \
--conf "spark.hadoop.fs.s3a.connection.timeout=300000" \
--conf "spark.hadoop.fs.s3a.threads.core=250"\
--conf "spark.reducer.maxBlocksInFlightPerAddress=20"
Any advice is appreciated.