How to partition dataframes which are partitionBy and joined with other keys? - scala

Question

I have a lot of dataframes with customer data and the history for different legal entities, simplified:
Val myData = spark.createDataframe(Seq(
(1, 1, “a lot of Data”, 2010-01-01-10.00.00”),
(1, 1, “a lot of Data”, 2010-01-20-10.31.00”),
(1, 1, “a lot of Data”, 2019-06-16-12.00.00”),
(2, 5, “a lot of Data”, 2010-01-01-10.00.00”),
(2,6, “a lot of Data”, 2010-01-01-10.00.00”),
(3, 7, “a lot of Data”, 2010-01-01-10.00.00”)))
.toDF(“legalentity”, “customernumber”,”anydata”,”changetimestamp”)
These dataframes are stored as parquet files and exposed has external hive tables.
The change timestamp is transformed to a >valid from<, >valid to< by views, like this
CREATE VIEW myview
AS SELECT
Legalentity, customernumber, anydata,
Changetimestamp as valid_from,
Coalesce(lead(changetimestamp) over (PARTITION by legalentity, customernumber ORDER BY changetimestamp ASC), “9999-12-31-00.00.00”) as valid_to
(this is simplified, there are some timestamp transformations inside needed)
There are a lot of joins between the dataframes / hive tables later.
These dataframes are store this way:
myDf
.orderBy(col(“legalentity”), col(“customernumber”))
.write
.format(“parquet_format”)
.mode(SaveMode.Append)
.partitionBy(“legalentity”)
.save(outputpath)
For legal reason the data of different legal entities must be stored in different hdfs pathes, that is done by the partitionBy clause which creates a separate folder for each legal entity.
There are small and big legal entities with huge number of customers and others with few customers.
The number of shuffle partitions is averaged over all legal entities, that fine.
Problems:
No more Columns to partition the dataframe are possible:
If we want to speed up all by adding a repartition with more columns as the partitionBy clause for writing like:
myDf
.orderBy(col(“legalentity”), col(“customernumber”))
.repartition(col(“legalentity”), col(“customernumber”))
.write
.format(“parquet_format”)
.mode(SaveMode.Append)
.partitionBy(“legalentity”)
.save(outputpath)
The number of shuffel partitions is used in every legal entity folder.
That causes partitions = >legal entity< * >number of shuffle partitions<
Too many partitions
There are small and big dataframes / tables. All get the same number of shuffle partitions, so small dataframes has partition sizes of 3 MB or less.
If we use different numbers of partitions for each table so that the file size gets close to 128 MB, everything is slowed down.
We get new data every day which we just append but therefor we don’t use the number of shuffle partitions, we repartition(1).
Sometimes we have to re-load all to compress all these partitions, but our processes are not slowed down by the new daily data.