Too Many Parameters - scala

Question

I have an application that has a single EntryPoint, it's a library to automate some data engineers stuffs.
case class DeltaContextConfig(
primaryKey: List[String],
columnToOrder: String,
filesCountFirstBatch: Int,
destinationPath: String,
sparkDf: DataFrame,
sparkContext: SparkSession,
operationType: String,
partitionColumn: Option[String] = None,
tableName: String,
databaseName: String,
autoCompaction: Option[Boolean] = Option(true),
idealFileSize: Option[Int] = Option(128),
deduplicationColumn: Option[String] = None,
compactionIntervalTime: Option[Int] = Option(180),
updateCondition: Option[String] = None,
setExpression: Option[String] = None
)
This is my case class, my single Entrypoint.
After that all these parameters are pass to other objects, I have objects to write in Datalake, to Compact files and so on. And these objects use some of these parameters, for example, I have a DeltaWriterConfig object:
DeltaWriterConfig(
sparkDf = deltaContextConfig.sparkDf,
columnToOrder = deltaContextConfig.columnToOrder,
destinationPath = deltaContextConfig.destinationPath,
primaryKey = deltaContextConfig.primaryKey,
filesCountFirstBatch = deltaContextConfig.filesCountFirstBatch,
sparkContext = deltaContextConfig.sparkContext,
operationType = deltaContextConfig.operationType,
partitionColumn = deltaContextConfig.partitionColumn,
updateCondition = deltaContextConfig.updateCondition,
setExpression = deltaContextConfig.setExpression
)
I use the DeltaWriterConfig, to pass these parameters to my class DeltaWriter. I was creating all these configs objects on the MAIN, but I think it is not good, because, I have 3 Config Objects to populate, so I have 3 big constructors on the application main.
Is there any pattern to solve this?

Answer 1

I think at least it would be better to replace creating another config from the first one to the companion object of DeltaWriterConfig:
case class DeltaWriterConfig(
sparkDf: DataFrame,
columnToOrder: String,
destinationPath: String,
primaryKey: List[String],
filesCountFirstBatch: Int,
sparkContext: SparkSession,
operationType: String,
partitionColumn: Option[String] = None,
updateCondition: Option[String] = None,
setExpression: Option[String] = None
)
case object DeltaWriterConfig {
def from(deltaContextConfig: DeltaContextConfig): DeltaWriterConfig =
DeltaWriterConfig(
sparkDf = deltaContextConfig.sparkDf,
columnToOrder = deltaContextConfig.columnToOrder,
destinationPath = deltaContextConfig.destinationPath,
primaryKey = deltaContextConfig.primaryKey,
filesCountFirstBatch = deltaContextConfig.filesCountFirstBatch,
sparkContext = deltaContextConfig.sparkContext,
operationType = deltaContextConfig.operationType,
partitionColumn = deltaContextConfig.partitionColumn,
updateCondition = deltaContextConfig.updateCondition,
setExpression = deltaContextConfig.setExpression
)
}
it gives us opportunity to create new config just in one line:
val deltaContextConfig: DeltaContextConfig = ???
val deltaWriterConfig = DeltaWriterConfig.from(deltaContextConfig)
but the better solution is have only that configs that are unique. For example if we have duplicates fields in DeltaContextConfig and DeltaWriterConfig why we couldn't have just composition of config and not duplicate these fields:
// instead of this DeltaContextConfig declaration
case class DeltaContextConfig(
tableName: String,
databaseName: String,
autoCompaction: Option[Boolean] = Option(true),
idealFileSize: Option[Int] = Option(128),
deduplicationColumn: Option[String] = None,
compactionIntervalTime: Option[Int] = Option(180),
sparkDf: DataFrame,
columnToOrder: String,
destinationPath: String,
primaryKey: List[String],
filesCountFirstBatch: Int,
sparkContext: SparkSession,
operationType: String,
partitionColumn: Option[String] = None,
updateCondition: Option[String] = None,
setExpression: Option[String] = None
)
case class DeltaWriterConfig(
sparkDf: DataFrame,
columnToOrder: String,
destinationPath: String,
primaryKey: List[String],
filesCountFirstBatch: Int,
sparkContext: SparkSession,
operationType: String,
partitionColumn: Option[String] = None,
updateCondition: Option[String] = None,
setExpression: Option[String] = None
)
we use such config structure:
case class DeltaContextConfig(
tableName: String,
databaseName: String,
autoCompaction: Option[Boolean] = Option(true),
idealFileSize: Option[Int] = Option(128),
deduplicationColumn: Option[String] = None,
compactionIntervalTime: Option[Int] = Option(180),
deltaWriterConfig: DeltaWriterConfig
)
case class DeltaWriterConfig(
sparkDf: DataFrame,
columnToOrder: String,
destinationPath: String,
primaryKey: List[String],
filesCountFirstBatch: Int,
sparkContext: SparkSession,
operationType: String,
partitionColumn: Option[String] = None,
updateCondition: Option[String] = None,
setExpression: Option[String] = None
)
but remember you should use the same config structure in your config file.