How to change the schema of existing dataframe - scala

Question

Problem statement : I have a csv file with around 100+ fields.I need to perform transformation on these fields and generate new 80+ fields and write only these new fields into s3 in parquet format.
The parquet predefined schema = 80+ newly populated fields + some non populated fields.
Is there any way to pass this predefined parquet schema while writing data to s3 so that these extra fields also populated with null data.
select will not be useful to select only 80+ fields as predefined schema might have around 120 predefined fields.
Below is sample data and transformation requirementCSV data
aid, productId, ts, orderId
1000,100,1674128580179,edf9929a-f253-487
1001,100,1674128580179,cc41a026-63df-410
1002,100,1674128580179,9732755b-1207-471
1003,100,1674128580179,51125ddd-4129-48a
1001,200,1674128580179,f4917676-b08d-41e
1004,200,1674128580179,dc80559d-16e6-4fa
1005,200,1674128580179,c9b743eb-457b-455
1006,100,1674128580179,e8611141-3e0e-4d5
1002,200,1674128580179,30be34c7-394c-43a
Parquet schema
def getPartitionFieldsSchema() = {
List(
Map("name" -> "company", "type" -> "long",
"nullable" -> true, "metadata" -> Map()),
Map("name" -> "epoch_day", "type" -> "long",
"nullable" -> true, "metadata" -> Map()),
Map("name" -> "account", "type" -> "string",
"nullable" -> true, "metadata" -> Map()),
)
}
val schemaMap = Map("type" -> "struct",
"fields" -> getPartitionFieldsSchema)
simple example
val dataDf = spark
.read
.format("csv")
.option("header", "true")
.option("inferSchema", "true")
.load("./scripts/input.csv")
dataDf
.withColumn("company",lit(col("aid")/100))
.withColumn("epoch_day",lit(col("ts")/86400))
.write // how to write only company, epoch_day, account ?
.mode("append")
.csv("/tmp/data2")
Output should have below columns: company, epoch_day, account

Answer 1

This is how I understand your problem:
you wanna read some csv and transform them to parquet in s3.
during the transformation, you need to create 3 new cols based on existing cols in csv files.
but since only 2 out of 3 new cols are calculated, the output are only showing two new cols but not 3.
In such case, you can create a external table in redshift, and you specify all the cols. As a result even some colums are not fed, there would be null in your external tables.