reading csv to dataframe with dynamic custom schema with pyspark - pyspark

Question

I'm working with databricks in a notebook. I want to read csv files with custom schema.
I'd like to be able to loop over all the csv files in a folder and read them with their respective schema.
So I have a schema for each csv file:
csv_1 = StructType([
StructField('foo', StringType(), False),
StructField('bar', StringType(), True),
])
csv_2 = StructType([
StructField('foo', StringType(), False),
StructField('bar', StringType(), True),
StructField('baz', StringType(), True),
])
csv_3 = StructType([
StructField('bar', StringType(), True)
])
Then I have this loop:
for file in os.listdir(path):
filename = os.path.splitext(file)[0]
dataframes[filename] = spark.read.csv(path+file, header=True, schema=???)
I guess I probably need to use some mapping somewhere but I'm not sure how.

Answer 1

filename_to_related_mapping = {
'name1': csv_1,
'name2': csv_2,
...
}.get(filename)
for file in os.listdir(path):
filename = os.path.splitext(file)[0]
dataframes[filename] = spark.read.csv(path+file, header=True, schema=filename_to_related_mapping[filename])
Anyway its just a CSV, another way is not pass the schema and it will be infered dynamically.