Dataframe headers get modified on saving as parquet file using glue - pyspark

Question

I'm writing a dataframe with headers by partitions to s3 using the below code:
df_dynamic = DynamicFrame.fromDF(
df_columned,
glue_context,
"temp_ctx"
)
print("\nUploading parquet to " + destination_path)
glue_context.write_dynamic_frame.from_options(
frame=df_dynamic,
connection_type="s3",
connection_options={
"path": destination_path,
"partitionKeys" : ["partition_id"]
},
format_options={
"header":"true"
},
format="glueparquet"
)
Once my files are created I see I have #1, #2 added after my column headers.
Example: if my column name is "Doc Data", it gets converted to Doc_Date#1
I thought its a parquet way of saving data.
Then when I try to read from the same files using the below code, my headers are no more the same. Now they come as Doc_Date#1. How do I fix this?
str_folder_path = str.format(
_S3_PATH_FORMAT,
args['BUCKET_NAME'],
str_relative_path
)
df_grouped = glue_context.create_dynamic_frame.from_options(
"s3",
{
'paths': [str_folder_path],
'recurse':True,
'groupFiles': 'inPartition',
'groupSize': '1048576'
},
format_options={
"header":"true"
},
format="parquet"
)
return df_grouped.toDF()

Answer 1

Issue Resolved!!
Issue was that I had spaces in my column names. Once I replaced them with underscores(_), the issue resolved.