Pyspark: Output to csv -- Timestamp format is different - pyspark

Question

I am working with a dataset with the following Timestamp format: yyyy-MM-dd HH:mm:ss
When I output the data to csv the format changes to something like this: 2019-04-29T00:15:00.000Z
Is there any way to get it to the original format like: 2019-04-29 00:15:00
Do I need to convert that column to string and then push it to csv?
I am saying my file to csv like so:
df.coalesce(1).write.format("com.databricks.spark.csv"
).mode('overwrite'
).option("header", "true"
).save("date_fix.csv")

Answer 1

Alternative
spark >=2.0.0
set option("timestampFormat", "yyyy-MM-dd HH:mm:ss") for format("csv")
df.coalesce(1).write.format("csv"
).mode('overwrite'
).option("header", "true"
).option("timestampFormat", "yyyy-MM-dd HH:mm:ss"
).save("date_fix.csv")
As per documentation-
timestampFormat (default yyyy-MM-dd'T'HH:mm:ss.SSSXXX): sets the string that indicates a timestamp format. Custom date formats follow the formats at java.text.SimpleDateFormat. This applies to timestamp type.
spark < 2.0.0
set option("dateFormat", "yyyy-MM-dd HH:mm:ss") for format("csv")
df.coalesce(1).write.format("com.databricks.spark.csv"
).mode('overwrite'
).option("header", "true"
).option("dateFormat", "yyyy-MM-dd HH:mm:ss"
).save("date_fix.csv")
As per documentation-
dateFormat: specifies a string that indicates the date format to use when reading dates or timestamps. Custom date formats follow the formats at java.text.SimpleDateFormat. This applies to both DateType and TimestampType. By default, it is null which means trying to parse times and date by java.sql.Timestamp.valueOf() and java.sql.Date.valueOf()
ref - readme

Answer 2

Yes, that's correct. The easiest way to achieve this is using pyspark.sql.functions.date_format such as:
import pyspark.sql.functions as f
df.withColumn(
"date_column_formatted",
f.date_format(f.col("timestamp"), "yyyy-MM-dd HH:mm:ss")
)
More info about it here https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.date_format.
Hope this helps!