How to save struct column as string to CSV/TSV in PySpark? - pyspark

Question

I've seen similar questions asked many times, but there's no clear answer to something that should be easy.
How can a struct column be saved to CSV (tsv actually) in PySpark? I want to serialize it and save as JSON.
I have a dataframe, which I read from parquet, that contains the following schema:
timestamp:long
timezoneOffset:string
dayInterval:integer
speed:double
heading:double
ignitionStatus:integer
segmentId:string
pointMM:struct
mmResult:array
element:struct
primitiveId:long
rnId:integer
internalId:integer
isFromTo:boolean
offset:double
probability:double
distanceToArc:double
headingDifference:double
isSuccessful:boolean
The pointMM column is a struct, that contains an array of structs, and another bool field (isSuccessful). I'm able to read this data from parquet and preview it:
If I want to save this data to CSV/TSV I get the following error:
df.write.csv(output_path, sep='\t')
AnalysisException: CSV data source does not support struct<mmResult:array<struct<primitiveId:bigint,rnId:int,internalId:int,isFromTo:boolean,offset:double,probability:double,distanceToArc:double,headingDifference:double>>,isSuccessful:boolean> data type.
Is there a way, even better an easy to convert the pointMM column to JSON string and save it to TSV?
Is there a way to do it with explicitly stating the schema of pointMM? Or a way to do it, even better, without knowing the schema?
I don't understand why this is difficult, because as you can see in the attached screenshot, that column is shown in JSON format.
EDIT 1: I understand that display() function somehow serializes the columns struct. Is there a way to use the same serialization without reinventing the wheel?
EDIT 2: .printSchema() shows the schema of the DataFrame. Can this be somehow used to help serialize pointMM column to JSON?

Answer 1

Use df.dtypes to get the type of each column. If type is a struct use to_json on this column to convert the data into a json string. Otherwise select the column as it is:
from pyspark.sql import functions as F
cols = [ F.to_json(c[0]).alias(c[0]) if c[1].startswith("struct") else F.col(c[0]) for c in df.dtypes]
df.select(cols).show(truncate=False)
Output:
+-----------+-------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-----+---------+--------------+
|dayInterval|heading|ignitionStatus|pointMM |segmentId|speed|timestamp|timezoneOffset|
+-----------+-------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-----+---------+--------------+
|1 |271.1 |4 |{"isSuccessful":true,"mmResult":[{"distanceToArc":12.211,"headingDifference":12.1,"internalId":5,"isFromTo":true,"offset":12.1,"primitiveId":12,"probability":0.12,"rnId":4},{"distanceToArc":12.211,"headingDifference":12.1,"internalId":5,"isFromTo":true,"offset":12.1,"primitiveId":13,"probability":0.12,"rnId":4}]}|abc |12.4 |12345678 |+1 |
+-----------+-------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-----+---------+--------------+