How to add columns to a dataframe without using withcolumn - pyspark

Question

I need to loop through a json file, flatten the results and add a column to a dataframe in each loop with respective values. But the end result will have around ~2000 columns. So, using withColumn to add the columns is extremely slow. Is their any other alternative to add columns to a dataframe?
Sample Input json:
[
{
"ID": "12345",
"Timestamp": "20140101",
"Usefulness": "Yes",
"Code": [
{
"event1": "A",
"result": "1"
}
]
},
{
"ID": "1A35B",
"Timestamp": "20140102",
"Usefulness": "No",
"Code": [
{
"event1": "B",
"result": "1"
}
]
}
]
My output should be:
ID Timestamp Usefulness Code_event1 Code_result
12345 20140101 Yes A 1
1A35B 20140102 No B 1
The json file I am working on is huge and consists of many columns. So, withColumn is not feasible in my case.
EDIT:
Sample code:
# Data file
df_data = spark.read.json(file_path)
# Schema file
with open(schemapath) as fh:
jsonschema = json.load(fh,object_pairs_hook=OrderedDict)
I am looping through the schema file and in the loop I am accessing the data for a particular key from the data DF (df_data). I am doing this because my data file has multiple records so I cant loop through the data json file or it will loop through every record.
def func_structs(json_file):
for index,(k,v) in enumerate(json_file.items()):
if isinstance(v, dict):
srccol = k
func_structs(v)
elif isinstance(v, list):
srccol = k
func_lists(v) # Separate function to loop through list elements to find nested elements
else:
try:
df_data = df_data.withColumn(srcColName,df_Data[srcCol])
except:
df_data = df_data.withColumn(srcColName,lit(None).cast(StringType()))
func_structs(jsonschema)
I am adding columns to the data DF (df_data) itself.

Answer 1

One way is to use Spark's built-in json parser to read the json into a DF:
df = (sqlContext
.read
.option("multiLine", True)
.option("mode", "PERMISSIVE")
.json('file:///mypath/file.json')) # change as necessary
The result is as follows:
+--------+-----+---------+----------+
| Code| ID|Timestamp|Usefulness|
+--------+-----+---------+----------+
|[[A, 1]]|12345| 20140101| Yes|
|[[B, 1]]|1A35B| 20140102| No|
+--------+-----+---------+----------+
The second step is then to break out the struct inside the Code column:
df = df.withColumn('Code_event1', f.col('Code').getItem(0).getItem('event1'))
df = df.withColumn('Code_result', f.col('Code').getItem(0).getItem('result'))
df.show()
which gives
+--------+-----+---------+----------+-----------+-----------+
| Code| ID|Timestamp|Usefulness|Code_event1|Code_result|
+--------+-----+---------+----------+-----------+-----------+
|[[A, 1]]|12345| 20140101| Yes| A| 1|
|[[B, 1]]|1A35B| 20140102| No| B| 1|
+--------+-----+---------+----------+-----------+-----------+
EDIT:
Based on comment below from #pault, here is a neater way to capture the required values (run this code after load statement):
df = df.withColumn('Code', f.explode('Code'))
df.select("*", "Code.*")