How to convert multiple json files to single csv in Pyspark - pyspark

Question

I have multiple files in ADLS I want to convert them into single csv but without using Pandas. Is it possible to convert them using Pyspark?
These files are coming from API which has 225 000 records. I am using this script to convert it to csv

Answer 1

You can place all JSON files in a folder and import all using: spark.read.option("multiLine", True).json("/path/to/folder").
However, importing JSON files into dataframe is little trickier as you may not get desired format and you may have to preprocess JSON file before import or Spark dataframe after import.
For example, assume JSON files per continent:
NA.json
{
"US" : {
"capital": "Washington, D.C.",
"population in million": 330
},
"Canada" : {
"capital": "Ottawa",
"population in million": 38
}
}
EU.json
{
"England" : {
"capital": "London",
"population in million": 56
},
"France" : {
"capital": "Paris",
"population in million": 67
}
}
AUS.json
{
"Australia" : {
"capital": "Canberra",
"population in million": 25
},
"New Zealand" : {
"capital": "Wellington",
"population in million": 5
}
}
These files get imported with root JSON objects mapped to each column and nested JSON data mapped as nested map:
df = spark.read.option("multiLine", True).json("/content/sample_data/json")
+--------------+------------+------------+-----------+---------------+--------------------+
| Australia| Canada| England| France| New Zealand| US|
+--------------+------------+------------+-----------+---------------+--------------------+
|{Canberra, 25}| null| null| null|{Wellington, 5}| null|
| null|{Ottawa, 38}| null| null| null|{Washington, D.C....|
| null| null|{London, 56}|{Paris, 67}| null| null|
+--------------+------------+------------+-----------+---------------+--------------------+
Depending on structure of your JSON files, you will have to deal with 2 things:
Mapping JSON object to a generalised schema to prevent fragmented column names as above.
Preventing nested data to be mapped as nested map type as shown with {Canberra, 25} above. You may want to transform this to bring all data to tabular form.