Flatten Json Key, values in Pyspark - pyspark

Question

In a table having 2 columns and 2 Records :
Record 1 : Column 1 - my_col value as: {"XXX": ["123","456"],"YYY": ["246","135"]} and Column 2 - ID as A123
Record 2 : Column 1 - my_col value as: {"XXX": ["123","456"],"YYY": ["246","135"], "ZZZ":["333","444"]} and Column 2 - ID as B222
Need to parse/flatten using pyspark
Expectation :
Key
Value
ID
XXX
123
A123
XXX
456
A123
YYY
246
A123
YYY
135
A123
ZZZ
333
B222
ZZZ
444
B222

Answer 1

If your column is a string, you may use the from_json and custom_schema to convert it to a MapType before using explode to extract it into the desired results. I assumed that your initial column was named my_col and that your data was in a dataframe named input_df.
An example is shown below
Approach 1: Using pyspark api
from pyspark.sql import functions as F
from pyspark.sql import types as T
custom_schema = T.MapType(T.StringType(),T.ArrayType(T.StringType()))
output_df = (
input_df.select(
F.from_json(F.col('my_col'),custom_schema).alias('my_col_json')
)
.select(F.explode('my_col_json'))
.select(
F.col('key'),
F.explode('value')
)
)
Approach 2: Using spark sql
# Step 1 : Create a temporary view that may be queried
input_df.createOrReplaceTempView("input_df")
# Step 2: Run the following sql on your spark session
output_df = sparkSession.sql("""
SELECT
key,
EXPLODE(value)
FROM (
SELECT
EXPLODE(from_json(my_col,"MAP<STRING,ARRAY<STRING>>"))
FROM
input_df
) t
""")
For json column
If already json
from pyspark.sql import functions as F
output_df = (
input_df.select(F.explode('my_col_json'))
.select(
F.col('key'),
F.explode('value')
)
)
or
# Step 1 : Create a temporary view that may be queried
input_df.createOrReplaceTempView("input_df")
# Step 2: Run the following sql on your spark session
output_df = sparkSession.sql("""
SELECT
key,
EXPLODE(value)
FROM (
SELECT
EXPLODE(my_col)
FROM
input_df
) t
""")
Let me know if this works for you.