flattening nested dict type json file - pyspark

Question

We are trying to flatten a nested json with below format :
{
"Type" : "Notification",
"MessageId" : "5b37cfab-8a88-5609-b179-93f87126030a",
"SequenceNumber" : "10000000000000010000",
"TopicArn" : "arn:aws:sns:us-east-1:956978673417:de-eec-org-group-association-info.fifo",
"Message" : "{\n\"messageType\": \"NEW_USER_ASSOCIATION\",\n\"messageStatus\": \"SUCCESS\",\n\"messageDetails\": \"Successfully Associated\",\n\"organizationId\": 1000830784,\n\"organizationName\": \"Pfizer\",\n\"emailDomains\": [\n\"#pfizer.com\",\n\"#gmail.com\"\n],\n\"parentDunsNumber\": \"879262386\",\n\"cdcId\": \"49f5b88e036d46f38e05ce260aeaeb2a\",\n\"isUserActive\": true,\n\"associatedDateTime\": 1674467158154,\n\"associatedBy\": \"01bdd9a4a21345929b7fa06acd3c53d5\"\n}",
"Timestamp" : "2023-02-01T05:32:30.825Z",
"UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:956978673417:de-eec-org-group-association-info.fifo:e986e954-6f3d-4309-b7c4-a4df2182227e"
}
Tried different way but it's not working , mostly the type of Message column is stringType need to change it in MapType and need to flatten the entire json file . Can anyone help?

Answer 1

Please try the following solution with explode() function :
from pyspark.sql.types import StructType, StructField, StringType, ArrayType, IntegerType, BooleanType
from pyspark.sql.functions import explode
# Define the schema for the nested JSON
schema = StructType([
StructField("messageType", StringType()),
StructField("messageStatus", StringType()),
StructField("messageDetails", StringType()),
StructField("organizationId", IntegerType()),
StructField("organizationName", StringType()),
StructField("emailDomains", ArrayType(StringType())),
StructField("parentDunsNumber", StringType()),
StructField("cdcId", StringType()),
StructField("isUserActive", BooleanType()),
StructField("associatedDateTime", IntegerType()),
StructField("associatedBy", StringType())
])
df = spark.createDataFrame([input_json])
df_message = df.select("Message").select(from_json("Message", schema).alias("data"))
df_flat = df_message.select("data.*").select(explode("emailDomains").alias("emailDomains"))
df_result = df.join(df_flat, "MessageId", "left").drop("Message")
# Show the result
df_result.show()