PySpark: Explode schema columns does not match with underlying nested schema - pyspark

Question

I am using pyspark in combination with Azure-Synapse. I am reading multiple nested JSON with the same structure in a dataframe using the below sample:
{
"AmountOfOrders": 2,
"TotalEarnings": 1800,
"OrderDetails": [
{
"OrderNumber": 1,
"OrderDate": "2022-7-06",
"OrderLine": [
{
"LineNumber": 1,
"Product": "Laptop",
"Price": 1000
},
{
"LineNumber": 2,
"Product": "Tablet",
"Price": 500
},
{
"LineNumber": 3,
"Product": "Mobilephone",
"Price": 300
}
]
},
{
"OrderNumber": 2,
"OrderDate": "2022-7-06",
"OrderLine": [
{
"LineNumber": 1,
"Product": "Printer",
"Price": 100,
"Discount": 0
},
{
"LineNumber": 2,
"Product": "Paper",
"Price": 50,
"Discount": 0
},
{
"LineNumber": 3,
"Product": "Toner",
"Price": 30,
"Discount": 0
}
]
}
]
}
I am trying to get the the LineNumbers of Ordernumber 1 in a separate dataframe using a generic function which extract the array and Struct of the dataframe. Using the code below:
def read_nested_structure(df,excludeList,messageType,coll):
display(df.limit(10))
print('read_nested_structure')
cols =[]
match = 0
match_field = ""
print(df.schema[coll].dataType.fields)
for field in df.schema[coll].dataType.fields:
for c in excludeList:
if c == field.name:
print('Match = ' + field.name)
match = 1
if match == 0:
# cols.append(coll)
cols.append(col(coll + "." + field.name).alias(field.name))
match = 0
# cols.append(coll)
print(cols)
df = df.select(cols)
return df
def read_nested_structure_2(df,excludeList,messageType):
cols =[]
match = 0
for coll in df.schema.names:
if isinstance(df.schema[coll].dataType, ArrayType):
print( coll + "-- : Array")
df = df.withColumn(coll, explode(coll).alias(coll))
cols.append(coll)
elif isinstance(df.schema[coll].dataType, StructType):
if messageType == 'Header':
for field in df.schema[coll].dataType.fields:
cols.append(col(coll + "." + field.name).alias(coll + "_" + field.name))
elif messageType == 'Content':
print('Struct - Content')
for field in df.schema[coll].dataType.fields:
cols.append(col(coll + "." + field.name).alias(field.name))
else:
for c in excludeList:
if c == coll:
match = 1
if match == 0:
cols.append(coll)
df = df.select(cols)
return df
df = spark.read.load(datalakelocation + '/test.json', format='json')
df = unpack_to_content_dataframe_simple_2(df,exclude)
df = df.filter(df.OrderNumber == 1)
df = unpack_to_content_dataframe_simple_2(df,exclude)
display(df.limit(10))
which result in the following dataframe:
as you can see the yellow marked attribute is added to the dataframe which is not part of OrderNumber 1. How can I filter a row in the dataframe which results in a update schema ( in this case without the Discount attribute)?

Answer 1

I have used read_nested_structure_2() function in the following way to get the same results as yours. The code I used to get this result using read_nested_structure_2() is as follows:
x = read_nested_structure_2(df,[],'Header')
y = read_nested_structure_2(x,[],'Content')
y = y.filter(y.OrderNumber == 1)
z = read_nested_structure_2(y,[],'Header')
final = read_nested_structure_2(z,[],'Content')
display(final)
The output of after using this code is:
The column Discount will be created even if it is specified for one Product in the entire input JSON. In order to remove this column, we have to do it separately to get another dataframe without Discount (only if it is invalid).
You are going to use the same function to extract data from StructType or ArrayType, it is not recommended to write code to remove fields (say Discount) having all null values, in the same function. Doing so would complicate the code.
Instead, we can write another function which does this work for us. This function should remove a column where all of its values are null. The following is the function that can be used to do this.
def exclude_fields_that_dont_exist(filtered_df):
cols=[]
#iterate through columns
for column in filtered_df.columns:
#null_count is the count of null values in a column
null_count = filtered_df.filter(filtered_df[column].isNull()).count()
#check if null_count equals the total column value count
#if they are equal, those columns are not required (Like Discount)
if(filtered_df.select(column).count() != null_count):
cols.append(column)
#return dataframe with required columns
return filtered_df.select(*cols)
When you use this function on the filtered dataframe (final in my case), then you get a resulting dataframe as shown below:
mydf = exclude_fields_that_dont_exist(final)
# removes columns from a dataframe that have all null values.
display(mydf)
NOTE:
For example, for OrderNumber=1, the product Laptop has a 10% discount and the rest of the products (for the same order number) don't have a discount value (in the JSON).
The function needs to include the Discount column since it is a required information.
To avoid using more loops inside a function, you can also consider replacing all the null values with 0 since a Product with no Discount specified (null value) is same as a Product with Discount value as 0 (If this is feasible, then you can use fill() or fillna() functions to fill null values with any desired value)