Spark DF schema verification with Decimal column - pyspark

Question

I want to verify the schema of a Spark dataframe against schema information it get from some other source (a dashboard tool). The information I get about the table is field name and field type (nullability is not important at this point).
However, for DecimalType columns I do not get the information about precision and scale (the two parameters of DecimalType). So I have to ignore these values in the comparison.
I currently re-write the schema so that the Decimal columns become Float column. But is there a more elegant way to do that?
Basically I want to write a function is_schema_valid() that works as such:
from pyspark.sql import types as T
df_schema = T.StructType([
T.StructField('column_1', T.StringType(), True),
T.StructField('column_2', T.DecimalType(20,5), True), # values in DecimalType are random
])
schema_info = [('column_1', 'String'), ('column_2', 'Decimal')]
is_schema_valid(schema_info, df_schema)
# Output: True

Answer 1

The best would probably to compare similar objects. You can transform a schema in JSON object (or python dict).
import json
_df_schema_dict = json.loads(df_schema.json())
df_schema_dict = {
field["name"]: field["type"]
for field in _df_schema_dict["fields"]
}
df_schema_dict
> {'column_1': 'string', 'column_2': 'decimal(20,5)'}
You can work with this object to compare with schema_info. Here is a very basic test you can do (I change a bit the content of schema_info):
import json
def is_schema_valid(schema_info, df_schema):
df_schema_dict = {
field["name"]: field["type"] for field in json.loads(df_schema.json())["fields"]
}
schema_info_dict = {elt[0]: elt[1] for elt in schema_info}
return schema_info_dict == df_schema_dict
df_schema = T.StructType(
[
T.StructField("column_1", T.StringType(), True),
T.StructField("column_2", T.DecimalType(20, 5), True),
]
)
schema_info = [("column_1", "string"), ("column_2", "decimal(20,5)")]
is_schema_valid(schema_info, df_schema)
# True
If you want to ignore decimal precision, you can always twist a little bit the dataframe schema. Replace field["type"] with field["type"] if "decimal" not in field["type"] else "decimal" for example.
import json
def is_schema_valid(schema_info, df_schema):
df_schema_dict = {
field["name"]: field["type"] if "decimal" not in field["type"] else "decimal"
for field in json.loads(df_schema.json())["fields"]
}
schema_info_dict = {elt[0]: elt[1] for elt in schema_info}
return schema_info_dict == df_schema_dict
df_schema = T.StructType(
[
T.StructField("column_1", T.StringType(), True),
T.StructField("column_2", T.DecimalType(20, 5), True),
]
)
schema_info = [("column_1", "string"), ("column_2", "decimal")]
is_schema_valid(schema_info, df_schema)
# True