Pyspark Sql Type: Union[int, float] - pyspark

Question

I am ingesting a data type that is normally an int, but could also be None or inf and creating a Spark DataFrame with it. I tried making it a LongType, by PySpark complains because inf is a float:
File "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 177, in main
process()
File "/opt/spark/python/lib/pyspark.zip/pyspark/worker.py", line 172, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 268, in dump_stream
vs = list(itertools.islice(iterator, batch))
File "/opt/spark/python/pyspark/sql/session.py", line 567, in prepare
verify_func(obj, schema)
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/types.py", line 1355, in _verify_type
_verify_type(obj.get(f.name), f.dataType, f.nullable)
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/types.py", line 1329, in _verify_type
raise TypeError("%s can not accept object %r in type %s" % (dataType, obj, type(obj)))
TypeError: LongType can not accept object inf in type <class 'float'>
How can I support this in pyspark.sql.types ?

Answer 1

For now, I have simply mapped the field to a float and used a DoubleType in the schema:
def convert_field(x):
try:
field = x.pop("fieldName")
except KeyError:
return x
return dict(fieldName=float(field) if field is not None else field, **x)
results = ...
spark.createDataFrame(results.map(convert_field), results_schema).cache