Pyspark Dataframe Creation DecimalType issue - pyspark

Question

I am trying to create a pyspark dataframe from a list of dict and a defined schema for the dataframe. One column in the defined schema is a DecimalType. While I create the dataframe, I get an error;
TypeError: field b: DecimalType(38,18) can not accept object 0.1 in type <class 'float'>
test_data = [{"a": "1", "b": 0.1}, {"a": "2", "b": 0.2}]
schema = StructType(
[
StructField("a", StringType()),
StructField("b", DecimalType(38, 18)),
]
)
# Create an dataframe
df = spark.createDataFrame(data = test_data,
schema = schema)
Could someone help out with this issue. How can I pass a decimaltype data in a list.?

Answer 1

If you can lose some accuracy then you can change the type to FloatType as Bala suggested .
You can also change to DoubleType if you need more accuracy. FloatType support 4 bytes of information while DoubleType have 8 bytes (see here).
If you need maximum accuracy, you can use Pythons Decimal module that by default has 28 digits after the dot:
from pyspark.sql.types import *
from decimal import Decimal
test_data = [{"a": "1", "b": Decimal(0.1) }, {"a": "2", "b": Decimal(0.2) }]
schema = StructType(
[
StructField("a", StringType()),
StructField("b", DecimalType(38, 18)),
]
)
# Create a dataframe
df = spark.createDataFrame(data = test_data,
schema = schema)
If for example we run this code:
from pyspark.sql.types import *
from decimal import Decimal
test_data = [
(1.9868968969869869045652421846, 1.9868968969869869045652421846, Decimal(1.9868968969869869045652421846)),
]
schema = StructType(
[
StructField("float_col", FloatType()),
StructField("double_col", DoubleType()),
StructField("decimal_col", DecimalType(38, 28)),
]
)
# Create an dataframe
df = spark.createDataFrame(data = test_data,
schema = schema)
we would get this difference:

Answer 2

Change it to FloatType
test_data = [{"a": "1", "b": 0.1}, {"a": "2", "b": 0.2}]
schema2 = StructType(
[
StructField("a", StringType()),
StructField("b", FloatType()),
]
)
df = spark.createDataFrame(data=test_data,schema=schema2)
df.show()
+---+---+
| a| b|
+---+---+
| 1|0.1|
| 2|0.2|
+---+---+