Checking whether a column has proper decimal number for special case - pyspark

Question

I have a dataframe (input_dataframe), which looks like as below:
id test_column
1 0.25
2 1.1
3 12
4 test
5 1.3334
6 12.0
I want to add a column result, which put values 1 if test_column has a decimal value and 0 if test_column has any other value. data type of test_column is string. Below is the expected output:
id test_column result
1 0.25 1
2 1.1 1
3 12 0
4 test 0
5 1.3334 1
6 12.0 1
I have below code for this operation:
import decimal
from pyspark.sql.types import IntType
def is_valid_decimal(s):
try:
return (0 if decimal.Decimal(val)._isinteger() else 1)
except decimal.InvalidOperation:
return 0
# register the UDF for usage
sqlContext.udf.register("is_valid_decimal", is_valid_decimal, IntType())
# Using the UDF
df.withColumn("result", is_valid_decimal("test_column"))
However this code is not working, when decimal values are like: 12.0 or 12.00 or 12.000
Is there a way this can be achieved in pyspark?

Answer 1

You mentioned it's a string column, so, I tired using regular expression. hope it helps,
>>> from pyspark.sql import functions as F
>>> from pyspark.sql.types import IntegerType
>>> import re
>>> df = spark.createDataFrame([(1,'0.25'),(2,'1.1'),(3,'12'),(4,'test'),(5,'1.3334'),(6,'12.0')],['id','test_col'])
>>> df.show()
+---+--------+
| id|test_col|
+---+--------+
| 1| 0.25|
| 2| 1.1|
| 3| 12|
| 4| test|
| 5| 1.3334|
| 6| 12.0|
+---+--------+
>>> udf1 = F.udf(lambda x : 1 if re.match('^\d*[.]\d*$',x) else 0,IntegerType())
>>> df = df.withColumn('result',udf1(df.test_col))
>>> df.show()
+---+--------+------+
| id|test_col|result|
+---+--------+------+
| 1| 0.25| 1|
| 2| 1.1| 1|
| 3| 12| 0|
| 4| test| 0|
| 5| 1.3334| 1|
| 6| 12.0| 1|
+---+--------+------+