string manipulations using Spark scala - scala

Question

I have the following Spark scala dataframe.
val someDF = Seq(
(1, "bat",1.3222),
(4, "cbat",1.40222),
(3, "horse",1.501212)
).toDF("number", "word","value")
I created a User Defined Function (UDF) to create a new variable as follows :
Logic : if words equals bat then value else zero.
import org.apache.spark.sql.functions.{col}
val func1 = udf( (s:String ,y:Double) => if(s.contains("bat")) y else 0 )
func1(col("word"),col("value"))
+------+-----+-------+
|number| word|cal_var|
+------+-----+-------+
| 1| bat| 1.3222|
| 4| cbat|1.40222|
| 3|horse| 0.0|
+------+-----+-------+
Here to check the equality i used contains function . Because of that i am getting the incorrect output .
My desired output should be like this :
+------+-----+-------+
|number| word|cal_var|
+------+-----+-------+
| 1| bat| 1.3222|
| 4| cbat| 0.0|
| 3|horse| 0.0|
+------+-----+-------+
Can anyone help me to figure out the correct string function that i should use to check the equality ?
Thank you

Answer 1

Try to avoid using UDF's as it gives poor performance,
Another approach:
val someDF = Seq(
(1, "bat",1.3222),
(4, "cbat",1.40222),
(3, "horse",1.501212)
).toDF("number", "word","value")
import org.apache.spark.sql.functions._
someDF.show
+------+-----+--------+
|number| word| value|
+------+-----+--------+
| 1| bat| 1.3222|
| 4| cbat| 1.40222|
| 3|horse|1.501212|
+------+-----+--------+
someDF.withColumn("value",when('word === "bat",'value).otherwise(0)).show()
+------+-----+------+
|number| word| value|
+------+-----+------+
| 1| bat|1.3222|
| 4| cbat| 0.0|
| 3|horse| 0.0|
+------+-----+------+

Answer 2

The solution is to use equals method rather than contains. contains checks whether string bat is present anywhere in the given string s and not the equality. The code is shown below:
scala> someDF.show
+------+-----+--------+
|number| word| value|
+------+-----+--------+
| 1| bat| 1.3222|
| 4| cbat| 1.40222|
| 3|horse|1.501212|
+------+-----+--------+
scala> import org.apache.spark.sql.functions._
import org.apache.spark.sql.functions._
scala> val func1 = udf( (s:String ,y:Double) => if(s.equals("bat")) y else 0 )
func1: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function2>,DoubleType,Some(List(StringType, DoubleType)))
scala> someDF.withColumn("col_var", func1(col("word"),col("value"))).drop("value").show
+------+-----+-------+
|number| word|col_var|
+------+-----+-------+
| 1| bat| 1.3222|
| 4| cbat| 0.0|
| 3|horse| 0.0|
+------+-----+-------+
Let me know if it helps!!