Getting Null Pointer Excepetion while Using Spark UDFs - scala

Question

I created a Dataframe and wrote a UDF for lowercase to uppercase conversion.
But when I tried calling nameUdf, I got NullPointerException.
case class Employee(id:Int, name:String, salary:Double)
val empList=List("111,aaa,20000.0", "222,bbb,300.00", "333,ccc,4000.00")
val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
val empDF=sc.parallelize(empList).map{line=>
val data=line.split(",")
Employee(data(0).toInt,data(1),data(2).toDouble)
}.toDF()
empDF.withColumn("NAME_UP",convert($"name")).show()
val nameUdf=udf{(name:String)=>name.toUpperCase}
val convert=udf[String,String](name=>name.toUpperCase)
Exception is shown below:
18/03/21 14:08:10 INFO BlockManagerMaster: Registered BlockManager
Exception in thread "main" java.lang.NullPointerException
at com.org.test.UDFTest$.delayedEndpoint$com$org$test$UDFTest$1(UDFTest.scala:22)
at com.org.test.UDFTest$delayedInit$body.apply(UDFTest.scala:8)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at com.org.test.UDFTest$.main(UDFTest.scala:8)
at com.org.test.UDFTest.main(UDFTest.scala)
18/03/21 14:08:12 INFO SparkContext: Invoking stop() from shutdown hook
How to call the UDF function with the existing dataframe?

Answer 1

Make sure the udf is defined before you use and case class to outside the object scope:
object QuickTest extends App {
val spark = SparkSession.builder().appName("test").master("local[*]").getOrCreate()
val empList=List("111,aaa,20000.0","222,bbb,300.00","333,ccc,4000.00")
import spark.implicits._
val empDF=spark.sparkContext.parallelize(empList).map{line=>
val data=line.split(",")
Employee(data(0).toInt,data(1),data(2).toDouble)
}.toDF()
val nameUdf=udf{(name:String)=>name.toUpperCase}
val convert=udf[String,String](name=>name.toUpperCase)
empDF.withColumn("NAME_UP",convert($"name")).show()
}//end of object QuickTest
case class Employee(id:Int,name:String,salary:Double)