Using default argument values in Scala UDF from pyspark? - scala

Question

I have a UDF defined in Scala with a default argument value like so:
package myUDFs
import org.apache.spark.sql.api.java.UDF3
class my_udf extends UDF3[Int, Int, Int, Int] {
override def call(a: Int, b: Int, c: Int = 6): Int = {
c*(a + b)
}
}
I then build this appropriately with build clean assembly (can provide more build details if needed) and extract the jar myUDFs-assembly-0.1.1.jar and include that in my Spark configuration in Python:
from pyspark.conf import SparkConf
from pyspark.sql import SparkSession
from pyspark.sql.types import IntType
spark_conf = SparkConf().setAll([
('spark.jars', 'myUDFs-assembly-0.1.1.jar')
])
spark = SparkSession.builder \
.appName('my_app') \
.config(conf = spark_conf) \
.enableHiveSupport() \
.getOrCreate()
spark.udf.registerJavaFunction(
"my_udf", "myUDFs.my_udf", IntType()
)
But, when I try to leverage the default, I'm rebuffed:
spark.sql('select my_udf(1, 2)').collect()
AnalysisException: 'Invalid number of arguments for function my_udf. Expected: 3; Found: 2; line x pos y'
Is it not possible to have a UDF with a default value like this? The output should be 6*(1+2) = 18.

Answer 1

Just looking at the call chain there is no chance for the default argument to be recognized here.
Python registerJavaFunction invokes its JVM UDFRegistration.registerJava.
registerJava invokes matching register implementation.
Which, in case of UDF3, looks like this:
* Register a deterministic Java UDF3 instance as user-defined function (UDF).
* #since 1.3.0
*/
def register(name: String, f: UDF3[_, _, _, _], returnType: DataType): Unit = {
val func = f.asInstanceOf[UDF3[Any, Any, Any, Any]].call(_: Any, _: Any, _: Any)
def builder(e: Seq[Expression]) = if (e.length == 3) {
ScalaUDF(func, returnType, e, e.map(_ => true), udfName = Some(name))
} else {
throw new AnalysisException("Invalid number of arguments for function " + name +
". Expected: 3; Found: " + e.length)
}
functionRegistry.createOrReplaceTempFunction(name, builder)
}
As you can see, the builder only verifies if the provided expression matches the arity of the function before the call is actually dispatched.
You might have a better luck with implementing an intermediate API which would handle default arguments and dispatch to UDF under the covers. This however will work only with DataFrame API, so it might not fit your needs.

Answer 2

You are passing only two argument while calling the function in spark sql. Try to pass three arguments
spark.sql('select my_udf(1, 2, 3 )').collect()