How to use a function or method on a Spark data frame column for transformation using Scala - scala

Question

I have created a function in scala equivalant to ORACLE DECODE function. I want to use the function with SPARK dataframes columns. I have tried it but getting multiple issues with Datatype mismatches.
I do not want to create UDF for each program. I want to create something generic and reuse it multiple times.
Function:
def ODECODE(column: Any, Param: Any*) : Any = {
var index = 0
while (index < Param.length) {
var P = Param(index)
var Q = column
if (P.equals(Q))
return Param(index + 1)
else index = index + 1
}
return Param (Param.length - 1)
}
I want to use it some thing like this:
Assuming "Emp" is a dataframe containing data from employee table with columns(first name, Last Name, Grade).
Emp.select(ODECODE("grade", "A", 1, "B", 2, "C", 3, "FAIL")).show()
This is one example. The datatype in the grade column can be String or Integer. So I have taken Datatypes in the decode function (Above) as ANY but with Dataframes it does not perform the Transformation. It gives datatype mismatches.
I want to create individual functions/Methods for some of the unsupported Oracle functions and reuse them where ever required in my transformations. So any suggestion to make this work is appreciated.

Answer 1

I know this is late, but I actually needed this and found your example. I was able to implement it with a few changes. I am no expert though, there may be a better way of doing this.
import util.control.Breaks._;
def ODECODE[T](column: String, params: Seq[T]) : String = {
try {
var index = 0;
breakable {
while (index < params.length) {
var P = params(index);
var Q = column;
if(P.equals(Q)) {
break;
}
index += 1;
}
}
params(index - 1).toString;
}catch {
case ife: Exception =>
ife.printStackTrace();
"0";
}
}
println(ODECODE("TEST", 0, "TEgST", 8, "***", 0))