ClassCastException when using two Dataframes in a UDF - scala

Question

I Am relativley new to both Object Orientend an Functional programming so please forgive me if I am asking a dumb question here. I have searched thoroughly and tried to find answer myself for hours but to no avail.
I am working withh Spark2 on scala an have the following problem.
I have two Dataframes both of which I would like to pass to a UDF which should output one resultant Dataframe. For simplicities sake I have created two "test" dataframes as input.
val seqs = List(("1234","abaab"), ("1235","aaab")).toDF
val actions = List(1,2).toDF
The actions list dataframe will be converted to an Array of Strings e.g.:
val actionsARR = actions.rdd.map(x=>(x.getInt(0)+96).toChar.toString).collect()
We then have a UDF to count the occurances of the actions in the seqs:
def countActions(sequence:String, actions:Array[String]):Array[Int] = {
return actions.map(x => x.r.findAllIn(sequence).size)
}
put together with :
val results = seqs.map(x=>(x.getString(0), countActions(x.getString(1),actionsARR))).toDF("sequence_u_key","action_counter")
This works fine in the spark shell running one command after the other, I now try to embed the code in a further UDF which will accept Dataframes as input:
def testFun(seqsin:DataFrame, actionsin:DataFrame ): DataFrame ={
seqsin.map(x=>(x.getString(0), countActions(x.getString(1),actionsARR))).toDF("sequence_u_key","action_counter")
}
calling this with :
testFun(seqs,actions).show
works, however it is currently not yet using the Dataframe actionsin but the array allready created from it called actionsARR.
Of course I want it to take the actionsin Dataframe and convert it to the Array within the UDF , so I tried :
def testFun(seqsin:DataFrame, actionsin:DataFrame ): DataFrame ={
val acts = actionsin.rdd.map(y=>(y.getInt(0)+96).toChar.toString).collect()
seqsin.map(x=>(x.getString(0), countActions(x.getString(1),acts))).toDF("sequence_u_key","action_counter")
}
But when I call the function with my dataframes as input I get :
testFun(seqs,actions).show 17/07/19 07:31:32 ERROR Executor: Exception
in task 0.0 in stage 38.0 (TID 64) java.lang.ClassCastException
17/07/19 07:31:32 WARN TaskSetManager: Lost task 0.0 in stage 38.0
(TID 64, localhost, executor driver): java.lang.ClassCastException
17/07/19 07:31:32 ERROR TaskSetManager: Task 0 in stage 38.0 failed 1
times; aborting job 17/07/19 07:31:32 WARN TaskSetManager: Lost task
1.0 in stage 38.0 (TID 65, localhost, executor driver): TaskKilled (killed intentionally) org.apache.spark.SparkException: Job aborted
due to stage failure: Task 0 in stage 38.0 failed 1 times, most recent
failure: Lost task 0.0 in stage 38.0 (TID 64, localhost, executor
driver): java.lang.ClassCastException
Driver stacktrace: at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
at scala.Option.foreach(Option.scala:257) at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:1931) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:1944) at
org.apache.spark.SparkContext.runJob(SparkContext.scala:1958) at
org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:935) at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) at
org.apache.spark.rdd.RDD.collect(RDD.scala:934) at
testFun(:44) ... 52 elided Caused by:
java.lang.ClassCastException
Maybee I am doing something fundamentatly wrong by trying to pass two Dataframes to a function , coverting one to an array and then using them in the function ? or maybee I am missing something simple ?
Oh, I can succesfully create a function that takes a single Dataframe and converts and returns this as an array.
Any help would be appreciated
Best Regards
James