Import custom Scala UDAF into PySpark - scala

Question

With the grace of StackOverflow experts I have managed to tinker one of the provided examples and create a Scala UDAF which provides me with the necessary functionality I am looking for. The structure of the UDAF/Function etc. is as below :-
case class InputRow(ddate: String, ccount: String, iitem: String)
case class Buffer(var max_ddate: String, var ddue_dt: Map[String,String])
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import org.apache.spark.sql.expressions.Aggregator
import org.apache.spark.sql.Encoder
import java.time.LocalDate
import java.time.format.DateTimeFormatter
object RecursiveAggregatorZ extends Aggregator[InputRow, Buffer, Buffer] {
override def zero: Buffer = Buffer(null, null)
override def reduce(buffer: Buffer, currentRow: InputRow): Buffer = {
<LOGIC HERE>
buffer
}
override def merge(b1: Buffer, b2: Buffer): Buffer = {
throw new NotImplementedError("should be used only over ordered window")
}
override def finish(reduction: Buffer): Buffer = reduction
override def bufferEncoder: Encoder[Buffer] = ExpressionEncoder[Buffer]
override def outputEncoder: Encoder[Buffer] = ExpressionEncoder[Buffer]
}
I can run the actual code via Zeppelin and then execute the below to register it for use within Spark :-
import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions.{col, udaf}
val recursiveAggregatorZ = udaf(RecursiveAggregatorZ)
spark.udf.register("recursiveAggregatorZ",recursiveAggregatorZ)
However, I am looking for a very to incorporate this into PySpark so that it could be used not only within Spark SQL alone. I rummaged through most of the Google provided hits wherein we have to first package the scala code into a jar and whatnot but most of them use classes as examples and not like the snip i have provided.
Would really appreciate it if anyone could guide me on:-
(1) How exactly to build a jar out of this scala function
(2) How exactly to push it into PySpark and have it registered within PySpark.
Just for sake of clarity, within Zeppelin I am able to run queries such as :-
select recursiveAggregatorZ(column1, column2, column3) over (partition by partition1, partition2, order by rn) as output from phase1
and get the output I am looking for.