SCALA - How could a function be called in a struct? - scala

Question

I build a dataframe, which is going to have a String column that is actuall a structure, turned to a JSON string.
val df = df_helper.select(
lit("some data").as("id"),
to_json(
struct(
col("id"),
col("type"),
col("path")
)
)).as("content")
I also built a function, that takes the identifier id:String as a parameter and spits out a string list.
def buildHierarchy(id:String) : List[String] = {
val check_df = hierarchy_df.select(
$"parent_id",
$"id"
).where($"id" === id)
val pathArray = List(id)
val parentString = check_df.select($"parent_id").first.getString(0)
if (parentString == null) {
return pathArray
}
else {
val pathList = buildHierarchy(parentString)
val finalList: List[String] = pathList ++ pathArray
return finalList
}
}
I want to call this function and replace the path column with the result of the function. Is this possible, or is there a workaround?
Thank you in advance!

Answer 1

With the help of the following blog post I built up the hierarchy:
https://www.qubole.com/blog/processing-hierarchical-data-using-spark-graphx-pregel-api/
I included all the necessary information in my final dataframe, that I need to build up the details for each elements for the structure, along with the hierarchy path for each elements. Then as a new column I created a structure for each elements with the details.
val content_df = hierarchy_df.select($"path")
.withColumn("content",
struct( col("id"),
col("type"),
col("path")
))
I exploded the path to reach each identifiers in it, but preserve the position order, to join on the content for each level in the path.
val exploded_df = content_df.select(
$"*",posexplode($"path"),
$"pos".as("exploded_level"),
$"col".as("exploded_id"))
Then finally, join the content with the path, and aggregate the content into a path that contains all the content for each level.
val level_content_df = exploded_df.as("e")
.join(content_df.as("ouc"), col("e.col") === col("ouc.id"), "left")
.select($"e.*", $"ouc.content".as("content"))
val pathFull_df = level_content_df
.groupBy(col("id").as("id"))
.agg(sort_array(collect_list("content")).as("pathFull"))
And then finally I just put the whole thing together again :)
val content_with_path_df = content_df.as("ou")
.join(pathFull_df.as("opf"), col("ou.id") === col("opf.id"), "left")
.select($"ou.*", $"opf.pathFull".as("pathFull"))
Feel free to reach out if it doesn't make sense, it took a while for me to make it happen! :D