Inline map function instead of loops - scala

Question

I have a table in dataframe with three columns. city_name, driver_name, vehicles out of which vehicle is a list.
I also have some other details such as driver hours, driver contact etc for each driver in mysql. Tables in database are in this format: city_name.driver_name.
scala> val tables = """
[
{"vehicles" : ["subaru","mazda"], "city_name" : "seattle", "driver_name" : "x"},
{"city_name" : "seattle", "driver_name" : "y"},
{"city_name" : "newyork", "driver_name" : "x"},
{"city_name" : "dallas", "driver_name" : "y"}
]
""" | | | | | | |
tables: String =
"
[
{"vehicles" : ["subaru","mazda"], "city_name" : "seattle", "driver_name" : "x"},
{"city_name" : "seattle", "driver_name" : "y"},
{"city_name" : "newyork", "driver_name" : "x"},
{"city_name" : "dallas", "driver_name" : "y"}
]
"
scala> val metadataRDD = sc.parallelize(tables.split('\n').map(_.trim.filter(_ >= ' ')).mkString :: Nil)
metadataRDD: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[5] at parallelize at <console>:30
scala> val metadataDF = spark.read.json(metadataRDD)
metadataDF: org.apache.spark.sql.DataFrame = [city_name: string, driver_name: string ... 1 more field]
scala> metadataDF.show
+---------+-----------+---------------+
|city_name|driver_name| vehicles|
+---------+-----------+---------------+
| seattle| x|[subaru, mazda]|
| seattle| y| null|
| newyork| x| null|
| dallas| y| null|
+---------+-----------+---------------+
For each of these driver I need to apply a function and write to a parquet. What I am trying to do is use a inline function as below but I can't get it to work:
metadataDF.map((e) => {
val path = "s3://test/"
val df = sparkJdbcReader.option("dbtable",
e.city_name + "." + e.driver_name).load()
val dir = path + e.driver_name + e.city_name
if (e.vehicles)
do something
else:
df.write.mode("overwrite").format("parquet").save(dir)
})
Basically the questions is around how to use that inline function.

Answer 1

A call to map() function always transforms the given input collection of type A to another collection of type B, using the supplied function. In your map function call you are saving the Dataframe to your Storage layer[presumably HDFS]. The save() method defined on the DataFrameWriter Class has a return type of Unit [think of it as void in Java]. Hence, your function will not work as it is transforming your DataFrame to essentially two types: the data-type returned from the if block and Unit returned from the else block.
You can refactor your code and break it up in two blocks or so:
import org.apache.spark.sql.functions.{concat,concat_ws,lit,col}
import org.apache.spark.sql.DataFrame
import org.apache.spark.rdd.RDD
val metadataRDD: RDD[String] = sc.parallelize(tables.split('\n').map(_.trim.filter(_ >= ' ')).mkString :: Nil)
val metadataDF: DataFrame = spark.read.json(metadataRDD)
val df_new_col: DataFrame = metadataDF
.withColumn("city_driver",concat_ws(".",col("city_name"),col("driver_name")))
.withColumn("dir",concat(lit("s3://test/"),col("city_name"),col("driver_name")))
you now have two columns where you have table names and their paths next to them. You can collect them and use them to read your dataframes to be stored in Parquet format.