Add multiple columns with map logic without using UDF - scala

Question

I want to parse the address column from the given table structure using addressParser function to get number, street, city and country.
Sample Input:
addressId
address
ADD001
"384, East Avenue Street, New York, USA
ADD002
"123, Maccolm Street, Copenhagen, Denmark"
The sample code is attached for reference:
object ParseAddress extends App {
val spark = SparkSession.builder().master("local[*]").appName("ParseAddress ").getOrCreate()
import spark.implicits._
case class AddressRawData(addressId: String, address: String)
case class AddressData(
addressId: String,
address: String,
number: Option[Int],
road: Option[String],
city: Option[String],
country: Option[String]
)
def addressParser(unparsedAddress: Seq[AddressData]): Seq[AddressData] = {
unparsedAddress.map(address => {
val split = address.address.split(", ")
address.copy(
number = Some(split(0).toInt),
road = Some(split(1)),
city = Some(split(2)),
country = Some(split(3))
)
}
)
}
val addressDS: Dataset[AddressRawData] = addressDF.as[AddressRawData]
}
Expected output:
addressId
address
number
road
city
country
ADD001
"384, East Avenue Street, New York, USA
384
East Avenue Street
New York
USA
ADD002
"123, Maccolm Street, Copenhagen, Denmark"
123
Maccolm Street
Copenhagen
Denmark
I am not sure how should I convert addressDS as an input to function to parse the column data. Some form of help to solve this problem is very much appreciated.

Answer 1

I think one important thing is that it's better to design your function to take a single input and return a single output (in your scenario), and if you have a collection or a dataset of rows, you can map each row to this function, that's why all these polymorphic functions (map, flatMap, fold, ...) are made, right? So you can implement a method, which receives an AddressRawData and returns an AddressData:
def singleAddressParser(unparsedAddress: AddressRawData): AddressData = {
val split = unparsedAddress.address.split(", ")
AddressData(
addressId = unparsedAddress.addressId,
address = unparsedAddress.address,
number = Some(split(0).toInt),
road = Some(split(1)),
city = Some(split(2)),
country = Some(split(3))
)
}
And then map each raw data to this function:
import org.apache.spark.sql.Dataset
val addressDS: Dataset[AddressData] =
addressDF.as[AddressRawData].map(singleAddressParser)
And this is the output, as expected:
scala> addressDS.show(false)
+---------+-----------------------------------------+------+------------------+----------+-------+
|addressId|address |number|road |city |country|
+---------+-----------------------------------------+------+------------------+----------+-------+
|ADD001 |384, East Avenue Street, New York, USA |384 |East Avenue Street|New York |USA |
|ADD002 |123, Malccolm Street, Copenhagen, Denmark|123 |Malccolm Street |Copenhagen|Denmark|
+---------+-----------------------------------------+------+------------------+----------+-------+