assign unique id to dataframe elements - scala

Question

I have a dataframe recording each person's name and address, denoted as
case class Person(name:String, addr:String)
dataframe looks like this,
+----+-----+
|name| addr|
+----+-----+
| u1|addr1|
| u1|addr2|
| u2|addr1|
+----+-----+
but now I need to assign a Long type unique id to each element in this dataframe, which could be denoted as
case class PersonX(name:String, name_id:Long, addr:String, addr_id:Long)
and dataframe looks like this,
+----+-------+-----+------+
|name|name_id| addr|addr_id|
+----+-------+-----+------+
| u1| 1|addr1| 2|
| u1| 1|addr2| 3|
| u2| 4|addr1| 2|
+----+-------+-----+------+
NOTE that, the elements in both columns (name and addr) share the same id space, which means, name_id should not have duplicates, and addr_id should not either, and furthermore name_ids & addr_ids should not overlap with each other.
How to achieve this?

Answer 1

The easiest way to assign id is to use dense_rank function from spark sql.
To make sure ids don't overlap between names and addresses you can make a trick:
compute ids of names and addresses separately
update ids of addresses by adding maximum id of name
In this way ids of addresses will be after ids of names
val input = spark.sparkContext.parallelize(List(
Person("u1", "addr1"),
Person("u1", "addr2"),
Person("u2", "addr1")
)).toDF("name", "addr")
input.createOrReplaceTempView("people")
val people = spark.sql(
"""select name,
| dense_rank() over(partition by 1 order by name) as name_id,
| addr,
| dense_rank() over(partition by 1 order by addr) as addr_id
| from people
""".stripMargin)
people.show()
//+----+-------+-----+-------+
//|name|name_id| addr|addr_id|
//+----+-------+-----+-------+
//| u1| 1|addr1| 1|
//| u2| 2|addr1| 1|
//| u1| 1|addr2| 2|
//+----+-------+-----+-------+
val name = people.col("name")
val nameId = people.col("name_id")
val addr = people.col("addr")
val addrId = people.col("addr_id")
val maxNameId = people.select(max(nameId)).first().getInt(0)
val shiftedAddrId = (addrId + maxNameId).as("addr_id")
people.select(name, addr, nameId, shiftedAddrId).as[PersonX].show()
//+----+-----+-------+-------+
//|name| addr|name_id|addr_id|
//+----+-----+-------+-------+
//| u1|addr1| 1| 3|
//| u2|addr1| 2| 3|
//| u1|addr2| 1| 4|
//+----+-----+-------+-------+