How to find the difference of two dataframes - scala

Question

I am working on some Unit Testing Spark code which should be able to generate the difference between two dataframes(raw bucket and curated bucket). Both dataframes(buckets) are the same and we want to execute this code to capture the possible changes after we copy files from raw to curated. . I am aware that I can use except function as follow:
val difference =CuratedDataFrame.union(RawDataFrame).except(CuratedDataFrame.intersect(RawDataFrame))
+-----------+-------+-------------+---------+---------------+
|record |pid |feetype |freq |default |
+-----------+-------+-------------+---------+---------------+
| 1| 45| FAC| Y| T|
| 1| 45| FAC| Y| TTY|
| 1| 47| FAC| R| M|
| 1| 99| FAC| R| M|
+-----------+-------+-------------+---------+---------------+
The except function is returning the entire row but my desired output is as follow :
+-----------+-------+-------------+---------+---------------+
|record |pid |feetype |freq |default |
+-----------+-------+-------------+---------+---------------+
| null|[47,99]| null| null| null |
| null| null| null| null| [T, TTY]|
+----------+-----------+-------+-------------+---------+-----
It means if there is a change in column then it should appear if there is no change then it should be hidden or Null.
For doing this I am using the following approach :
val mapDiffs=(name: String) => when($"l.$name" === $"r.$name", null )
.otherwise(array($"l.$name", $"r.$name")).as(name)
val result = difference.as("l")
.join(RawDataFrame.as("r"), $"l.primaryKey" === $"r.primaryKey","inner")
.select($"l.primaryKey" :: cols.map(mapDiffs): _*)
The above approach requires primary key to be able to join both dataframes and compare them row by row. None of the dataframes have primary key so I had to combine some of the columns to specify a primary key :
+-----------+-------+-------------+---------+---------------+----------+
|record |pid |feetype |freq |default |primaryKey|
+-----------+-------+-------------+---------+---------------+----------+
| 1| 40| FAC| A| N| FAC40A|
| 1| 45| FAC| Y| T| FAC45Y|
| 1| 47| FAC| R| M| FAC47R|
+-----------+-------+-------------+---------+---------------+----------+
The problem is that if any changes happen in the target bucket, the primary key will be consequently changed so comparing both dataframes would be impossible.