spark dataframe with column when condition - scala

Question

My requirement is as below
Joined two data frames as below:
var c = a.join(b,keys,"fullouter")
c.printSchema() as below:
|-- add: string (nullable = true)
|-- sub: string (nullable = true)
|-- delete: string (nullable = true)
|-- mul: long (nullable = true)
|-- ADD: string (nullable = true)
|-- SUB: string (nullable = true)
|-- DELETE: string (nullable = true)
|-- MUL: long (nullable = true)
It's good until here.
Now i am doing a withcolumn when condition as below
val d = c.withColumn("column", when(c("a.add") === c("b.ADD"),
"Neardata"))
error as below:
Exception in thread "main" org.apache.spark.sql.AnalysisException:
Cannot resolve column name "a.add"
I tried as below also
val d = c.withColumn("column", when(col("a.add") === col("b.ADD"), "Neardata"))
Again error.
Please suggest.

Answer 1

You have to define alias with datframe.as("a") and dataframe1.as("b")
Example :
import spark.sqlContext.implicits._
val data = List(("James","","Smith","36636","M",60000),
("Michael","Rose","","40288","M",70000),
("Robert","","Williams","42114","",400000),
("Maria","Anne","Jones","39192","F",500000),
("Jen","Mary","Brown","","F",0))
val cols = Seq("first_name","middle_name","last_name","dob","gender","salary")
val df = spark.createDataFrame(data).toDF(cols:_*).as("a")
val df2 = df.withColumn("a.new_gender", when(col("a.gender") === "M","Male")
.when(col("a.gender") === "F","Female")
.otherwise("Unknown")).show
Output :
+----------+-----------+---------+-----+------+------+------------+
|first_name|middle_name|last_name| dob|gender|salary|a.new_gender|
+----------+-----------+---------+-----+------+------+------------+
| James| | Smith|36636| M| 60000| Male|
| Michael| Rose| |40288| M| 70000| Male|
| Robert| | Williams|42114| |400000| Unknown|
| Maria| Anne| Jones|39192| F|500000| Female|
| Jen| Mary| Brown| | F| 0| Female|
+----------+-----------+---------+-----+------+------+------------+
I think with out alias you are trying to access like this... that might be the cause.
val df2 = df.withColumn("df.new_gender", when(col("df.gender") === "M","Male")
.when(col("df.gender") === "F","Female")
.otherwise("Unknown")).show