Check if join stream was successful using Apache Spark - Scala - scala

Question

I am new to Apache Spark, using Scala. I am able to join a table to the stream using following command:
Updated_DF = Inbound_DF.join(colToAdd, colToAdd("key") <=> Inbound_DF("key"), "left")
.withColumnRenamed("Data_DF","site").drop("Id","key")
Now I want to check if colToAdd("key") and Inbound_DF("key") matched and join was successful or not. For example, colToAdd:
Id key Data_DF
S31 S3 {"name":"nick","region":"IN"}
S21 S2 {"name":"john","region":"CA"}
S11 S1 {"name":"ashley","region":"CA"}
S51 S5 {"name":"bella","region":"UK"}
S41 S4 {"name":"kumar","region":"In"}
S6 S6 {"name":"ben","region":"US"}
P11 P1 {"name":"MKD","region":"UAE"}
P21 P2 {"name":"ahmad","region":"UAE"}
Message from incoming stream look like:
cusId key item price
1897 S2 book 54
After join, the updated message should look like:
cusId key item price site
1897 S2 book 54 {"name":"john","region":"CA"}
But if I get a stream message with key = S9, the join will not happen and then I want to log a message:
------- join failed, key not found ---------
As far as I know, this can be achieved using the filter method but I am not sure how to implement that. Please help me how this can be done or is there any better way to do the same.

Answer 1

There are multiple ways of doing it. I am just providing you with an idea of how this could be done and you can adjust according to your use case.
First the way you are doing the left join is incorrect, you need to swap the dataframes. The Stream dataframe should be left dataframe.
//Source data
val df = Seq(("S31","S3","""{"name":"nick","region":"IN"}"""),("S21","S2","""{"name":"john","region":"CA"}"""),("S11","S1","""{"name":"john","region":"CA"}""")).toDF("Id","Key","Data_DF")
val df1 = Seq((1897,"S2","book",54),(1920,"S9","movie",200)).toDF("custId","Key","item","price")
//initial join and the count of the records
val df2 = df1.join(df,Seq("Key"),"left").drop("Id").withColumnRenamed("Data_DF","site")
val initialjoincount = df2.count()
//filter and count of the records
val filteredDF = df2.filter($"site".isNotNull)
val filtereddfcount = filteredDF.count()
//compare both the counts and print message/log
if(filtereddfcount == initialjoincount)
{
println("Join Happened")
}
else
{
println("Value not found in stream.")
}