How to execute spark SQL using withColumn for streaming dataframe? - scala

Question

There is a scenario in which SCHOOL_GROUP column from Streaming data needs to be updated based on one mapping table (static dataframe).
Matching logic needs to be applied on AREA and SCHOOL_GROUP of streaming DF (teachersInfoDf) with SPLIT_CRITERIA and SCHOOL_CODE column to fetch SCHOOL from static DF(mappingDf).
teachersInfoDf (Streaming Data):
FNAME
LNAME
DOB
GENDER
SALARY
SCHOOL_GROUP
AREA
Williams
Kylie
1996
M
2000
ABCD
CNTRL-1
Maria
Brown
1992
F
2000
ABCD
CNTRL-5
John
Snow
1997
M
5000
XYZA
MM-RLBH1
Merry
Ely
1993
F
1000
PQRS
J-20
Michael
Rose
1998
M
1000
XYZA
DAY-20
Andrew
Simen
1990
M
1000
STUV
LVL-20
John
Dear
1997
M
5000
PQRS
P-RLBH1
mappingDf (Mapping Table data-Static):
SCHOOL_CODE
SPLIT_CRITERIA
SCHOOL
ABCD
(AREA LIKE 'CNTRL-%')
GROUP-1
XYZA
(AREA IN ('KK-DSK','DAY-20','MM-RLBH1','KM-RED1','NN-RLBH2'))
MULTI
PQRS
(AREA LIKE 'P-%' OR AREA LIKE 'V-%' OR AREA LIKE 'J-%')
WEST
STUV
(AREA NOT IN ('SS_ENDO2','SS_GRTGED','SS_GRTMMU','PC_ENDO1','PC_ENDO2','GRTENDO','GRTENDO1')
CORE
Required Dataframe:
FNAME
LNAME
DOB
GENDER
SALARY
SCHOOL_GROUP
AREA
Williams
Kylie
2006
M
2000
GROUP-1
CNTRL-1
Maria
Brown
2002
F
2000
GROUP-1
CNTRL-5
John
Snow
2007
M
5000
MULTI
MM-RLBH1
Merry
Ely
2003
F
1000
WEST
J-20
Michael
Rose
2002
M
1000
MULTI
DAY-20
Andrew
Simen
2008
M
1000
CORE
LVL-20
John
Dear
2007
M
5000
WEST
P-RLBH1
Using Spark SQL how can I achieve that?
(I know in streaming we can't show data like this. Streaming DF examples are for reference only.)
(For now, I created static DF to apply the logic.)
I am using below way but getting error:
def deriveSchoolOnArea: UserDefinedFunction = udf((area: String, SPLIT_CRITERIA: String, SCHOOL: String) => {
if (area == null || SPLIT_CRITERIA == null || SCHOOL == null) {
return null
}
val splitCriteria = SPLIT_CRITERIA.replace("AREA", area)
val query = """select """" + SCHOOL + """" AS SCHOOL from dual where """ + splitCriteria
print(query)
val dualDf = spark.sparkContext.parallelize(Seq("dual")).toDF()
dualDf.createOrReplaceGlobalTempView("dual")
print("View Created")
val finalHosDf = spark.sql(query)
print("Query Executed")
var finalSchool = ""
if (finalHosDf.isEmpty){
return null
}
else{
finalSchool = finalHosDf.select(col("SCHOOL")).first.getString(0)
}
print(finalSchool)
finalSchool
})
val dfJoin = teachersInfoDf.join(mappingDf,mappingDf("SCHOOL_CODE") === teachersInfoDf("SCHOOL_GROUP"), "left")
val dfJoin2 = dfJoin.withColumn("SCHOOL_GROUP", coalesce(deriveSchoolOnArea(col("area"), col("SPLIT_CRITERIA"), col("SCHOOL")), col("SCHOOL_GROUP")))
dfJoin2.show(false)
But Getting below error:
dfJoin2.show(false)
org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:416)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:406)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2459)