how join two dataframe in pyspark with a special condition? - pyspark

Question

I have 2 dataframes I wanna join them with respect of 3 column. The first column have an exact match the second column has an exact match if the value of df1 is not NA. but third column is not an exact match it is inclusion:
join rows of df1 with df2 if col1 in df1 and col1 in df2 have the same values and elements of col3 in df1 is included in elements of col3 in df2 and if col1 in df1 is not NA and have the same value in col2 in df2 (if it is NA but other 2 conditions are valid then match them ).
Example
df1:
col1 col2 col3
us NA amazon
ca Vancouver Facebook
IN Ottawa IBM
df2:
col1 col2 col3
ca Vancouver /n Facebook us
IN Boston IBM
us new york amazon IN
output:
ca Vancouver Facebook
us new york amazon

Answer 1

from pyspark.sql import functions as F
df1 = spark.createDataFrame(
[
('us',None,'amazon')
,('ca','Vancouver','Facebook')
,('IN','Ottawa','IBM')
],
['col1','col2','col3']
)
df2 = spark.createDataFrame(
[
('ca','Vancouver','/n Facebook us')
,('IN','Boston','IBM')
,('us','new york','amazon IN')
],
['col1','col2','col3']
)
df1.show()
df2.show()
# +----+---------+--------+
# |col1| col2| col3|
# +----+---------+--------+
# | us| null| amazon|
# | ca|Vancouver|Facebook|
# | IN| Ottawa| IBM|
# +----+---------+--------+
# +----+---------+--------------+
# |col1| col2| col3|
# +----+---------+--------------+
# | ca|Vancouver|/n Facebook us|
# | IN| Boston| IBM|
# | us| new york| amazon IN|
# +----+---------+--------------+
df1\
.join(df2, 'col1')\
.withColumn('flag', F.when(df1.col2.isNull() & df2.col3.contains(df1.col3) , 1).
when(df1.col2 == df2.col2, 1).otherwise(0))\
.filter(F.col('flag') == 1)\
.select(df1.col1, df2.col2, df1.col3)\
.show()
# +----+---------+--------+
# |col1| col2| col3|
# +----+---------+--------+
# | ca|Vancouver|Facebook|
# | us| new york| amazon|
# +----+---------+--------+