How to Stop Spark Execution if df is empty - pyspark

Question

Here's my use case:
df_1:
user_name
Mark
Jane
Mary
df_2:
participant_name
Mark
Jane
Mary
Bill
Expectation:
Compare these 2 df, if they are different, throw exception and stop the execution.
Example:
In the above example, since user_name does not contain Bill, so the session should be stopped and message should say, Bill is not in user_name df
My approach:
Use Left Anti Join to see if I can get an empty df as result, if not, throw exception.
My questions are:
Any neat way to do this comparison in pyspark?
How to throw exception and stop the execution if condition not met?

Answer 1

Performing left anti join can be a correct approach here. The only demerit is that we need to specify the join condition with each column if we have multiple columns in both the dataframes. We have a similar function in pyspark exceptAll() which does exactly what you are looking for. df1.exceptAll(df2) returns a dataframe having all the rows which are present in df1 but not in df2. Similarly, we can use it the other way and if we get empty dataframes for both, only then the dataframes are similar.
def is_equal(df1: DataFrame, df2: DataFrame):
if len(df1.exceptAll(df2).take(1)) > 0:
return False
if len(df2.exceptAll(df1).take(1)) > 0:
return False
return True
df_1 = spark.createDataFrame([("Mark",), ("Jane",), ("Mary",)], schema="user_name string")
df_2 = spark.createDataFrame([("Mark",), ("Jane",), ("Mary",), ("Bill",)], schema="participant_name string")
if not is_equal(df_1, df_2):
raise Exception("DataFrames are different")
The above code will throw an Exception as df_2 has "Bill" while df_1 does not.
len(df1.exceptAll(df2).take(1)) > 0 is used to find if the returned dataframe is non-empty. If yes, there is a difference in the dataframes and we return False.
For your 2nd point, we can raise an exception using raise.

Answer 2

You can use subtract function, and count the result dataset, if count is zero means 2 dataframes are equals.
You also can use inner join, and if the joined dataframe has the same count as the originals then two of them are equals.
The exception can be thrown as typical Pythonic way:
if df1.subtract(df2).count() > 0:
raise Exception("some items in df1 does not exist in df2")
if df2.subtract(df1).count() > 0:
raise Exception("some items in df2 does not exist in df1")