PySpark: Muliple Where condition vs AND (&&) [closed] - pyspark

Question

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
.where((col('Country')==Country) & (col('Year')>startYear))
I can do the where conditions both ways. I think the one below add readability. Is there any other difference and which is the best?
.where(col('Country')==Country)
.where(col('Year')>startYear)

Answer 1

if the question is readability, i would suggest something like this :
.where(F.expr("Country <=> 'Country' and Year > 'startYear'")
Here <=> is used for equality null safe, there is a something in spark where nulls values are ignored in condition.
I had worked with a sample , both are giving same results. So there would be no other differences.
data.show()
+---+---------+----+
| id| Country|year|
+---+---------+----+
| 1| india|2018|
| 2| usa|2018|
| 3| france|2019|
| 4| china|2019|
| 5| india|2020|
| 6|australia|2021|
| 7| india|2016|
| 8| usa|2019|
+---+---------+----+
data.where((col('Country')=='india') & (col('Year')>2017)).show()
#consider country:'india',startyear:2017
+---+-------+----+
| id|Country|year|
+---+-------+----+
| 1| india|2018|
| 5| india|2020|
+---+-------+----+
data.where(col('Country')=='india')\
.where(col('Year')>2017).show()
+---+-------+----+
| id|Country|year|
+---+-------+----+
| 1| india|2018|
| 5| india|2020|
+---+-------+----+

Answer 2

The method explain is useful in order to understand how a query is performed. It shows the execution plan with all the steps involved and it can be used in this case to compare the two filtering strategies.
Give the following example:
from pyspark.sql.functions import col
df = spark.createDataFrame([("Spain", 2020),
("Italy", 2020),
("Andorra", 2021),
("Spain", 2021),
("Spain", 2022)], ("Country", "Year"))
df.show()
Country = "Spain"
startYear = 2020
The extended output of the AND strategy is:
df.where((col('Country') == Country) & (col('Year') > startYear)).explain(True)
== Parsed Logical Plan ==
'Filter (('Country = Spain) AND ('Year > 2020))
+- LogicalRDD [Country#80, Year#81L], false
== Analyzed Logical Plan ==
Country: string, Year: bigint
Filter ((Country#80 = Spain) AND (Year#81L > cast(2020 as bigint)))
+- LogicalRDD [Country#80, Year#81L], false
== Optimized Logical Plan ==
Filter (((isnotnull(Country#80) AND isnotnull(Year#81L)) AND (Country#80 = Spain)) AND (Year#81L > 2020))
+- LogicalRDD [Country#80, Year#81L], false
== Physical Plan ==
*(1) Filter (((isnotnull(Country#80) AND isnotnull(Year#81L)) AND (Country#80 = Spain)) AND (Year#81L > 2020))
+- *(1) Scan ExistingRDD[Country#80,Year#81L]
while the plan of the multiple where strategy is:
df.where(col('Country') == Country).where(col('Year') > startYear).explain(True)
== Parsed Logical Plan ==
'Filter ('Year > 2020)
+- Filter (Country#80 = Spain)
+- LogicalRDD [Country#80, Year#81L], false
== Analyzed Logical Plan ==
Country: string, Year: bigint
Filter (Year#81L > cast(2020 as bigint))
+- Filter (Country#80 = Spain)
+- LogicalRDD [Country#80, Year#81L], false
== Optimized Logical Plan ==
Filter (((isnotnull(Country#80) AND isnotnull(Year#81L)) AND (Country#80 = Spain)) AND (Year#81L > 2020))
+- LogicalRDD [Country#80, Year#81L], false
== Physical Plan ==
*(1) Filter (((isnotnull(Country#80) AND isnotnull(Year#81L)) AND (Country#80 = Spain)) AND (Year#81L > 2020))
+- *(1) Scan ExistingRDD[Country#80,Year#81L]
The query engine came up with the same physical plan regardless the filtering strategy, thus the queries are equivalent. I agree with you the second one is better for readability