pyspark: withcolumn Analysis error after column is converted to lowercase - pyspark

Question

I have a dataframe that looks like below
+------------+------+
| food|pounds|
+------------+------+
| bacon| 4.0|
|STRAWBERRIES| 3.5|
| Bacon| 7.0|
|STRAWBERRIES| 3.0|
| BACON| 6.0|
|strawberries| 9.0|
|Strawberries| 1.0|
| pecans| 3.0|
+------------+------+
And the expected output is
+------------+------+---------+
| food|pounds|food_type|
+------------+------+---------+
| bacon| 4.0| meat|
|STRAWBERRIES| 3.5| fruit|
| Bacon| 7.0| meat|
|STRAWBERRIES| 3.0| fruit|
| BACON| 6.0| meat|
|strawberries| 9.0| fruit|
|Strawberries| 1.0| fruit|
| pecans| 3.0| other|
+------------+------+---------+
So I essentially defined a new_column based on my logic and applied that on .withcolumn
new_column = when((col('food') == 'bacon') | (col('food') == 'BACON') | (col('food') == 'Bacon'), 'meat'
).when((col('food') == 'STRAWBERRIES') | (col('food') == 'strawberries') | (col('food') == 'Strawberries'), 'fruit'
).otherwise('other')
And then
df.withColumn("food_type", new_column).show()
Which works fine. But I wanted to update the new_column statement with less code, so rewrite is as below
new_column = when(lower(col('food') == 'bacon') , 'meat'
).when(lower(col('food') == 'strawberries'), 'fruit'
).otherwise('other')
Now when I do df.withColumn("food_type", new_column).show()
I get error
AnalysisException: "cannot resolve 'CASE WHEN lower(CAST((`food` = 'bacon') AS STRING)) THEN 'meat' WHEN lower(CAST((`food` = 'strawberries') AS STRING)) THEN 'fruit' ELSE 'other' END' due to data type mismatch: WHEN expressions in CaseWhen should all be boolean type, but the 1th when expression's type is lower(cast((food#165 = bacon) as string));;\n'Project [food#165, pounds#166, CASE WHEN lower(cast((food#165 = bacon) as string)) THEN meat WHEN lower(cast((food#165 = strawberries) as string)) THEN fruit ELSE other END AS food_type#197]\n+- Relation[food#165,pounds#166] csv\n"
What am I missing?

Answer 1

Your parentheses are mismatched.
new_column = when(lower(col('food')) == 'bacon' , 'meat').when(lower(col('food')) == 'strawberries', 'fruit').otherwise('other')

Answer 2

I would like to share another approach which is more similar to SQL queries and can be more appropriate with more complex and nested conditions as well.
from pyspark.sql.functions import *
cond = """case when lower(food) in ('bacon') then 'meat'
else case when lower(food) in ('strawberries') then 'fruit'
else 'other'
end
end"""
newdf = df.withColumn("food_type", expr(cond))
Hope it helps.
Regards,
Neeraj

Answer 3

simplified:
new_column=when(lower(col("food"))=="bacon",'meat').when(lower(col("food"))=='strawberries','fruit').otherwise('other')
df.withColumn("food_type", new_column).show()