Non-sense in pySpark calculation using a lambda over a Row - pyspark

Question

I have a df in pyspark representing KM run by cars by day.
Explaination of fields:
x[3] is the day of the week in Italian, i.e:
LUN (MON)
MAR (TUE)
MER (WED)
GIO (THU)
VEN (FRI)
SAB (SAT)
DOM (SUN)
X[22] are the kilometers in night hours
X[23] are the kilometers in day time
The objective is to insert 3 new columns. In pseudo code:
km_totali (total km): x[22]+x[23]
km_festivo (week-end km): x[22]+x[23] if x[3] in [SAB, DOM] i.e. SAT or SUN; else 0
km_feriale (weekday km): x[22]+x[23] if x[3] in [LUN-VEN]. i.e. MON-FRI; else 0
In other words, km_totali should always be calculated.
km_festivo should be equal to km_totali on the weekend days, 0 otherwise
km_feriale should be equal to km_totali on the week-days, 0 otherwise.
This apparently easy task is driving me crazy.
This is my code:
new_df=df.rdd.map(lambda x: Row(
giorno_della_settimana=x[3],
km_totali=x[22]+x[23],
km_feriale=x[22]+x[23] if x[3] in ["LUN","MAR", "MER", "GIO", "VEN] else 0,
km_festivo=x[22]+x[23] if x[3] in ["SAB","DOM"] else 0
)).toDF()
As you can see in the below output, km_totali is ALWAYS calculated (and it is correct, I made my checks). It also correctly caluclates km_festivo and km_feriale during week-days (MER and MON in the below example). But it ALWAYS fails to calulate km_festivo and km_feriale during week-end days (Nones in the DOM line in the below example).
OUTPUT:
Row(giorno_della_settimana=u'DOM', km_feriale=None, km_festivo=None, km_totali=106.5),
Row(giorno_della_settimana=u'MER', km_feriale=8.2, km_festivo=0, km_totali=8.2),
Row(giorno_della_settimana=u'LUN', km_feriale=3.0, km_festivo=0, km_totali=3.0),
Notice that km_totali is correctly calculated also in the "DOM" case. In facts, th desired output for the DOM line would be:
Row(giorno_della_settimana=u'DOM', km_feriale=0, km_festivo=106.5, km_totali=106.5)
What it drives me compleately crazy is that If I extend the list of days in the km_festivo's condition list to the whole week (meaningless from a semantic poit of view), the km_festivo field gets correctly calculated also in the DOM line:
new_df=df.map(lambda x: Row(
giorno_della_settimana=x[3],
km_totali=x[22]+x[23],
km_feriale=x[22]+x[23] if x[3] in ["LUN","MAR", "MER","GIO", "VEN"] else 0,
extended-> km_festivo=x[22]+x[23] if x[3] in ["LUN","MAR", "MER","GIO", "VEN","SAB","DOM"] else 0
)).toDF()
OUTPUT for the DOM line:
Row(giorno_della_settimana=u'DOM', km_feriale=None, km_festivo=106.5, km_totali=106.5),
As you can see, the km_festivo is now calculated for the DOM line (i.e a week-end day) ONLY because I have included Week-days in the condition list. This is a non-sense!!!!
I feel this is is not a matter of coding but I cannot figure out what this could possibly due to!!
HELP

Answer 1

SOLVED!
In the else clause, I have to return 0.0 instead of 0.
Otherwise I have a column with mixed types. Somtimes floats and somtimes integers
This is the right code:
new_df=df.rdd.map(lambda x: Row(
giorno_della_settimana=x[3],
km_totali=x[22]+x[23],
km_feriale=x[22]+x[23] if x[3] in ["LUN","MAR", "MER", "GIO", "VEN] else 0.0,
km_festivo=x[22]+x[23] if x[3] in ["SAB","DOM"] else 0.0
)).toDF()

Answer 2

Could you provide sample data?
I don't see anything wrong with your code apart from the fact that you're doing redundant computations (x[22] +x[23] at least twice) instead of computing it afterwards.
Could you try:
import pyspark.sql.functions as psf
new_df = df.map(lambda x: Row(
giorno_della_settimana=x[3],
km_totali=x[22]+x[23]
)).toDF()
new_df2 = new_df.select(
"*",
psf.when(new_df.giorno_della_settimana.isin(["LUN","MAR", "MER","GIO", "VEN"]), new_df.km_totali).otherwise(0).alias("km_feriale"),
psf.when(new_df.giorno_della_settimana.isin(["SAB","DOM"]), new_df.km_totali).otherwise(0).alias("km_festivo"),
)