WithColumn and nulls, Scala Spark - scala

I trying to create a new column and compare it with another one, if are equal I have to put "Yes" else "No" as you can see here:
+----+-------+-----------+----------+
|Game| statB | statPrev | Change |
+----+-------+-----------+----------+
| CA| 2 | 2 | No |
| BL| 5 | 2 | Yes |
| CD| null | null | No |
| NT| 4 | 5 | Yes |
| FT| 6 | null | Yes |
+----+-------+-----------+----------+
What I am trying is:
var df1 = df.withColumn("Change",
when($"statB" =!= $"statPrev"
|| $"statPrev".isNull && $"statB".isNotNull
|| $"statPrev".isNotNull && $"statB".isNotNull, "Yes").otherwise("No"))
But for example when StatB and statPrev are both nulls, I get an "Yes"... What am I doing wrong?


To compare equality with nulls, you can use eqNullSafe for a simpler syntax:
val df2 = df.withColumn(
"Change",
when($"statB".eqNullSafe($"statPrev"), "Yes").otherwise("No")
)
df2.show
+----+-----+--------+------+
|Game|statB|statPrev|Change|
+----+-----+--------+------+
| CA| 2| 2| Yes|
| BL| 5| 2| No|
| CD| null| null| Yes|
| NT| 4| 5| No|
| FT| 6| null| No|
+----+-----+--------+------+


According to your question, if are equal I have to put "Yes" else "No"
It should be
var df1 = df.withColumn("Change1",
when($"statB" === $"statPrev" || ($"statB".isNull && $"statPrev".isNull),
"Yes").otherwise("No"))
df1.show(false)
Or you could use null safe equal operator as
df.withColumn("Change1",
when(($"statB" <=> $"statPrev" ), "Yes").otherwise("No"))
.show(false)
Result:
+----+-----+--------+------+
|Game|statB|statPrev|Change|
+----+-----+--------+------+
|CA |2 |2 |Yes |
|BL |5 |2 |No |
|CD |null |null |Yes |
|NT |4 |5 |No |
|FT |6 |null |No |
+----+-----+--------+------+


If stateB and statePrev equals :
df.withColumn("Change", when($"stateB" === $"statePrev", lit("YES")).otherwise("NO")).show(false);
output
+----+------+---------+---+
|Game|stateB|statePrev|Change|
+----+------+---------+---+
|CA |2 |2 |YES|
|BL |5 |2 |NO |
|CD |null |null |YES|
|NT |4 |5 |NO |
|FT |6 |null |NO |
+----+------+---------+---+
if you want to tell No if the null values on stateB and statePrev -
df.withColumn("Change",
when(($"stateB" === $"statePrev") && ($"stateB".notEqual( "null")
&& $"statePrev".notEqual( "null")),
lit("YES")).otherwise("NO")).show(false)
output
+----+------+---------+------+
|Game|stateB|statePrev|Change|
+----+------+---------+------+
|CA |2 |2 |YES |
|BL |5 |2 |NO |
|CD |null |null |NO |
|NT |4 |5 |NO |
|FT |6 |null |NO |
+----+------+---------+------+

Related

group by per day pyspark

I have a PySpark DataFrame :
From id To id Price Date
a b 20 30/05/2019
b c 5 30/05/2019
c a 20 30/05/2019
a d 10 02/06/2019
d c 5 02/06/2019
id Name
a Claudia
b Manuella
c remy
d Paul
The output that i want is :
Date Name current balance
30/05/2019 Claudia 0
30/05/2019 Manuella 15
30/05/2019 Remy -15
30/05/2019 Paul 0
02/06/2019 Claudia -10
02/06/2019 Manuella 15
02/06/2019 Remy -10
02/06/2019 Paul 5
I want to get the current balance in each day for all users.
my idea is to make a groupby per user and calculate the sum of the TO column minus the From column. But how to do it per day? especially it's cumulative and not per day?
Thank You

I took a bit of an effort to get the requirements right. Here's my version of the solution.
from pyspark.sql import Row
from pyspark.sql.types import *
from pyspark import SparkContext, SQLContext
import pyspark.sql.functions as F
from pyspark.sql import Window
sc = SparkContext('local')
sqlContext = SQLContext(sc)
data1 = [
("a","b",20,"30/05/2019"),
("b","c",5 ,"30/05/2019"),
("c","a",20,"30/05/2019"),
("a","d",10,"02/06/2019"),
("d","c",5 ,"02/06/2019"),
]
df1Columns = ["From_Id", "To_Id", "Price", "Date"]
df1 = sqlContext.createDataFrame(data=data1, schema = df1Columns)
df1 = df1.withColumn("Date",F.to_date(F.to_timestamp("Date", 'dd/MM/yyyy')).alias('Date'))
print("Actual initial data")
df1.show(truncate=False)
data2 = [
("a","Claudia"),
("b","Manuella"),
("c","Remy"),
("d","Paul"),
]
df2Columns = ["id","Name"]
df2 = sqlContext.createDataFrame(data=data2, schema = df2Columns)
print("Actual initial data")
df2.show(truncate=False)
alldays_df = df1.select("Date").distinct().repartition(20)
allusers_df = df2.select("id").distinct().repartition(10)
crossjoin_df = alldays_df.crossJoin(allusers_df)
crossjoin_df = crossjoin_df.withColumn("initial", F.lit(0))
crossjoin_df = crossjoin_df.withColumnRenamed("id", "common_id").cache()
crossjoin_df.show(n=40, truncate=False)
from_sum_df = df1.groupby("Date", "From_Id").agg(F.sum("Price").alias("from_sum"))
from_sum_df = from_sum_df.withColumnRenamed("From_Id", "common_id")
from_sum_df.show(truncate=False)
from_sum_df = crossjoin_df.alias('cross').join(
from_sum_df.alias('from'), ['Date', 'common_id'], how='outer'
).select('Date', 'common_id',
F.coalesce('from.from_sum', 'cross.initial').alias('from_amount') ).cache()
from_sum_df.show(truncate=False)
to_sum_df = df1.groupby("Date", "To_Id").agg(F.sum("Price").alias("to_sum"))
to_sum_df = to_sum_df.withColumnRenamed("To_Id", "common_id")
to_sum_df.show(truncate=False)
to_sum_df = crossjoin_df.alias('cross').join(
to_sum_df.alias('to'), ['Date', 'common_id'], how='outer'
).select('Date', 'common_id',
F.coalesce('to.to_sum', 'cross.initial').alias('to_amount') ).cache()
to_sum_df.show(truncate=False)
joined_df = to_sum_df.join(from_sum_df, ["Date", "common_id"], how='inner')
joined_df.show(truncate=False)
balance_df = joined_df.withColumn("balance", F.col("to_amount") - F.col("from_amount"))
balance_df.show(truncate=False)
final_df = balance_df.join(df2, F.col("id") == F.col("common_id"))
final_df.show(truncate=False)
final_cum_sum = final_df.withColumn('cumsum_balance', F.sum('balance').over(Window.partitionBy('common_id').orderBy('Date').rowsBetween(-sys.maxsize, 0)))
final_cum_sum.show()
Following are all the outputs for your progressive understanding. I am not explaining the steps. You can figure them out.
Actual initial data
+-------+-----+-----+----------+
|From_Id|To_Id|Price|Date |
+-------+-----+-----+----------+
|a |b |20 |2019-05-30|
|b |c |5 |2019-05-30|
|c |a |20 |2019-05-30|
|a |d |10 |2019-06-02|
|d |c |5 |2019-06-02|
+-------+-----+-----+----------+
Actual initial data
+---+--------+
|id |Name |
+---+--------+
|a |Claudia |
|b |Manuella|
|c |Remy |
|d |Paul |
+---+--------+
+----------+---------+-------+
|Date |common_id|initial|
+----------+---------+-------+
|2019-05-30|a |0 |
|2019-05-30|d |0 |
|2019-05-30|b |0 |
|2019-05-30|c |0 |
|2019-06-02|a |0 |
|2019-06-02|d |0 |
|2019-06-02|b |0 |
|2019-06-02|c |0 |
+----------+---------+-------+
+----------+---------+--------+
|Date |common_id|from_sum|
+----------+---------+--------+
|2019-06-02|a |10 |
|2019-05-30|a |20 |
|2019-06-02|d |5 |
|2019-05-30|c |20 |
|2019-05-30|b |5 |
+----------+---------+--------+
+----------+---------+-----------+
|Date |common_id|from_amount|
+----------+---------+-----------+
|2019-06-02|a |10 |
|2019-06-02|c |0 |
|2019-05-30|a |20 |
|2019-05-30|d |0 |
|2019-06-02|b |0 |
|2019-06-02|d |5 |
|2019-05-30|c |20 |
|2019-05-30|b |5 |
+----------+---------+-----------+
+----------+---------+------+
|Date |common_id|to_sum|
+----------+---------+------+
|2019-06-02|c |5 |
|2019-05-30|a |20 |
|2019-06-02|d |10 |
|2019-05-30|c |5 |
|2019-05-30|b |20 |
+----------+---------+------+
+----------+---------+---------+
|Date |common_id|to_amount|
+----------+---------+---------+
|2019-06-02|a |0 |
|2019-06-02|c |5 |
|2019-05-30|a |20 |
|2019-05-30|d |0 |
|2019-06-02|b |0 |
|2019-06-02|d |10 |
|2019-05-30|c |5 |
|2019-05-30|b |20 |
+----------+---------+---------+
+----------+---------+---------+-----------+
|Date |common_id|to_amount|from_amount|
+----------+---------+---------+-----------+
|2019-06-02|a |0 |10 |
|2019-06-02|c |5 |0 |
|2019-05-30|a |20 |20 |
|2019-05-30|d |0 |0 |
|2019-06-02|b |0 |0 |
|2019-06-02|d |10 |5 |
|2019-05-30|c |5 |20 |
|2019-05-30|b |20 |5 |
+----------+---------+---------+-----------+
+----------+---------+---------+-----------+-------+
|Date |common_id|to_amount|from_amount|balance|
+----------+---------+---------+-----------+-------+
|2019-06-02|a |0 |10 |-10 |
|2019-06-02|c |5 |0 |5 |
|2019-05-30|a |20 |20 |0 |
|2019-05-30|d |0 |0 |0 |
|2019-06-02|b |0 |0 |0 |
|2019-06-02|d |10 |5 |5 |
|2019-05-30|c |5 |20 |-15 |
|2019-05-30|b |20 |5 |15 |
+----------+---------+---------+-----------+-------+
+----------+---------+---------+-----------+-------+---+--------+
|Date |common_id|to_amount|from_amount|balance|id |Name |
+----------+---------+---------+-----------+-------+---+--------+
|2019-05-30|a |20 |20 |0 |a |Claudia |
|2019-06-02|a |0 |10 |-10 |a |Claudia |
|2019-05-30|b |20 |5 |15 |b |Manuella|
|2019-06-02|b |0 |0 |0 |b |Manuella|
|2019-05-30|c |5 |20 |-15 |c |Remy |
|2019-06-02|c |5 |0 |5 |c |Remy |
|2019-06-02|d |10 |5 |5 |d |Paul |
|2019-05-30|d |0 |0 |0 |d |Paul |
+----------+---------+---------+-----------+-------+---+--------+
+----------+---------+---------+-----------+-------+---+--------+--------------+
| Date|common_id|to_amount|from_amount|balance| id| Name|cumsum_balance|
+----------+---------+---------+-----------+-------+---+--------+--------------+
|2019-05-30| d| 0| 0| 0| d| Paul| 0|
|2019-06-02| d| 10| 5| 5| d| Paul| 5|
|2019-05-30| c| 5| 20| -15| c| Remy| -15|
|2019-06-02| c| 5| 0| 5| c| Remy| -10|
|2019-05-30| b| 20| 5| 15| b|Manuella| 15|
|2019-06-02| b| 0| 0| 0| b|Manuella| 15|
|2019-05-30| a| 20| 20| 0| a| Claudia| 0|
|2019-06-02| a| 0| 10| -10| a| Claudia| -10|
+----------+---------+---------+-----------+-------+---+--------+--------------+

Spark Scala, merging two columnar dataframes duplicating the second dataframe each time

I want to merge 2 columns or 2 dataframes like
df1
+--+
|id|
+--+
|1 |
|2 |
|3 |
+--+
df2 --> this one can be a list as well
+--+
|m |
+--+
|A |
|B |
|C |
+--+
I want to have as resulting table
+--+--+
|id|m |
+--+--+
|1 |A |
|1 |B |
|1 |C |
|2 |A |
|2 |B |
|2 |C |
|3 |A |
|3 |B |
|3 |C |
+--+--+

def crossJoin(right: org.apache.spark.sql.Dataset[_]): org.apache.spark.sql.DataFrame
Using crossJoin function you can get same result. Please check code below.
scala> dfa.show
+---+
| id|
+---+
| 1|
| 2|
| 3|
+---+
scala> dfb.show
+---+
| m|
+---+
| A|
| B|
| C|
+---+
scala> dfa.crossJoin(dfb).orderBy($"id".asc).show(false)
+---+---+
|id |m |
+---+---+
|1 |B |
|1 |A |
|1 |C |
|2 |A |
|2 |B |
|2 |C |
|3 |C |
|3 |B |
|3 |A |
+---+---+

how to rename the Columns Produced by count() function in Scala

I have the below df:
+------+-------+--------+
|student| vars|observed|
+------+-------+--------+
| 1| ABC | 19|
| 1| ABC | 1|
| 2| CDB | 1|
| 1| ABC | 8|
| 3| XYZ | 3|
| 1| ABC | 389|
| 2| CDB | 946|
| 1| ABC | 342|
|+------+-------+--------+
I wanted to add a new frequency column groupBy two columns "student", "vars" in SCALA.
val frequency = df.groupBy($"student", $"vars").count()
This code generates a "count" column with the frequencies BUT losing observed column from the df.
I would like to create a new df as follows without losing "observed" column
+------+-------+--------+------------+
|student| vars|observed|total_count |
+------+-------+--------+------------+
| 1| ABC | 9|22
| 1| ABC | 1|22
| 2| CDB | 1|7
| 1| ABC | 2|22
| 3| XYZ | 3|3
| 1| ABC | 8|22
| 2| CDB | 6|7
| 1| ABC | 2|22
|+------+-------+-------+--------------+

You cannot do this directly but there are couple of ways,
You can join original df with count df. check here
You collect the observed column while doing aggregation and explode it again
With explode:
val frequency = df.groupBy("student", "vars").agg(collect_list("observed").as("observed_list"),count("*").as("total_count")).select($"student", $"vars",explode($"observed_list").alias("observed"), $"total_count")
scala> frequency.show(false)
+-------+----+--------+-----------+
|student|vars|observed|total_count|
+-------+----+--------+-----------+
|3 |XYZ |3 |1 |
|2 |CDB |1 |2 |
|2 |CDB |946 |2 |
|1 |ABC |389 |5 |
|1 |ABC |342 |5 |
|1 |ABC |19 |5 |
|1 |ABC |1 |5 |
|1 |ABC |8 |5 |
+-------+----+--------+-----------+

We can use Window functions as well
val windowSpec = Window.partitionBy("student","vars")
val frequency = df.withColumn("total_count", count(col("student")) over windowSpec)
.show
+-------+----+--------+-----------+
|student|vars|observed|total_count|
+-------+----+--------+-----------+
|3 |XYZ |3 |1 |
|2 |CDB |1 |2 |
|2 |CDB |946 |2 |
|1 |ABC |389 |5 |
|1 |ABC |342 |5 |
|1 |ABC |19 |5 |
|1 |ABC |1 |5 |
|1 |ABC |8 |5 |
+-------+----+--------+-----------+

Incremental addition with condition in dataframe [duplicate]

This question already has answers here:
How to calculate cumulative sum using sqlContext
(4 answers)
Closed 4 years ago.
I have a DataFrame like this :
finalSondDF.show()
+---------------+------------+----------------+
|webService_Name|responseTime|numberOfSameTime|
+---------------+------------+----------------+
| webservice1| 80| 1|
| webservice1| 87| 2|
| webservice1| 283| 1|
| webservice2| 77| 2|
| webservice2| 80| 1|
| webservice2| 81| 1|
| webservice3| 63| 3|
| webservice3| 145| 1|
| webservice4| 167| 1|
| webservice4| 367| 2|
| webservice4| 500| 1|
+---------------+------------+----------------+
and I want to get a result like this :
+---------------+------------+----------------+------+
|webService_Name|responseTime|numberOfSameTime|Result|
+---------------+------------+----------------+------+
| webservice1| 80| 1| 1|
| webservice1| 87| 2| 3| ==> 2+1
| webservice1| 283| 1| 4| ==> 1+2+1
| webservice2| 77| 2| 2|
| webservice2| 80| 1| 3| ==> 2+1
| webservice2| 81| 1| 4| ==> 2+1+1
| webservice3| 63| 3| 3|
| webservice3| 145| 1| 4| ==> 3+1
| webservice4| 167| 1| 1|
| webservice4| 367| 2| 3| ==> 1+2
| webservice4| 500| 1| 4| ==> 1+2+1
+---------------+------------+----------------+------+
here the result is the sum of numberOfSameTime inferior of the current responseTime
I can't find a logic to do that. Can any one help me !!

If your data is in increasing order with responseTime column for each group of webService_Name column then you can benefit from cumulative sum using Window function as below
import org.apache.spark.sql.expressions._
def windowSpec = Window.partitionBy("webService_Name").orderBy("responseTime")
import org.apache.spark.sql.functions._
df.withColumn("Result", sum("numberOfSameTime").over(windowSpec)).show(false)
and you should have
+---------------+------------+----------------+------+
|webService_Name|responseTime|numberOfSameTime|Result|
+---------------+------------+----------------+------+
|webservice1 |80 |1 |1 |
|webservice1 |87 |2 |3 |
|webservice1 |283 |1 |4 |
|webservice2 |80 |1 |3 |
|webservice2 |81 |1 |4 |
|webservice2 |77 |2 |2 |
|webservice3 |145 |1 |4 |
|webservice3 |63 |3 |3 |
|webservice4 |167 |1 |1 |
|webservice4 |367 |2 |3 |
|webservice4 |500 |1 |4 |
+---------------+------------+----------------+------+
Note that the responseTime as to be number type and in increasing order for each webService_Name for the above case to work

You can use Window function available in spark and calculate the cumulative sum as below.
//dummy data
val d1 = spark.sparkContext.parallelize(Seq(
("webservice1", 80, 1),
("webservice1", 87, 2),
("webservice1", 283, 1),
("webservice2", 77, 2),
("webservice2", 80, 1),
("webservice2", 81, 1),
("webservice3", 63, 3),
("webservice3", 145, 1),
("webservice4", 167, 1),
("webservice4", 367, 2),
("webservice4", 500, 1)
)).toDF("webService_Name","responseTime","numberOfSameTime")
//window functionn
val window = Window.partitionBy("webService_Name").orderBy($"webService_Name")
.rowsBetween(Long.MinValue, 0)
// create new column for Result
d1.withColumn("Result", sum("numberOfSameTime").over(window)).show(false)
Output:
+---------------+------------+----------------+------+
|webService_Name|responseTime|numberOfSameTime|Result|
+---------------+------------+----------------+------+
|webservice4 |167 |1 |1 |
|webservice4 |367 |2 |3 |
|webservice4 |500 |1 |4 |
|webservice2 |77 |2 |2 |
|webservice2 |80 |1 |3 |
|webservice2 |81 |1 |4 |
|webservice3 |63 |3 |3 |
|webservice3 |145 |1 |4 |
|webservice1 |80 |1 |1 |
|webservice1 |87 |2 |3 |
|webservice1 |283 |1 |4 |
+---------------+------------+----------------+------+
Hope this helps!

apply an aggregate result to all ungrouped rows of a dataframe in spark

assume there is a dataframe as follows:
machine_id | value
1| 5
1| 3
2| 6
2| 9
2| 14
I want to produce a final dataframe like this
machine_id | value | diff
1| 5| 1
1| 3| -1
2| 6| -4
2| 10| 0
2| 14| 4
the values in "diff" column is computed as groupBy($"machine_id").avg($"value") - value.
note that the avg for machine_id==1 is (5+3)/2 = 4 and for machine_id ==2 is (6+10+14)/3 = 10
What is the best way to produce such a final dataframe in Apache Spark?

You can use Window function to get the desired output
Given the dataframe as
+----------+-----+
|machine_id|value|
+----------+-----+
|1 |5 |
|1 |3 |
|2 |6 |
|2 |10 |
|2 |14 |
+----------+-----+
You can use following code
df.withColumn("diff", avg("value").over(Window.partitionBy("machine_id")))
.withColumn("diff", 'value - 'diff)
to get the final result as
+----------+-----+----+
|machine_id|value|diff|
+----------+-----+----+
|1 |5 |1.0 |
|1 |3 |-1.0|
|2 |6 |-4.0|
|2 |10 |0.0 |
|2 |14 |4.0 |
+----------+-----+----+