How to find coefficient of variation (CV) in pyspark? - pyspark

Question

I have 2 pyspark dataframe i want to find coefficient of variation of that two dataframe.
dataframe1 :
hours total
00 75969.0
01 75302.0
02 74636.0
03 73969.0
04 73302.0
05 72635.0
dataframe2 :-
hours total1
00 71535
01 71182
02 77628
03 75984
04 75276
05 67259
And i want output like these :-
dataframe3 :-
hours total total1 CV
00 75969.0 71535 3.006020
01 75302.0 71182 2.812594
02 74636.0 77628 1.965008
03 73969.0 75984 1.343754
04 73302.0 75276 1.328595
05 72635.0 67259 3.842910
I have done these by converting pyspark-dataframe to pandas dataframe but i want to calculate CV without going to pandas.
I have done like these
pd1=dataframe1.toPandas()
pd2=dataframe2.toPandas()
a4=[]
list1=[]
count=len(pd1)
print(count)
import numpy as np
for i in range(count):
del a4[:]
p9=(pd1.total[i])
p10=(pd2.total1[i])
a4.append(p10)
a4.append(p9)
standard_d1=np.std(a4,ddof=0)
mean1=np.mean(a4)
cv=(standard_d1/mean1)*100
list1.append(cv)
pd1['cv']=list1

Answer 1

There are several ways to solve this:
Use window
Use udf
Use window + udf
First, let's build the DataFrame:
from pyspark.sql.window import Window
from pyspark.sql import functions as F
from pyspark.sql.types import *
from pyspark.sql.functions import mean, pandas_udf, PandasUDFType
a1 = [(0, 75969.0), (1, 75302.0), (2, 74636.0), (3, 73969.0), (4, 73302.0), (5, 72635.0)]
a2 = [(0, 71535.0), (1, 71182.0), (2, 77628.0), (3, 75984.0), (4, 75276.0), (5, 67259.0)]
df1 = spark.createDataFrame(a1, ['hours', 'total'])
df2 = spark.createDataFrame(a2, ['hours', 'total'])
df = df1.union(df2)
df.show()
+-----+-------+
|hours| total|
+-----+-------+
| 0|75969.0|
| 1|75302.0|
| 2|74636.0|
| 3|73969.0|
| 4|73302.0|
| 5|72635.0|
| 0|71535.0|
| 1|71182.0|
| 2|77628.0|
| 3|75984.0|
| 4|75276.0|
| 5|67259.0|
+-----+-------+
Using udf only
#pandas_udf(FloatType(), PandasUDFType.GROUPED_AGG)
def _udf(v):
return 100.0*np.std(v, ddof=0)/np.mean(v)
df = df.groupBy('hours').agg(_udf(df['total']).alias('CV')).orderBy('hours')
df.show()
Use window
w = Window.partitionBy('hours')
df = df.withColumn('std', F.stddev_pop('total').over(w))
df = df.withColumn('mean', F.mean('total').over(w))
df = df.withColumn('CV', 100.0*df['std']/df['mean']).dropDuplicates(['hours']).drop(*['total', 'std', 'mean']).orderBy('hours')
df.show()
Use window + udf
w = Window.partitionBy('hours')
df = df.withColumn('CV',_udf('total').over(w)).dropDuplicates(['hours']).orderBy('hours')
df.show()
All the above methods give you the results:
+-----+---------+
|hours| CV|
+-----+---------+
| 0| 3.00602|
| 1| 2.812594|
| 2|1.9650081|
| 3|1.3437544|
| 4| 1.328595|
| 5|3.8429096|
+-----+---------+

Answer 2

Here since you have only two elements, we have the property that standard_dev = |total - total1| / 2. Since mean = (total + total1) / 2, we have CV = 100 * |total - total1| / (total + total1).
Therefore :
from pyspark.sql import functions as F
df = dataframe1.join(dataframe2, 'hours', 'inner')
df_final = df.withColumn('CV', F.lit(100) * F.abs(df['total'] - df['total1']) / (df['total'] + df['total1']))