How to apply best fit distributions in pyspark? - pyspark

Question

I currently working on a migration from python to pyspark,and I have one step where I find the best fit distribution using a modified function of Fitting empirical distribution to theoretical ones with Scipy (Python)? where I apply best_fit_distribution to each group od Id's, and save the output in a dictionary,there is some way to do that in pyspark? I was doing research about pyspark statistics and I don't find any library that could help me.
For the needs of the development I need to do this part in pyspark, so keep in original python can't be an option.
import scipy.stats as st
import numpy as np
import warnings
def best_fit_distribution(data, bins=200, ax=None)
y, x = np.histogram(data, bins=bins, density=True)
x = (x + np.roll(x, -1))[:-1] / 2.0
# Distributions to check
distribution_list = [st.alpha,st.chi2, st.pearson3] #This is an example
# Best holders
best_distribution = st.norm
best_params = (0.0, 1.0)
best_sse = np.inf
for distribution in distribution_list:
try:
with warnings.catch_warnings():
warnings.filterwarnings("ignore")
params = distribution.fit(data)
arg = params[:-2]
loc = params[-2]
scale = params[-1]
pdf = distribution.pdf(x, loc=loc, scale=scale, *arg)
sse = np.sum(np.power(y - pdf, 2.0))
# if axis pass in add to plot
try:
if ax:
pd.Series(pdf, x).plot(ax=ax)
#end
except Exception:
pass
# identify if this distribution is better
if best_sse > sse > 0:
best_distribution = distribution
best_params = params
best_sse = sse
except Exception:
pass
return (best_distribution.name, best_params)
This is an example and description of my df:
Id
Values
8
59.25
8
25.1
8
39.0333
8
138.3737
8
79.5002
8
52.9
8
0.1674
9
33.8667
9
0.75
9
78.05
9
76.9167
9
14.6667
9
80.3166
9
32.7333
9
0.8333
9
76.95
9
84.4
9
23.1667
9
23.1
9
76.6667
summary
Id
Values
count
34052
1983107
min
8
0.0
max
2558
59646.1712