Here's how I calculate histogram on one column:
val df = spark.read.format("csv").option("header", "true").load("/project/test.csv")
df.map(row => row.getString(2).toDouble).rdd.histogram(10)
I want to calculate histograms on all columns. I can simply repeat the second line (see code above) and call histogram separately on each column. But my concern is that Spark will load data from disk each time I call histogram(), which means that if there are 10 columns, data is loaded 10 times. Is there a more efficient way to do this? How can I calculate histograms on all 10 columns in one shot?
Edit
Here's one way to combine multiple histogram() calls into one expression:
val histograms = {
val a = df.map(row => row.getString(0).toDouble).rdd.histogram(10)
val b = df.map(row => row.getString(1).toDouble).rdd.histogram(15)
(a, b)
}
Does this guarantee that the histograms will be computed with only one pass over the data? Is combining multiple histogram calls into one expression the trick? Or is that even necessary? Doesn't Spark delay evaluation until the result is used in any case, even when separate statements are used?
Related
I have a DataFrame with millions of rows. I have columns A, B and C. I have to cluster C over A and B.
For now what I'm doing is a for over every A and B set. Obviously it's slow, but I can't really parallelize it. I tried using Window and pandas_udf, but it doens't work with scalar functions.
Edit: some code to help understand my problem
# suppose we have a variable my_df which is a dataframe with millions of rows
# its schema is type, code, quantity
# I have to cluster quantity by types
# hundreds of types in real case
types = [1, 2, 3]
for index, t in enumerate(types):
filtered_my_df = my_df.filter(my_df.type == t)
# then I apply kmeans on filtered_my_df...
When I have hundres or even thousands of types, this code obviously is not scalable. But I can't figure out how to do this in scalable way.
In case of Kmeans, it randomly selects k objects from the whole objects which represent initial cluster centers. Then, each remaining object is assigned to the cluster to which it is the most similar, based on the distance between the object and the cluster center. Then new mean for each cluster is calculated. This process iterates until the criterion function converges. This is why, it becomes very time costly.
You can use Parallel K-Means Based on MapReduce.
The distance computations between one object with the centers is irrelevant to the distance computations between other objects with the corresponding centers. distance computations between different objects with centers can be parallel executed.
MLlib implementation of k-means corresponds to the algorithm called K-Means\5 which is a parallel version of the original one. The method header is defined as follows: Let's say parseddata is you spark RDD.
KMeans.train(k, maxIterations, initializationMode, runs)
from pyspark.mllib.clustering import KMeans
clusters = KMeans.train(parsedData, 2, maxIterations=10, runs=10, initializationMode='random')
# Evaluation
from math import sqrt
def error(point):
center = clusters.centers[clusters.predict(point)]
return sqrt(sum([x**2 for x in (point - center)]))
WSSSE = (parsedData.map(lambda point:error(point)).reduce(lambda x, y: x+y))
print('Within Set Sum of Squared Error = ' + str(WSSSE))
For example,
I have a dataframe with 12000 rows, and I define a threshold of 3000(externally supplied to my code via config), so I would like to split this dataframe into 4 dataframes with 3000 rows each.
If the dataframe has 12500 rows, I will split it into 5 dataframes, 4 with 3000 rows and last one with 500.
**The significance of dataframe here is that if the rowcount of the dataframe is lesser than this, I will not tinker with it.
In order to get a close behavior (but not exactly what you specified), you can use the function randomSplit: randomSplit(weights: Array[Double])
The weights are the proportions in each output dataframe. Henve, you cannot specify exactly the number.
In your case (12500 rows), df.randomSplit(Array(1,1,1,1)) to separate it into 4 approximately equal parts. Or:
val n = (df.length / 4).toInt
df.randomSplit((1 to n).map(x => 1).toArray)
PS: I think you will cuts of a spark dataframe with "exact number of rows" with Spark
I want to do kmeans labels for numClusters = 6 so that I can group by the labels later.
How do I select the columns to do kmeans on?
val clusterThis = scaledDF.select($"id",$"setting1",$"setting2",$"setting3")
// dataset description lists six operation modes
val operatingModes = 6
// Cluster the data into two classes using KMeans
val numClusters = operatingModes
val numIterations = 20
import sqlContext.implicits._
val clusters = KMeans.train(clusterThis.rdd, numClusters, numIterations)
clusters.predict(clusterThis)
//... join back on id
As you can see in KMeans's Example the object uses just one column as features. In that example and by coincidence it has the same name. However, that name depends on you, but the important thing is that this column must be a Vector (dense or sparse).
Thus, you would need to combine your features (different columns) into one, for this task you can use a VectorAssembler.
By the way, K-means doesn't work with categorical features. You can read this post K-means clustering for mixed numeric and categorical data to notice the reasons.
I know several questions has been asked on similar topics but I couldn't apply any of the answers to my problem, also I am wondering about best practices.
I have loaded a dateset for ML to a SQL database. I want to apply mllib's clustering function according to it. I have loaded the SQL database to DataFrame using sqlContext, dropped the irrelevant columns. then happened the problematic part, I create a vector by parsing each row of the DataFrame.
The Vector is then transformed to RDD using the toJavaRDD function.
Here is the code (works):
val usersDF = sqlContext.read.format("jdbc").option("url","jdbc:mysql://localhost/database").
option("driver","com.mysql.jdbc.Driver").option("dbtable","table").
option("user","woot").option("password","woot-password").load()
val cleanDF = usersDF.drop("id").drop("username")
cleanDF.show()
val parsedData = cleanDF.map(s => Vectors.dense(s.toString().replaceAll("[\\[\\]]", "").trim.split(',').map(_.toDouble))).cache()
val splits = parsedData.randomSplit(Array(0.6,0.4), seed = 11L)
val train_set = splits(0).cache()
val gmm = new GaussianMixture().setK(2).run(train_set)
My main question regards to what I read on spark documentation about: Local vector, in my understanding the DataFrame mapping will be performed on the workers and later will be sent to the Driver when creating the Vector(Is that the meaning of local vector) only to later be sent to the workers again? isn't there a better way to achieve this?
Another things is that it seems a little odd to load SQL to DataFrame only to turn it into string and parse it again. Are there any other best practices suggestions?
From the link you suggested
A local vector has integer-typed and 0-based indices and double-typed
values, stored on a single machine. MLlib supports two types of local
vectors: dense and sparse.
A distributed matrix has long-typed row and column indices and
double-typed values, stored distributively in one or more RDDs.
The local vector are behaving like any object you would use for your RDD (String, Integer, Array), they are created and stored on a single machine, the worker node, and only if you collect them they will be sent to the driver node.
If you consider a vector x of size 2n storing it distributively you would separate it in two halfs of length n, x1 and x2, (x = x1::x2). To perform the dot product with another vector y, the workers will perform r1=x1*y1 (on machine 1) and r2=x2*y2 (on machine 2) and then you will need to group the partial results giving r=r1+r2. Your vector x is distributed, the vectors x1 and x2 are again local vectors. If you have x as a local vector then in a single step you can perform on a worker node r=x*y.
For your second question, I do not see why you would store the vectors in SQL format. Having a CSV file like this would be sufficient:
label feature1 feature2 ...
1, 0.5, 1.2 ...
0, 0.2, 0.5 ...
I am into a process of doing a POC on Retail Transaction Data using few Machine learning Algorithms and coming up with a prediction model for Out of stock analysis. My questions might sound stupid but I would really appreciate if you or anyone else can answer me.
So far I have been able to get a data set ==> Convert the features into a (labelpoint , Feature Vectors) ==> Train a ML model ==> Run the model on Test DataSet and ==> Get the predictions.
Problem 1:
Since I have no experience on any of the JAVA/Python/Scala languages, I am building my features in the database and saving that data as a CSV file for my machine learning Algorithm.
How do we create features using Scala from raw data.
Problem 2:
The Source Data set consists of many features for a set of (Store, Product , date) and their recorded OOS events (Target)
StoreID(Text column), ProductID(Text Column), TranDate , (Label/Target), Feature1, Feature2........................FeatureN
Since the Features can only contain numeric values so, I just create features out of the numeric columns and not the text ones (Which is the natural key for me). When I run the model on a validation set I get a (Prediction, Label) array back.
Now how do I link this resultant set back to the original data set and see which specific (Store, Product, Date) might have a possible Out Of Stock event ?
I hope the problem statement was clear enough.
MJ
Spark's Linear Regression Example
Here's a snippet from the Spark Docs Linear Regression example that is fairly instructive and easy to follow.
It solves both your "Problem 1" and "Problem 2"
It doesn't need a JOIN and doesn't even rely on RDD order.
// Load and parse the data
val data = sc.textFile("data/mllib/ridge-data/lpsa.data")
Here data is a RDD of text lines
val parsedData = data.map { line =>
val parts = line.split(',')
LabeledPoint(parts(0).toDouble, Vectors.dense(parts(1).split(' ').map(_.toDouble)))
}.cache()
Problem 1: Parsing the Features
This is data dependent. Here we see that lines are being split on , into fields. It appears this data was a CSV of entirely numeric data.
The first field is treated as the label of a labelled point (dependent variable), and the rest of the fields are converted from text to double (floating point) and stuck in a vector. This vector holds the features or independent variables.
In your own projects, the part of this you need to remember is the goal of parsing into an RDD of LabeledPoints where the 1st parameter of LabeledPoint, the label, is the true dependent numeric value and the features, or 2nd parameter, is a Vector of numbers.
Getting the data into this condition requires knowing how to code. Python may be easiest for data parsing. You can always use other tools to create a purely numeric CSV, with the dependent variable in the first column, and the numeric features in the other columns, and no header line -- and then duplicate the example parsing function.
// Building the model
val numIterations = 100
val model = LinearRegressionWithSGD.train(parsedData, numIterations)
At this point we have a trained model object. The model object has a predict method that operates on feature vectors and returns estimates of the dependent variable.
Encoding Text features
The ML routines typically want numeric feature vectors, but you can often translate free text or categorical features (color, size, brand name) into numeric vectors in some space. There are a variety of ways to do this, such as Bag-Of-Words for text, or One Hot Encoding for categorical data where you code a 1.0 or 0.0 for membership in each possible category (watch out for multicollinearity though). These methodologies can create large feature vectors, which is why there are iterative methods available in Spark for training models. Spark also has a SparseVector() class, where you can easily create vectors with all but certain feature dimensions set to 0.0
Problem 2: Comparing model Predictions to the True values
Next they test this model with the training data, but the calls
would be the same with external test data provided that the test data is a RDD of LabeledPoint( dependent value, Vector(features)). The input could be changed by changing the variable parsedData to some other RDD.
// Evaluate model on training examples and compute training error
val valuesAndPreds = parsedData.map { point =>
val prediction = model.predict(point.features)
(point.label, prediction)
}
Notice that this returns tuples of the true dependent variable previously stored in point.label, and the model's prediction from the point.features for each row or LabeledPoint.
Now we are ready to do Mean Squared Error, since the valuesAndPreds RDD contains tuples (v,p) of true value v and the prediction p both of type Double.
The MSE is a single number, first the tuples are mapped to an rdd of squared distances ||v-p||**2 individually, and then averaged, yielding a single number.
val MSE = valuesAndPreds.map{case(v, p) => math.pow((v - p), 2)}.mean()
println("training Mean Squared Error = " + MSE)
Spark's Logistic Example
This is similar, but here you can see data is already parsed and split into training and test sets.
// Split data into training (60%) and test (40%).
val splits = data.randomSplit(Array(0.6, 0.4), seed = 11L)
val training = splits(0).cache()
val test = splits(1)
Here the model is trained against the training set.
// Run training algorithm to build the model
val model = new LogisticRegressionWithLBFGS()
.setNumClasses(10)
.run(training)
And tested (compared) against the test set. Notice that even though this is a different model (Logistic instead of Linear) there is still a model.predict method that takes a point's features vector as a parameter and returns the prediction for that point.
Once again the prediction is paired with the true value, from the label, in a tuple for comparison in a performance metric.
// Compute raw scores on the test set.
val predictionAndLabels = test.map { case LabeledPoint(label, features) =>
val prediction = model.predict(features)
(prediction, label)
}
// Get evaluation metrics.
val metrics = new MulticlassMetrics(predictionAndLabels)
val precision = metrics.precision
println("Precision = " + precision)
What about JOIN? So RDD.join comes in if you have two RDDs of (key, value) pairs, and need an RDD corresponding to the intersection of keys with both values. But we didn't need that here.