I am trying to implement cosine similarity to calculate Item-Item Similairity using Input Dataset which looks like this -
UserID, ProductID, Transactions
where UserID, ProductID are Long values and Transaction is Integer.
I am following this example in Spark -
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/mllib/CosineSimilarity.scala
In above example it expects a dense Vector as input , which gets converted to RowMatrix.
Could you please help me convert my input data set -
U1,P1,T1
U1,P3,T2
U2,P1,T4
U3,P1,T6
U3,P3,T7
to a Matrix of form -
|P1|P2|P3
u1 |T1| |T2
u2 |T4|T5|
u3 |T6| |T7
I am aware of the way that I can create a CooridnateMatrix something like this -
val mat = new CoordinateMatrix(transactions.map( entry => MatrixEntry(entry.user,entry.product, entry.txns)))
But, this one uses actual user and product id values in place of indices of Matrix, and fails as soon as the values extend beyond Integer.
I need a way that i can convert my data to an indiced matrix.
Related
I'm trying to create a product recommender with the code below. I'm using matrix factorization from spark ml. I have data that has a customer_id, product_id, and a numeric rating value that has been normalized. So all rating values are between 0 and 1. My dataset has almost 10M records with no null product_id or customer_id. Yet after I train the model and then try to make predictions for the entire dataset, I'm getting back a very small number of predictions. I'm getting similar size test_pred dataframe returned by the model whether I set the coldStartStrategy to "nan" or "drop". Does anyone see what the issue may be and can you suggest how to fix?
code:
print(data.count())
print(data[['customer_id']].distinct().count())
print(data[['product_id']].distinct().count())
output:
9943626
1715292
308792
code:
from pyspark.ml.recommendation import ALS
(training, test) = data.randomSplit([0.7, 0.3])
als = ALS(implicitPrefs=True,
maxIter=5,
regParam=0.01,
userCol="customer_id_index",
itemCol="product_id_index",
ratingCol="rating",
coldStartStrategy="nan")
model = als.fit(training)
test_pred=model.transform(data)
print(test_pred.count())
print(test_pred[['customer_id']].distinct().count())
print(test_pred[['product_id']].distinct().count())
output:
3346
522
760
looks like the issue was that I needed to save my data after I initially converted one of the id fields to a string index. seems like after I trained the model and made predictions, the pyspark code must've changed the indexes for the records. so it couldn't find the same customer product pairs to make predictions for. Once I saved the data out after initially string indexing and then read back in and applied transform to it with the trained model, I got much higher record count for the resulting dataframe.
I have a multiclass classification problem I'm looking to sort with logistic regression. I know this can also be tackled by decision trees and random forest, but wish to stick specifically with "LogisticRegressionWithLBFGS".
I have all the data tidying done. I have my data nice and tidy in a dataframe with a:
label field (String), a feature vector (vector of features/ numbers) and a third column "LabelIndex" (numbers representing the class).
When I do a train test split on the data frame and try to fit it to: LogisticRegressionWithLBFGS
val model = new LogisticRegressionWithLBFGS().setNumClasses(10).setIntercept(true).setValidateData(true).run("trainingData")
It doesn't like the "run" part.
The example I am working off, loads a data file in via:
val data = MLUtils.loadLibSVMFile(Spark.sparkContext, "data/mnist.bz2")
(i'm trying to copy the example, and slot in my own data. But its in a different format, looks different etc)
I was doing a bit of reading, and I'd come across, I need to convert my dataframe to a RDD[LabeledPoint].
I need to map it.
I'm having problems finding good info on how to do this.
How do I simply convert a Dataframe with 3 fields as described above, "Label" (String), "Features" (feature vector), "IndexedLabel" (Double)
into a RDD[LabeledPoint]?
Got it working:
Can't convert Dataframe to Labeled Point
This link showed me how to make the conversion successfully.
requirement -
In the Picture attached, consider the first 3 columns as my raw data. Some rows have quantity column as NULL value which is exactly what I want to fill up.
In an Ideal case, I would fill up any NULL value with the previous KNOWN value.
Spark Imputer seemed to be a very easily implementable library that can help me fill missing values.
But here the issue is,Spark Imputer is limited to mean or Median calculation according to all NON-BULL values present in the data frame as a result of which I don't get desired result (4th column in the Pic).
Logic -
val imputer = new Imputer()
.setInputCols(Array("quantity"))
.setOutputCols(Array("quantity_imputed"))
.setStrategy("mean")
val model = imputer.fit(new_combinedDf)
model.transform(new_combinedDf).show()
Result -
Now is it possible to limit the Mean calculation for EACH null value to be the MEAN of last n values ?
i.e
For 2020-09-26 , where we get the first null value, Is it possible to tweak Spark Imputer to calculate the Mean over last n values only instead of all non-null values in the dataframe ?
I know several questions has been asked on similar topics but I couldn't apply any of the answers to my problem, also I am wondering about best practices.
I have loaded a dateset for ML to a SQL database. I want to apply mllib's clustering function according to it. I have loaded the SQL database to DataFrame using sqlContext, dropped the irrelevant columns. then happened the problematic part, I create a vector by parsing each row of the DataFrame.
The Vector is then transformed to RDD using the toJavaRDD function.
Here is the code (works):
val usersDF = sqlContext.read.format("jdbc").option("url","jdbc:mysql://localhost/database").
option("driver","com.mysql.jdbc.Driver").option("dbtable","table").
option("user","woot").option("password","woot-password").load()
val cleanDF = usersDF.drop("id").drop("username")
cleanDF.show()
val parsedData = cleanDF.map(s => Vectors.dense(s.toString().replaceAll("[\\[\\]]", "").trim.split(',').map(_.toDouble))).cache()
val splits = parsedData.randomSplit(Array(0.6,0.4), seed = 11L)
val train_set = splits(0).cache()
val gmm = new GaussianMixture().setK(2).run(train_set)
My main question regards to what I read on spark documentation about: Local vector, in my understanding the DataFrame mapping will be performed on the workers and later will be sent to the Driver when creating the Vector(Is that the meaning of local vector) only to later be sent to the workers again? isn't there a better way to achieve this?
Another things is that it seems a little odd to load SQL to DataFrame only to turn it into string and parse it again. Are there any other best practices suggestions?
From the link you suggested
A local vector has integer-typed and 0-based indices and double-typed
values, stored on a single machine. MLlib supports two types of local
vectors: dense and sparse.
A distributed matrix has long-typed row and column indices and
double-typed values, stored distributively in one or more RDDs.
The local vector are behaving like any object you would use for your RDD (String, Integer, Array), they are created and stored on a single machine, the worker node, and only if you collect them they will be sent to the driver node.
If you consider a vector x of size 2n storing it distributively you would separate it in two halfs of length n, x1 and x2, (x = x1::x2). To perform the dot product with another vector y, the workers will perform r1=x1*y1 (on machine 1) and r2=x2*y2 (on machine 2) and then you will need to group the partial results giving r=r1+r2. Your vector x is distributed, the vectors x1 and x2 are again local vectors. If you have x as a local vector then in a single step you can perform on a worker node r=x*y.
For your second question, I do not see why you would store the vectors in SQL format. Having a CSV file like this would be sufficient:
label feature1 feature2 ...
1, 0.5, 1.2 ...
0, 0.2, 0.5 ...
I am into a process of doing a POC on Retail Transaction Data using few Machine learning Algorithms and coming up with a prediction model for Out of stock analysis. My questions might sound stupid but I would really appreciate if you or anyone else can answer me.
So far I have been able to get a data set ==> Convert the features into a (labelpoint , Feature Vectors) ==> Train a ML model ==> Run the model on Test DataSet and ==> Get the predictions.
Problem 1:
Since I have no experience on any of the JAVA/Python/Scala languages, I am building my features in the database and saving that data as a CSV file for my machine learning Algorithm.
How do we create features using Scala from raw data.
Problem 2:
The Source Data set consists of many features for a set of (Store, Product , date) and their recorded OOS events (Target)
StoreID(Text column), ProductID(Text Column), TranDate , (Label/Target), Feature1, Feature2........................FeatureN
Since the Features can only contain numeric values so, I just create features out of the numeric columns and not the text ones (Which is the natural key for me). When I run the model on a validation set I get a (Prediction, Label) array back.
Now how do I link this resultant set back to the original data set and see which specific (Store, Product, Date) might have a possible Out Of Stock event ?
I hope the problem statement was clear enough.
MJ
Spark's Linear Regression Example
Here's a snippet from the Spark Docs Linear Regression example that is fairly instructive and easy to follow.
It solves both your "Problem 1" and "Problem 2"
It doesn't need a JOIN and doesn't even rely on RDD order.
// Load and parse the data
val data = sc.textFile("data/mllib/ridge-data/lpsa.data")
Here data is a RDD of text lines
val parsedData = data.map { line =>
val parts = line.split(',')
LabeledPoint(parts(0).toDouble, Vectors.dense(parts(1).split(' ').map(_.toDouble)))
}.cache()
Problem 1: Parsing the Features
This is data dependent. Here we see that lines are being split on , into fields. It appears this data was a CSV of entirely numeric data.
The first field is treated as the label of a labelled point (dependent variable), and the rest of the fields are converted from text to double (floating point) and stuck in a vector. This vector holds the features or independent variables.
In your own projects, the part of this you need to remember is the goal of parsing into an RDD of LabeledPoints where the 1st parameter of LabeledPoint, the label, is the true dependent numeric value and the features, or 2nd parameter, is a Vector of numbers.
Getting the data into this condition requires knowing how to code. Python may be easiest for data parsing. You can always use other tools to create a purely numeric CSV, with the dependent variable in the first column, and the numeric features in the other columns, and no header line -- and then duplicate the example parsing function.
// Building the model
val numIterations = 100
val model = LinearRegressionWithSGD.train(parsedData, numIterations)
At this point we have a trained model object. The model object has a predict method that operates on feature vectors and returns estimates of the dependent variable.
Encoding Text features
The ML routines typically want numeric feature vectors, but you can often translate free text or categorical features (color, size, brand name) into numeric vectors in some space. There are a variety of ways to do this, such as Bag-Of-Words for text, or One Hot Encoding for categorical data where you code a 1.0 or 0.0 for membership in each possible category (watch out for multicollinearity though). These methodologies can create large feature vectors, which is why there are iterative methods available in Spark for training models. Spark also has a SparseVector() class, where you can easily create vectors with all but certain feature dimensions set to 0.0
Problem 2: Comparing model Predictions to the True values
Next they test this model with the training data, but the calls
would be the same with external test data provided that the test data is a RDD of LabeledPoint( dependent value, Vector(features)). The input could be changed by changing the variable parsedData to some other RDD.
// Evaluate model on training examples and compute training error
val valuesAndPreds = parsedData.map { point =>
val prediction = model.predict(point.features)
(point.label, prediction)
}
Notice that this returns tuples of the true dependent variable previously stored in point.label, and the model's prediction from the point.features for each row or LabeledPoint.
Now we are ready to do Mean Squared Error, since the valuesAndPreds RDD contains tuples (v,p) of true value v and the prediction p both of type Double.
The MSE is a single number, first the tuples are mapped to an rdd of squared distances ||v-p||**2 individually, and then averaged, yielding a single number.
val MSE = valuesAndPreds.map{case(v, p) => math.pow((v - p), 2)}.mean()
println("training Mean Squared Error = " + MSE)
Spark's Logistic Example
This is similar, but here you can see data is already parsed and split into training and test sets.
// Split data into training (60%) and test (40%).
val splits = data.randomSplit(Array(0.6, 0.4), seed = 11L)
val training = splits(0).cache()
val test = splits(1)
Here the model is trained against the training set.
// Run training algorithm to build the model
val model = new LogisticRegressionWithLBFGS()
.setNumClasses(10)
.run(training)
And tested (compared) against the test set. Notice that even though this is a different model (Logistic instead of Linear) there is still a model.predict method that takes a point's features vector as a parameter and returns the prediction for that point.
Once again the prediction is paired with the true value, from the label, in a tuple for comparison in a performance metric.
// Compute raw scores on the test set.
val predictionAndLabels = test.map { case LabeledPoint(label, features) =>
val prediction = model.predict(features)
(prediction, label)
}
// Get evaluation metrics.
val metrics = new MulticlassMetrics(predictionAndLabels)
val precision = metrics.precision
println("Precision = " + precision)
What about JOIN? So RDD.join comes in if you have two RDDs of (key, value) pairs, and need an RDD corresponding to the intersection of keys with both values. But we didn't need that here.